[Edgecore SONiC] LAG (Link Aggregation) – Edgecore Help Center

LAG (Link Aggregation) design that aggregates multiple interfaces into single interface for load balancing or failover.

The minimum number of links required to bring up the aggregated link by the parameter of "min-links".

When the LACP fallback feature is set to true, one member port per port channel will be selected as active during fallback mode. The LACP fallback feature allows an active LACP-enabled port to establish an aggregated link (LAG) before it receives an LACP PDUs from its peer.

LAG configuration
LAG with min-links
LAG with fallback
Setting the specific LACP key

Restriction:

Both community SONiC and ecSONiC support only LACP. (Does NOT support static link aggregation)
LACP is from open source project - teamd in SONiC. It follows IEEE802.3ad and supports LACP only for front physical ports and breakout ports.

Here's the naming rule that must be followed, otherwise it cannot create the port channel successfully.

root@sonic:~# sudo config portchannel add portchanne123
Usage: config portchannel add [OPTIONS] <portchannel_name>
Try "config portchannel add -h" for help.

Error: portchanne123 is invalid!, name should have prefix 'PortChannel' and suffix '<0-9999>'

Before LACP establish, the LAG blocks all packets. (default LACP fallback = False) If you enable LACP fallback will let one of member ports as Portchannel behavior before LACP portchannel establish. LACP fallback will run on the members of lower port number. (It will only run on one member port).
SONiC LACP rate(LACP PDU interval) is 30 seconds(slow mode) in default configuration. And it's NOT configurable.
SONiC Fast-Reboot(Fast-Reload) requires LACP slow mode for all LAG interface. (https://github.com/Azure/SONiC/wiki/Fast-Reboot)

LAG configuration

Tested model & firmware version:

Switch model name:
DCS203 (AS7326-56X)

Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0

Topology:

Procedure :

Steps 1. Checking the port configuration

Here's the command to check the port configuration.

admin@sonic:~$ show vlan brief 
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status

Caution: have to remove VLAN and IP address from the LAG member ports before LAG Establish. Refer to the following articles.

Steps 2. Create a PortChannel interface (Default LACP fallback = False)

admin@sonic:~$ sudo config portchannel add PortChannel1

(Optional) LACP fallback = True configuration:

Caution:

If you'd like to change Fallback status from False to True and vice versa, please delete the port channel and create it again with Fallback setting. Otherwise, the fallback setting won't take effect.

admin@sonic:~$ sudo config portchannel add PortChannel1 --fallback=true

Steps 3. Add member ports to PortChannel interface

admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet48        
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet52

Caution: Please be sure the port interfce name is correct, in current version it didn't have error handling when your port name is incorrectly. There is error handling in the 202012 branch.

Steps 4. Save the setting to config_db.json

admin@sonic:~$ sudo config save -y

Steps 5. Check the appended setting.

admin@sonic:/etc/sonic$ sudo vi config_db.json
  {
      ...
     "PORTCHANNEL": {
          "PortChannel1": {
              "admin_status": "up",
              "mtu": "9100"
          }
      },
      "PORTCHANNEL_MEMBER": {
          "PortChannel1|Ethernet48": {},
          "PortChannel1|Ethernet52": {}
      },
      ...
  }

Steps 6. Check the setting status on PortChannel.

admin@sonic:~$ show interfaces portchannel
  Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
         S - selected, D - deselected, * - not synced
    No.  Team Dev      Protocol     Ports
  -----  ------------  -----------  ---------------------------
      1  PortChannel1  LACP(A)(Up)  Ethernet52(S) Ethernet48(S)

If you see there are no port members (N/A), please return to Step 1.

admin@sonic:~$ show interfaces portchannel 
  Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
         S - selected, D - deselected, * - not synced
    No.  Team Dev      Protocol     Ports
  -----  ------------  -----------  -------
      1  PortChannel1  LACP(A)(Dw)  N/A

Steps 7. Check the interface status

admin@LEAF01:~$ show interfaces status Ethernet48-52,PortChannel1
   Interface        Lanes    Speed    MTU    FEC            Alias          Vlan    Oper    Admin             Type    Asym PFC    Oper Speed
------------  -----------  -------  -----  -----  ---------------  ------------  ------  -------  ---------------  ----------  ------------
  Ethernet48  77,78,79,80     100G   9100   none  Eth49/1(Port49)  PortChannel1    up         up   QSFP+ or later         N/A          100G
  Ethernet52  85,86,87,88     100G   9100   none  Eth50/1(Port50)  PortChannel1    up         up  QSFP28 or later         N/A          100G
PortChannel1          N/A     200G   9100    N/A              N/A        routed    up         up              N/A         N/A          200G

LAG with min-links

Tested model & firmware version:

Switch model name:
DCS204 (AS7726-56X)

Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0

Topology:

Pre-configuration:

Ethernet4 should remove IP binding from default configuration.(refer to Management and port IPv4/IPv6 Address)
Ethernet4 and Ethernet5 is breakout from Ethernet4.(refer to Dynamic Port Breakout)
The physical link is disconnect between Server(ens11f2) to AS7726-32X(Ethernet5).

Expect result:

With min-links configuration, the only one LAG member is up, the PortChannel1 is down.

Without min-links configuration, the only one LAG member is up, the PortChannel1 is up.

Procedure:

Steps 1. Create a portchannel with min-links

admin@sonic:~$ sudo config portchannel add PortChannel1 --min-links=1

Steps 2. Append Ethernet4 and Ethernet5 to the PortChannel1 member. Check PortChannel1 status as "Result" section.

admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet5

Steps 3. Remove PortChannel1.

admin@sonic:~$ sudo config portchannel member del PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member del PortChannel1 Ethernet5
admin@sonic:~$ sudo config portchannel del PortChannel1

Steps 4. Create PortChannel1 without min-links configuration. Check PortChannel1 status as "Result" section.

admin@sonic:~$ sudo config portchannel add PortChannel1
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet5

Result:

PortChannel create with min-links configuration.

admin@sonic:~$ show interfaces status Ethernet4-5
  Interface    Lanes    Speed    MTU    FEC          Alias          Vlan    Oper    Admin            Type    Asym PFC    Oper Speed
-----------  -------  -------  -----  -----  -------------  ------------  ------  -------  --------------  ----------  ------------
  Ethernet4        5      10G   9100   none  Eth2/1(Port2)  PortChannel1      up       up  QSFP+ or later         N/A           10G
  Ethernet5        6      10G   9100   none  Eth2/2(Port2)  PortChannel1    down       up  QSFP+ or later         N/A           10G
admin@sonic:~$ show interfaces status PortChannel1
   Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin    Type    Asym PFC    Oper Speed
------------  -------  -------  -----  -----  -------  ------  ------  -------  ------  ----------  ------------
PortChannel1      N/A      20G   9100    N/A      N/A  routed    down       up     N/A         N/A           20G
admin@sonic:~$ show interfaces portchannel

Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev      Protocol     Ports
-----  ------------  -----------  -------------------------
    1  PortChannel1  LACP(A)(Dw)  Ethernet4(S) Ethernet5(D)

PortChannel create without min-links configuration.

admin@sonic:~$ show interfaces status Ethernet4-5
  Interface    Lanes    Speed    MTU    FEC          Alias          Vlan    Oper    Admin            Type    Asym PFC    Oper Speed
-----------  -------  -------  -----  -----  -------------  ------------  ------  -------  --------------  ----------  ------------
  Ethernet4        5      10G   9100   none  Eth2/1(Port2)  PortChannel1      up       up  QSFP+ or later         N/A           10G
  Ethernet5        6      10G   9100   none  Eth2/2(Port2)  PortChannel1    down       up  QSFP+ or later         N/A           10G
admin@sonic:~$ show interfaces status PortChannel1
   Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin    Type    Asym PFC    Oper Speed
------------  -------  -------  -----  -----  -------  ------  ------  -------  ------  ----------  ------------
PortChannel1      N/A      20G   9100    N/A      N/A  routed      up       up     N/A         N/A           20G
admin@sonic:~$ show interfaces portchannel

Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev      Protocol     Ports
-----  ------------  -----------  -------------------------
    1  PortChannel1  LACP(A)(Up)  Ethernet4(S) Ethernet5(D)

LAG with fallback

Tested model & firmware version:

Switch model name:
DCS203 (AS7326-56X)
DCS204 (AS7726-32X)

Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0

Topology:

Pre-configuration:

Ethernet0 should remove IP binding from default configuration on AS7726-32X.(refer to Management and port IPv4/IPv6 Address)
The Ethernet48 of AS7362-56X bind ip as topology.(refer to Management and port IPv4/IPv6 Address)

Expect result:

The traffic can forward well between AS7326-56X and AS7726-32X, if the portchannel let fallback true on AS7726-32X.
The traffic cannot forward well between AS7326-56X and AS7726-32X, if the portchannel is not set fallback on AS7726-32X.

Procedure:

Steps 1. Create PortChannel2 to fallback true on AS7726-32X.

admin@sonic:~$ sudo config portchannel add PortChannel2 --fallback=true

Steps 2. Append member port to PortChannel2 on AS7726-32X.

admin@sonic:~$ sudo config portchannel member add PortChannel2 Ethernet0
admin@sonic:~$ show interfaces portchannel

Steps 3. Binding IP address to interface as topology.

AS7726-32X:

admin@sonic:~$ sudo config interface ip add PortChannel2 192.168.10.1/24

AS7326-56X:

admin@sonic:~$ sudo config interface ip add Ethernet48 192.168.10.2/24

Steps 4. Remove PortChannel2 configuration and create again without fallback.

admin@sonic:~$ sudo config interface ip remove PortChannel2 192.168.10.1/24
admin@sonic:~$ sudo config portchannel member del PortChannel2 Ethernet0
admin@sonic:~$ sudo config portchannel del PortChannel2
admin@sonic:~$ sudo config portchannel add PortChannel2
admin@sonic:~$ sudo config portchannel member add PortChannel2 Ethernet0
admin@sonic:~$ sudo config interface ip add PortChannel2 192.168.10.1/24

Result:

The fallback is true on PortChannel2.

admin@sonic:~$ show interface status Ethernet0
  Interface    Lanes    Speed    MTU    FEC          Alias          Vlan    Oper    Admin            Type    Asym PFC    Oper Speed
-----------  -------  -------  -----  -----  -------------  ------------  ------  -------  --------------  ----------  ------------
  Ethernet0  1,2,3,4     100G   9100   none  Eth1/1(Port1)  PortChannel2      up       up  QSFP+ or later         N/A          100G
admin@sonic:~$ show interface portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev      Protocol     Ports
-----  ------------  -----------  ------------
    2  PortChannel2  LACP(A)(Up)  Ethernet0(S)
admin@sonic:~$ ping 192.168.10.2 -c 2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=0.667 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=0.251 ms
--- 192.168.10.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 18ms
rtt min/avg/max/mdev = 0.251/0.459/0.667/0.208 ms

The fallback is not set on PortChannel.

admin@sonic:~$ show interface status Ethernet0
  Interface    Lanes    Speed    MTU    FEC          Alias          Vlan    Oper    Admin            Type    Asym PFC    Oper Speed
-----------  -------  -------  -----  -----  -------------  ------------  ------  -------  --------------  ----------  ------------
  Ethernet0  1,2,3,4     100G   9100   none  Eth1/1(Port1)  PortChannel2      up       up  QSFP+ or later         N/A          100G
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev      Protocol     Ports
-----  ------------  -----------  ------------
    2  PortChannel2  LACP(A)(Dw)  Ethernet0(D)
admin@sonic:~$ ping -c 2 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
From 192.168.10.1 icmp_seq=1 Destination Host Unreachable
From 192.168.10.1 icmp_seq=2 Destination Host Unreachable

 --- 192.168.10.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 5ms
pipe 2

 
Setting the specific LACP key 
Tested model & firmware version:
Switch model name:
DCS203 (AS7326-56X)

Edgecore SONiC version: 
202012.3 ~ 202012.4
202111.0

Topology:

Restriction:

It cannot modify the LACP key on the existing PortChannel, it could only remove it and re-create a new one.
The default LACP key is "0".

Procedure:

Step 1: Checking the port configuration

Here's the command to check the port configuration.

admin@sonic:~$ show vlan brief 
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status

caution: It has to remove VLAN and IP address from the LAG member ports before LAG is established. Refer to the following articles.

Step 2: Create a PortChannel interface and set the specific LACP key.

admin@sonic:~$ sudo config portchannel add PortChannel4 --lacp-key 15

Note:

The range of lacp-key value is 1 - 65535.
Except for the integer, it could also use "auto" to specific the LACP key.
'auto" doesn't mean auto-detect. It means it will give you the LACP key automatically. It depends on your PortChannel number. The rule is 1 + "PortChannel number".
For example, the PortChannel4, and its LACP key is "14"
```
admin@sonic:~$ sudo config portchannel add PortChannel4 --lacp-key auto 
```

Step 3: Add member ports to PortChannel interface

admin@sonic:~$ sudo config portchannel member add PortChannel4 Ethernet48

Step 4. Save the setting to config_db.json

admin@sonic:~$ sudo config save -y

Step 5: Check the LACP key
Before 202012.4 version, checking the LACP key could only use the command of the "teamd".

admin@sonic:~$ teamdctl PortChannel4 state dump
{
... 
omitted
...
            "runner": {
                "actor_lacpdu_info": {
                    "key": 15,
                    "port": 6,
                    "port_priority": 255,
                    "state": 61,
                    "system": "68:21:5f:dc:3a:c8",
                    "system_priority": 65535
                },
...
omitted
...
}

After 202012.4 version, SONiC command supports checking the LACP key.

admin@sonic:~$ show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev      Protocol     Ports            Oper_key    Admin_key
-----  ------------  -----------  -------------  ----------  -----------
    4  PortChannel4  LACP(A)(Up)  Ethernet48(S)          15           15

Category

[Edgecore SONiC] LAG (Link Aggregation) Follow

Restriction:

LAG configuration

Tested model & firmware version:

Topology:

Procedure :

LAG with min-links

Tested model & firmware version:

Topology:

Pre-configuration:

Expect result:

Procedure:

Result:

LAG with fallback

Tested model & firmware version:

Topology:

Pre-configuration:

Expect result:

Procedure:

Result:

Setting the specific LACP key

Tested model & firmware version:

Topology:

Restriction:

Procedure:

Comments

Articles in this section

Category

Restriction:

LAG configuration

Tested model & firmware version:

Topology:

Procedure :

LAG with min-links

Tested model & firmware version:

Topology:

Pre-configuration:

Expect result:

Procedure:

Result:

LAG with fallback

Tested model & firmware version:

Topology:

Pre-configuration:

Expect result:

Procedure:

Result:

Setting the specific LACP key

Tested model & firmware version:

Topology:

Restriction:

Procedure:

Related articles

Articles in this section