[Enterprise SONiC] LAG (Link Aggregation) Follow
LAG (Link Aggregation) design that aggregates multiple interfaces into single interface for load balancing or failover.
The minimum number of links required to bring up the aggregated link by the parameter of "min-links".
When the LACP fallback feature is set to true, one member port per port channel will be selected as active during fallback mode. The LACP fallback feature allows an active LACP-enabled port to establish an aggregated link (LAG) before it receives an LACP PDUs from its peer.
- LAG configuration
- LAG with min-links
- LAG with fallback
- Setting the specific LACP key
- Setting the specific LACP rate mode
- Static LAG
Restriction:
- LACP is from open source project - teamd in SONiC. It follows IEEE802.3ad and supports LACP only for front physical ports and breakout ports.
-
Here's the naming rule that must be followed, otherwise it cannot create the port channel successfully.
root@sonic:~# sudo config portchannel add portchanne123
Usage: config portchannel add [OPTIONS] <portchannel_name>
Try "config portchannel add -h" for help.
Error: portchanne123 is invalid!, name should have prefix 'PortChannel' and suffix '<0-9999>' - Before LACP establish, the LAG blocks all packets. (default LACP fallback = False) If you enable LACP fallback will let one of member ports as Portchannel behavior before LACP portchannel establish. LACP fallback will run on the members of lower port number. (It will only run on one member port).
- SONiC LACP rate(LACP PDU interval) is 30 seconds(slow mode) in default configuration. And it's NOT configurable.
- SONiC Fast-Reboot(Fast-Reload) requires LACP slow mode for all LAG interface. (https://github.com/Azure/SONiC/wiki/Fast-Reboot)
- Due to limitations in the Linux kernel, it is not possible to change the TPID of dynamic port channels to 0x9100 or 0x9200. Changing it to 0x9100 or 0x9200 would result in the port channel interfaces being shut down.
LAG configuration
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0 ~ 202111.8
Topology:
Procedure :
Steps 1. Checking the port configuration
Here's the command to check the port configuration.
admin@sonic:~$ show vlan brief
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status
Caution: have to remove VLAN and IP address from the LAG member ports before LAG Establish. Refer to the following articles.
Steps 2. Create a PortChannel interface (Default LACP fallback = False)
admin@sonic:~$ sudo config portchannel add PortChannel1
(Optional) LACP fallback = True configuration:
Caution:
- If you'd like to change Fallback status from False to True and vice versa, please delete the port channel and create it again with Fallback setting. Otherwise, the fallback setting won't take effect.
admin@sonic:~$ sudo config portchannel add PortChannel1 --fallback=true
Steps 3. Add member ports to PortChannel interface
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet48 admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet52
Caution: Please be sure the port interfce name is correct, in current version it didn't have error handling when your port name is incorrectly. There is error handling in the 202012 branch.
Steps 4. Save the setting to config_db.json
admin@sonic:~$ sudo config save -y
Steps 5. Check the appended setting.
admin@sonic:/etc/sonic$ sudo vi config_db.json
{
...
"PORTCHANNEL": {
"PortChannel1": {
"admin_status": "up",
"fast_rate": "false",
"lacp_key": "auto",
"min_links": "1",
"mtu": "9100"
}
},
"PORTCHANNEL_MEMBER": {
"PortChannel1|Ethernet48": {},
"PortChannel1|Ethernet52": {}
},
...
}
Steps 6. Check the setting status on PortChannel.
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced,
M - mixed speed
No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate
----- ------------ ----------- --------------------------- ---------- ----------- -----------
1 PortChannel1 LACP(A)(Up) Ethernet52(S) Ethernet48(S) 11 auto false
If you see there are no port members (N/A), please return to Step 1.
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced,
M - mixed speed
No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate
----- ------------ ----------- ------- ---------- ----------- -----------
1 PortChannel1 LACP(A)(Dw) N/A N/A auto false
Steps 7. Check the interface status
admin@sonic:~$ show interfaces status Ethernet48-52,PortChannel1
Interface Lanes Speed MTU Oper FEC Alias Vlan Oper Admin ProtoDown Eff Admin Type Asym PFC Oper Speed
------------ ----------- ------- ----- ---------- ------------- ------------ ------ ------- ----------- ----------- ---------------- ---------- ------------
Ethernet48 77,78,79,80 100G 9100 none Eth49(Port49) PortChannel1 up up False up *QSFP28 or later N/A 100G
Ethernet52 85,86,87,88 100G 9100 none Eth50(Port50) PortChannel1 up up False up *QSFP28 or later N/A 100G
PortChannel1 N/A 200G 9100 N/A N/A routed up up False up N/A N/A 200G
LAG with min-links
Tested model & firmware version:
- Switch model name:
DCS204 (AS7726-56X)
- Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0 ~ 202111.3
Topology:
Pre-configuration:
- Ethernet4 should remove IP binding from default configuration.(refer to Management and port IPv4/IPv6 Address)
- Ethernet4 and Ethernet5 is breakout from Ethernet4.(refer to Dynamic Port Breakout)
- The physical link is disconnect between Server(ens11f2) to AS7726-32X(Ethernet5).
Expect result:
With min-links configuration, the only one LAG member is up, the PortChannel1 is down.
Without min-links configuration, the only one LAG member is up, the PortChannel1 is up.
Procedure:
Steps 1. Create a portchannel with min-links
admin@sonic:~$ sudo config portchannel add PortChannel1 --min-links=2
Steps 2. Append Ethernet4 and Ethernet5 to the PortChannel1 member. Check PortChannel1 status as "Result" section.
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet5
Steps 3. Remove PortChannel1.
admin@sonic:~$ sudo config portchannel member del PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member del PortChannel1 Ethernet5
admin@sonic:~$ sudo config portchannel del PortChannel1
Steps 4. Create PortChannel1 without min-links configuration. Check PortChannel1 status as "Result" section.
admin@sonic:~$ sudo config portchannel add PortChannel1
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet4
admin@sonic:~$ sudo config portchannel member add PortChannel1 Ethernet5
Result:
- PortChannel create with min-links configuration.
admin@sonic:~$ show interfaces status Ethernet4-5
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC Oper Speed
----------- ------- ------- ----- ----- ------------- ------------ ------ ------- -------------- ---------- ------------
Ethernet4 5 10G 9100 none Eth2/1(Port2) PortChannel1 up up QSFP+ or later N/A 10G
Ethernet5 6 10G 9100 none Eth2/2(Port2) PortChannel1 down up QSFP+ or later N/A 10G
admin@sonic:~$ show interfaces status PortChannel1
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC Oper Speed
------------ ------- ------- ----- ----- ------- ------ ------ ------- ------ ---------- ------------
PortChannel1 N/A 20G 9100 N/A N/A routed down up N/A N/A 20G
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced
No. Team Dev Protocol Ports
----- ------------ ----------- -------------------------
1 PortChannel1 LACP(A)(Dw) Ethernet4(S) Ethernet5(D)
- PortChannel create without min-links configuration.
admin@sonic:~$ show interfaces status Ethernet4-5
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC Oper Speed
----------- ------- ------- ----- ----- ------------- ------------ ------ ------- -------------- ---------- ------------
Ethernet4 5 10G 9100 none Eth2/1(Port2) PortChannel1 up up QSFP+ or later N/A 10G
Ethernet5 6 10G 9100 none Eth2/2(Port2) PortChannel1 down up QSFP+ or later N/A 10G
admin@sonic:~$ show interfaces status PortChannel1
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC Oper Speed
------------ ------- ------- ----- ----- ------- ------ ------ ------- ------ ---------- ------------
PortChannel1 N/A 20G 9100 N/A N/A routed up up N/A N/A 20G
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced
No. Team Dev Protocol Ports
----- ------------ ----------- -------------------------
1 PortChannel1 LACP(A)(Up) Ethernet4(S) Ethernet5(D)
LAG with fallback
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
DCS204 (AS7726-32X)
- Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0 ~ 202111.8
Topology:
Pre-configuration:
- Ethernet0 should remove IP binding from default configuration on AS7726-32X.(refer to Management and port IPv4/IPv6 Address)
- The Ethernet48 of AS7362-56X bind ip as topology.(refer to Management and port IPv4/IPv6 Address)
Expect result:
- The traffic can forward well between AS7326-56X and AS7726-32X, if the portchannel let fallback true on AS7726-32X.
- The traffic cannot forward well between AS7326-56X and AS7726-32X, if the portchannel is not set fallback on AS7726-32X.
Procedure:
Steps 1. Create PortChannel2 to fallback true on AS7726-32X.
admin@sonic:~$ sudo config portchannel add PortChannel2 --fallback=true
Steps 2. Append member port to PortChannel2 on AS7726-32X.
admin@sonic:~$ sudo config portchannel member add PortChannel2 Ethernet0
admin@sonic:~$ show interfaces portchannel
Steps 3. Binding IP address to interface as topology.
AS7726-32X:
admin@sonic:~$ sudo config interface ip add PortChannel2 192.168.10.1/24
AS7326-56X:
admin@sonic:~$ sudo config interface ip add Ethernet48 192.168.10.2/24
Steps 4. Remove PortChannel2 configuration and create again without fallback.
admin@sonic:~$ sudo config interface ip remove PortChannel2 192.168.10.1/24
admin@sonic:~$ sudo config portchannel member del PortChannel2 Ethernet0
admin@sonic:~$ sudo config portchannel del PortChannel2
admin@sonic:~$ sudo config portchannel add PortChannel2
admin@sonic:~$ sudo config portchannel member add PortChannel2 Ethernet0
admin@sonic:~$ sudo config interface ip add PortChannel2 192.168.10.1/24
Result:
- The fallback is true on PortChannel2.
admin@sonic:~$ show interfaces status Ethernet0 Interface Lanes Speed MTU Oper FEC Alias Vlan Oper Admin ProtoDown Eff Admin Type Asym PFC Oper Speed ----------- ------- ------- ----- ---------- ----------- ------ ------ ------- ----------- ----------- ------ ---------- ------------ Ethernet0 1,2,3,4 100G 9100 rs Eth1(Port1) routed up up False up N/A N/A 100G admin@sonic:~$ show interfaces portchannel Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected, * - not synced, M - mixed speed No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate ----- ------------ ----------- ------------ ---------- ----------- ----------- 2 PortChannel2 LACP(A)(Up) Ethernet0(S) 12 auto false
admin@sonic:~$ ping 192.168.10.2 -c 2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=0.667 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=0.251 ms
--- 192.168.10.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 18ms
rtt min/avg/max/mdev = 0.251/0.459/0.667/0.208 ms
- The fallback is not set on PortChannel.
admin@sonic:~$ show interfaces status Ethernet0 Interface Lanes Speed MTU Oper FEC Alias Vlan Oper Admin ProtoDown Eff Admin Type Asym PFC Oper Speed ----------- ------- ------- ----- ---------- ----------- ------ ------ ------- ----------- ----------- ------ ---------- ------------ Ethernet0 1,2,3,4 100G 9100 rs Eth1(Port1) routed up up False up N/A N/A 100G admin@sonic:~$ show interfaces portchannel Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected, * - not synced, M - mixed speed No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate ----- ------------ ----------- ------------ ---------- ----------- ----------- 2 PortChannel2 LACP(A)(Dw) Ethernet0(D) 12 auto false
admin@sonic:~$ ping -c 2 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
From 192.168.10.1 icmp_seq=1 Destination Host Unreachable
From 192.168.10.1 icmp_seq=2 Destination Host Unreachable
--- 192.168.10.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 5ms
pipe 2
Setting the specific LACP key
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202012.3 ~ 202012.4
202111.0 ~ 202111.8
Topology:
Restriction:
- It cannot modify the LACP key on the existing PortChannel, it could only remove it and re-create a new one.
- The default LACP key is "0".
Procedure:
Step 1: Checking the port configuration
Here's the command to check the port configuration.
admin@sonic:~$ show vlan brief
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status
caution: It has to remove VLAN and IP address from the LAG member ports before LAG is established. Refer to the following articles.
Step 2: Create a PortChannel interface and set the specific LACP key.
admin@sonic:~$ sudo config portchannel add PortChannel4 --lacp-key 15
Note:
- The range of lacp-key value is 1 - 65535.
-
Except for the integer, it could also use "auto" to specific the LACP key.
'auto" doesn't mean auto-detect. It means it will give you the LACP key automatically. It depends on your PortChannel number. The rule is 1 + "PortChannel number".
For example, the PortChannel4, and its LACP key is "14"admin@sonic:~$ sudo config portchannel add PortChannel4 --lacp-key auto
Step 3: Add member ports to PortChannel interface
admin@sonic:~$ sudo config portchannel member add PortChannel4 Ethernet48
Step 4. Save the setting to config_db.json
admin@sonic:~$ sudo config save -y
Step 5: Check the LACP key
Before 202012.4 version, checking the LACP key could only use the command of the "teamd".
admin@sonic:~$ teamdctl PortChannel4 state dump
{
...
omitted
...
"runner": {
"actor_lacpdu_info": {
"key": 15,
"port": 49,
"port_priority": 255,
"state": 0,
"system": "68:21:5f:dc:3a:c8",
"system_priority": 65535
},
...
omitted
...
}
After 202012.4 version, SONiC command supports checking the LACP key.
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced,
M - mixed speed
No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate
----- ------------ ----------- ------------- ---------- ----------- -----------
4 PortChannel4 LACP(A)(Up) Ethernet48(S) 15 15 false
Setting the specific LACP rate mode
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202111.2
202111.8
Topology:
Restriction:
- the default LACP rate is slow mode.
Procedure:
Step 1: Checking the port configuration
Here's the command to check the port configuration.
admin@sonic:~$ show vlan brief
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status
caution: It has to remove VLAN and IP address from the LAG member ports before LAG is established. Refer to the following articles.
Step 2: Create a PortChannel interface and set the LACP rate mode.
admin@sonic:~$ sudo config portchannel add PortChannel01 --fast-rate true
Note:
fast-rate, specify LACP rate mode.
- true, LACP fast rate mode, request partner to transmit LACPDUs every 1 second. LACP timeout is 3 seconds.
- false, LACP slow rate mode, request partner to transmit LACPDUs every 30 seconds. LACP timeout is 90 seconds
Step 3: Add member ports to PortChannel interface
admin@sonic:~$ sudo config portchannel member add PortChannel01 Ethernet48
Step 4. Save the setting to config_db.json
admin@sonic:~$ sudo config save -y
Step 5: Check the LACP rate
admin@sonic:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced,
M - mixed speed
No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate
----- ------------- ----------- ------------ ---------- ----------- -----------
01 PortChannel01 LACP(A)(Up) Ethernet48(S) 101 auto true
Static LAG
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202111.2 ~ 202111.8
Topology:
Procedure:
Step 1: Checking the port configuration
Here's the command to check the port configuration.
admin@sonic:~$ show vlan brief
admin@sonic:~$ show ip interfaces
admin@sonic:~$ show interfaces status
caution: It has to remove VLAN and IP address from the LAG member ports before LAG is established. Refer to the following articles.
Step 2: Create a PortChannel interface
admin@sonic:~$ sudo config portchannel add PortChannel01 --static true
Note:
-
static, specify the port channel type, either true for static port channel or false for dynamic LACP port channel.
Step 3: Add member ports to PortChannel interface
admin@sonic:~$ sudo config portchannel member add PortChannel01 Ethernet48
Step 4. Save the setting to config_db.json
admin@sonic:~$ sudo config save -y
Step 5: Check the LACP status
admin@sonic:~$ show interfaces portchannel Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected, * - not synced, M - mixed speed No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate ----- ------------- ----------- ------------ ---------- ----------- ----------- 01 PortChannel01 NONE(-)(Up) Ethernet48(S) N/A N/A N/A
Comments
0 comments
Please sign in to leave a comment.