[Enterprise SONiC] Dynamic Port Breakout Follow
Ports can be broken out to different speeds with various lanes. E.g. If we want to change port 0 from 100G to port 0-4 with 4x25G.
How to configure the DPB (Dynamic Port Breakout) on Edgecore SONiC.
Restriction
-
[202111.0] known issue:
-
[SONIC-4681] Switch failed to break out interface if eth0 has static IP address
The issue is fixed on 202111.3
- [SONIC-4331] If there had been ACL that without IP_Type binding on some ports, once other ports try to do the dynamic port breakout, they will face an error.
-
[SONIC-4681] Switch failed to break out interface if eth0 has static IP address
- Before breakout, please make sure the port restore to default configuration. In 202012 branch, "advertised-speed" command is supported. If you have advertised-speed configuration, please remember to remove the configuration. (Refer to this restored guide for config restore. )
- Admin status of interfaces is down by default after applying Dynamic Port Breakout setting.
- AS5835-54X(T), it could only set the port breakout on Ethernet48 and Ethernet60.
- On the version "Edgecore-SONiC_20210209_051110_ec202006_134", AS9716-32D support more breakout mode "2x50G(4), 4x25G[10G](4)".
- AS7816-64X, due to the logical port limitation by ASIC, breakout is realized only in row1 (Ethernet0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120 as the yellow block) in current design.
- For the mode ‘4x25[10G](4)’ of AS9716-32D, the default breakout is 4x25G. If you need 4x10G, you need to break out first and then use the down-speed command as following.
admin@sonic:~$ sudo config interface breakout Ethernet24 '4x25G[10G](4)'
Do you want to Breakout the port, continue? [y/N]: y
Running Breakout Mode : 1x400G
Target Breakout Mode : 4x25G[10G](4)
Ports to be deleted :
{
"Ethernet24": "400000"
}
Ports to be added :
{
"Ethernet25": "25000",
"Ethernet24": "25000",
"Ethernet27": "25000",
"Ethernet26": "25000"
}
After running Logic to limit the impact
Final list of ports to be deleted :
{
"Ethernet24": "400000"
}
Final list of ports to be added :
{
"Ethernet25": "25000",
"Ethernet24": "25000",
"Ethernet27": "25000",
"Ethernet26": "25000"
}
Loaded below Yang Models
['sonic-acl', 'sonic-extension', 'sonic-interface', 'sonic-loopback-interface', 'sonic-port', 'sonic-portchannel', 'sonic-types', 'sonic-vlan']
Note: Below table(s) have no YANG models:
CONTAINER_FEATURE, VERSIONS, DEVICE_METADATA, TELEMETRY, FEATURE, FLEX_COUNTER_TABLE, BREAKOUT_CFG, MGMT_INTERFACE, CRM,
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run `config save`.
admin@sonic:~$ sudo config interface speed Ethernet24 10000
Dynamic Port Breakout
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202006.4
202012.0 ~ 202012.4
202111.0 ~ 202111.8
Procedure:
Step 1. We should remove the ip interface/ VLAN member and portchannel member on ports which we want to run the DPB.
Assume if we want to run DPB on "Ethernet48", we need to remove the ip interface on the port.
admin@sonic:~$ show ip interfaces Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP ----------- -------- ------------------- ------------ -------------- ------------- Ethernet48 192.168.1.1/24 up/up N/A N/A docker0 240.127.1.1/24 up/down N/A N/A eth0 188.188.9.3/16 up/up N/A N/A lo 127.0.0.1/16 up/up N/A N/A
admin@sonic:~$ sudo config interface ip remove Ethernet48 192.168.1.1/24
Assume if we want to run DPB on "Ethernet52", we should remove it from the VLAN member of VLAN100.
admin@sonic:~$ show vlan brief +-----------+----------------+------------+-----------+----------+---------------+----------------------------+ | VLAN ID | IP Address | Ports | Port | Proxy | DHCP Helper | DHCP Relay Configuration | | | | | Tagging | ARP | Address | | +===========+================+============+===========+==========+===============+============================+ | 100 | 192.168.2.2/24 | Ethernet52 | untagged | disabled | | Source Interface: | | | | | | | | Link Selection: | | | | | | | | Server Vrf: | | | | | | | | Server ID Override: | +-----------+----------------+------------+-----------+----------+---------------+----------------------------+ admin@sonic:~$ sudo config vlan member del 100 Ethernet52
Assume if we want to run the DPB on "Ethernet0", we should remove it from portchannel member of PortChannel01.
admin@sonic:~$ show interfaces portchannel Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected, * - not synced, M - mixed speed No. Team Dev Protocol Ports Oper Key Admin Key Fast Rate ----- ------------- ----------- ------------ ---------- ----------- ----------- 01 PortChannel01 LACP(A)(Dw) Ethernet0(D) 101 auto false
admin@sonic:~$ sudo config portchannel member del PortChannel01 Ethernet0
Step 2. Check breakout option availability and current breakout mode on Ethernet48.
admin@sonic:~$ show interfaces breakout
...omitted... "Ethernet48": { "index": "13,13,13,13", "lanes": "49,50,51,52", "breakout_modes": { "1x100G[40G]": [ "Eth13(Port13)" ], "2x50G": [ "Eth13/1(Port13)", "Eth13/2(Port13)" ], "4x25G": [ "Eth13/1(Port13)", "Eth13/2(Port13)", "Eth13/3(Port13)", "Eth13/4(Port13)" ], "4x10G": [ "Eth13/1(Port13)", "Eth13/2(Port13)", "Eth13/3(Port13)", "Eth13/4(Port13)" ] }, "default_brkout_mode": "1x100G[40G]", "autoneg": "off", "Current Breakout Mode": "1x100G[40G]", "child ports": "Ethernet48", "child port speeds": "100G" }, ...omitted... admin@sonic:~$ show interfaces breakout current-mode Ethernet48 +-------------+-------------------------+ | Interface | Current Breakout Mode | +=============+=========================+ | Ethernet48 | 1x100G[40G] | +-------------+-------------------------+
Step 3. Configure dynamic port breakout
admin@sonic:~$ sudo config interface breakout Ethernet48 '4x25G' -y Running Breakout Mode : 1x100G[40G] Target Breakout Mode : 4x25G Ports to be deleted : { "Ethernet48": "100000" } Ports to be added : { "Ethernet48": "25000", "Ethernet49": "25000", "Ethernet50": "25000", "Ethernet51": "25000" } After running Logic to limit the impact Final list of ports to be deleted : { "Ethernet48": "100000" } Final list of ports to be added : { "Ethernet48": "25000", "Ethernet49": "25000", "Ethernet50": "25000", "Ethernet51": "25000" } sonic_yang(6):Note: Below table(s) have no YANG models: REST_SERVER, SNMP, SNMP_COMMUNITY, XCVRD_LOG Breakout process got successfully completed. Please note loaded setting will be lost after system reboot. To preserve setting, run `config save`.
Caution: Be careful, please use include the mode setting with ' '.
Step 4. Show breakout current mode status and interface status
admin@sonic:~$ show interfaces breakout current-mode Ethernet48
+-------------+-------------------------+
| Interface | Current Breakout Mode |
+=============+=========================+
| Ethernet48 | 4x25G |
+-------------+-------------------------+
admin@sonic:~$ show interfaces status Ethernet48-51
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC
----------- ------- ------- ----- ----- --------------- ------ ------ ------- -------------- ----------
Ethernet48 77 25G 9100 none Eth49/1(Port49) routed down down QSFP+ or later N/A
Ethernet49 78 25G 9100 none Eth49/2(Port49) routed down down N/A N/A
Ethernet50 79 25G 9100 none Eth49/3(Port49) routed down down N/A N/A
Ethernet51 80 25G 9100 none Eth49/4(Port49) routed down down N/A N/A
Step 5. Set the admin status of interface to startup
admin@sonic:~$ sudo config interface startup Ethernet48-51
admin@sonic:~$ show interfaces status Ethernet48-51
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC
----------- ------- ------- ----- ----- --------------- ------ ------ ------- -------------- ----------
Ethernet48 77 25G 9100 none Eth49/1(Port49) routed up up QSFP+ or later N/A
Ethernet49 78 25G 9100 none Eth49/2(Port49) routed up up N/A N/A
Ethernet50 79 25G 9100 none Eth49/3(Port49) routed up up N/A N/A
Ethernet51 80 25G 9100 none Eth49/4(Port49) routed up up N/A N/A
Step 6. Save configuration to startup configuration.
admin@sonic:~$ sudo config save -y
Running command: /usr/local/bin/sonic-cfggen -d --print-data > /etc/sonic/config_db.json
Configure Interface Breakout Mode 8x100G for TH5
Start from version 202211.2, TH5(AS9817-64D/O) can support 8x100G, but it can't support 8x100G for all ports. For ports in row 1 or row 3, they can breakout to 8 ports, but ports in row 2 or row 4, can only breakout to 2 ports. (does not support "4x200G[100G]" and "4x100G[50G](4)")
Tested model & firmware version:
- Switch model name:
AS9817-64D
- Edgecore SONiC version:
202211.4
By default, the switch does not support the 8x100G mode.
admin@sonic:~$ sudo config interface breakout Ethernet0 8x100G -y
[ERROR] Target mode 8x100G is not available for the port Ethernet0
Aborted!
Procedure:
Step 1. Switch to the root account.
admin@sonic:~$ sudo su
root@sonic:/home/admin#
Step 2. Switch to the directory base the switch model
For AS9817-64D
root@sonic:/home/admin# cd /usr/share/sonic/device/x86_64-accton_as9817_64d-r0/
root@sonic:/usr/share/sonic/device/x86_64-accton_as9817_64d-r0#
For AS9817-64O
root@sonic:/home/admin# cd /usr/share/sonic/device/x86_64-accton_as9817_64o-r0/
root@sonic:/usr/share/sonic/device/x86_64-accton_as9817_64o-r0#
Step 3. Use "dpb_port_profile" to switch DPB profile from 0(default) to 1
Running "dpb_port_profile -p" to change the profile would only change the supported port breakout mode, not the current port breakout mode.
Before running "dpb_port_profile -p", make sure that the current port breakout mode allowed the profile you want to change.
root@sonic:/usr/share/sonic/device/x86_64-accton_as9817_64d-r0# ./dpb_port_profile -p 1
Change DPB profile to 1
You may also use the following commands to check the current configuration
root@sonic:/usr/share/sonic/device/x86_64-accton_as9817_64d-r0# ./dpb_port_profile -s Show the port breakout capability of each profile Profile 0(default profile): ================================================================================ |Port # | Breakout Mode | Default Breakout Mode | -------------------------------------------------------------------------------- |1 ~ 64 | 1x800G, 1x400G, 2x400G[200G], 4x200G[100G],| 1x400G | | | 4x100G[50G](4),1x100G(4),2x50G(4) | | ================================================================================ Profile 1: ======================================================================================== | Port # | Breakout Mode | Default Breakout Mode | ---------------------------------------------------------------------------------------- |1,3,5,...,63 | 1x800G, 1x400G, 2x400G[200G], 4x200G[100G],| | |(Row 1/3 Port) | 4x100G[50G](4),1x100G(4),2x50G(4), 8x100G | | -------------------------------------------------------------- 1x400G | |2,4,6,...,64 | 1x800G, 1x400G, 2x400G[200G], | | |(Row 2/4 Port) | 1x100G(4),2x50G(4) | | ======================================================================================= The current DPB profile is 1
Step 4. Configure dynamic port breakout mode to 8x100G mode
admin@sonic:~$ sudo config interface breakout Ethernet0 8x100G -y
Running Breakout Mode : 1x800G
Target Breakout Mode : 8x100G
Ports to be deleted :
{
"Ethernet0": "800000"
}
Ports to be added :
{
"Ethernet0": "100000",
"Ethernet1": "100000",
"Ethernet2": "100000",
"Ethernet3": "100000",
"Ethernet4": "100000",
"Ethernet5": "100000",
"Ethernet6": "100000",
"Ethernet7": "100000"
}
sonic_yang(6):Note: Below table(s) have no YANG models: REST_SERVER
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run `config save`.
Step 5. Set the admin status of interface to startup
admin@sonic:~$ sudo config interface startup Ethernet0-7
Step 6. Save configuration to startup configuration
admin@sonic:~$ sudo config save -y
Running command: /usr/local/bin/sonic-cfggen -d --print-data /etc/sonic/config_db.json
Frequently Asked Question
Why failed to breakout after typing the correct command?
Tested model & firmware version:
- Switch model name:
DCS203 (AS7326-56X)
- Edgecore SONiC version:
202012.4
202111.3
Here are some example cases where users were unable to successfully perform a breakout:
1. Problem description:
admin@sonic:~$ sudo config interface breakout Ethernet48 '4x25G' -y
Running Breakout Mode : 1x100G[40G]
Target Breakout Mode : 4x25G
Ports to be deleted :
{
"Ethernet48": "100000"
}
Ports to be added :
{
"Ethernet48": "25000",
"Ethernet49": "25000",
"Ethernet50": "25000",
"Ethernet51": "25000"
}
After running Logic to limit the impact
Final list of ports to be deleted :
{
"Ethernet48": "100000"
}
Final list of ports to be added :
{
"Ethernet48": "25000",
"Ethernet49": "25000",
"Ethernet50": "25000",
"Ethernet51": "25000"
}
Loaded below Yang Models
['sonic-acl', 'sonic-crm', 'sonic-device_metadata', 'sonic-device_neighbor', 'sonic-extension', 'sonic-flex_counter', 'sonic-interface', 'sonic-loopback-interface', 'sonic-port', 'sonic-portchannel', 'sonic-types', 'sonic-versions', 'sonic-vlan']
Note: Below table(s) have no YANG models:
VRF, VXLAN_EVPN_NVO, BREAKOUT_CFG, FEATURE, VXLAN_TUNNEL, SNMP_COMMUNITY, VXLAN_TUNNEL_MAP, SNMP, REST_SERVER, KDUMP, MGMT_INTERFACE,
libyang[0]: Value "not-provisioned" does not satisfy the constraint "ToRRouter|LeafRouter|SpineChassisFrontendRouter|ChassisBackendRouter|ASIC" (range, length, or pattern). (path: /sonic-device_metadata:sonic-device_metadata/DEVICE_METADATA/localhost/type)
sonic_yang(3):Data Loading Failed:Value "not-provisioned" does not satisfy the constraint "ToRRouter|LeafRouter|SpineChassisFrontendRouter|ChassisBackendRouter|ASIC" (range, length, or pattern).
Data Loading Failed
Value "not-provisioned" does not satisfy the constraint "ToRRouter|LeafRouter|SpineChassisFrontendRouter|ChassisBackendRouter|ASIC" (range, length, or pattern).
ConfigMgmt Class creation failed
Failed to break out Port. Error: Failed to load the config. Error: ConfigMgmtDPB Class creation failed
Once you found you cannot set the breakout successfully, you may check the error message.
In the error message, there's a hint that explains why, please refer to the description in green.
The error message said your type is wrong in DEVICE_METADATA from the config_db.json.
"DEVICE_METADATA": {
"localhost": {
"buffer_model": "traditional",
"default_bgp_status": "up",
"default_pfcwd_status": "disable",
"docker_routing_config_mode": "split",
"hostname": "sonic",
"hwsku": "Accton-AS7326-56X",
"mac": "68:21:5f:dc:3a:c8",
"platform": "x86_64-accton_as7326_56x-r0",
"type": "not-provisioned"
}
},
Solution:
The type "not-provisioned" typically occurs when users do not reset to default after installing SONiC and execute config save.
As a result, the type will be set as "not-provisioned" instead of the supported types, and certain sections in the CONFIG_DB will not be properly defined.
Please follow the instructions in this article to reset to default first, and then re-execute the breakout command.
2. Problem description:
admin@sonic:~$ sudo config interface breakout Ethernet0 "4x10G" -y Running Breakout Mode : 1x100G[40G] Target Breakout Mode : 4x10G Ports to be deleted : { "Ethernet0": "100000" } Ports to be added : { "Ethernet0": "10000", "Ethernet1": "10000", "Ethernet2": "10000", "Ethernet3": "10000" } After running Logic to limit the impact Final list of ports to be deleted : { "Ethernet0": "100000" } Final list of ports to be added : { "Ethernet0": "10000", "Ethernet1": "10000", "Ethernet2": "10000", "Ethernet3": "10000" } sonic_yang(6):Note: Below table(s) have no YANG models: XCVRD_LOG libyang[0]: Leafref "/sonic-mgmt_port:sonic-mgmt_port/sonic-mgmt_port:MGMT_PORT/sonic-mgmt_port:MGMT_PORT_LIST/sonic-mgmt_port:name" of value "eth0" points to a non-existing leaf. (path: /sonic-mgmt_interface:sonic-mgmt_interface/MGMT_INTERFACE/MGMT_INTERFACE_LIST[name='eth0'][ip_prefix='188.188.9.14/16']/name) sonic_yang(3):Data Loading Failed:Leafref "/sonic-mgmt_port:sonic-mgmt_port/sonic-mgmt_port:MGMT_PORT/sonic-mgmt_port:MGMT_PORT_LIST/sonic-mgmt_port:name" of value "eth0" points to a non-existing leaf. Data Loading Failed Leafref "/sonic-mgmt_port:sonic-mgmt_port/sonic-mgmt_port:MGMT_PORT/sonic-mgmt_port:MGMT_PORT_LIST/sonic-mgmt_port:name" of value "eth0" points to a non-existing leaf. ConfigMgmt Class creation failed Failed to break out Port. Error: Failed to load the config. Error: ConfigMgmtDPB Class creation failed
Once you found you cannot set the breakout successfully, you may check the error message.
In the error message, there's a hint that explains why, please refer to the description in green.
The error message said the MGMT_PORT is not defined in the CONFIG_DB.
Solution:
This issue typically occurs when users do not reset to default after installing SONiC.
As a result, certain sections in the CONFIG_DB are not properly defined.
Please follow the instructions in this article to reset to default first, and then re-execute the breakout command.
Comments
1 comment
Hello there this is João from Brasil, is there a way to configure the Edgecore as7316-26xb to do the Dynamic Port Breakout? We really need to use this configuration on our network.
Please sign in to leave a comment.