[Edgecore SONiC] SONiC RoCEv1 Tuning Tutorial Follow

Reference model:

• Switch model name: AS7726-32X
  

Restriction:

Causing by chipset limitation, PFC only can apply to 1 or 2 queues.

Topology:

Tuning Procedure:

Step 1. Measure bandwidth performance and latency of each traffic with Perftest tool. In this topology example, measure bandwidth performance of CE0 → CE1 and CE2 → CE1. From this measurement, you may get the big picture of current real performance and how to tune it later. Please check your traffic PFC queue number. It's necessary for configuring port QOS setting on next step.

Step 2. Configure TC to priority group map setting. Set up TC-to-Priority group map by adding below group map configuration into config_db.json. By default, we will use PTP to classify traffic to queue.

"TC_TO_PRIORITY_GROUP_MAP": {                         
    "AZURE": {                                            → group map profile name
        "0": "0",
        "1": "0",
        "2": "0",
        "3": "3",
        "4": "4",
        "5": "0",
        "6": "0",
        "7": "7"
    }
},

Step 3. Enable PFC on queue. In this example, we will enable queue 3 of PFC on Ethernet0, Ethernet4, Ethernet20 respectively.

"PORT_QOS_MAP": {
    "Ethernet0": {
        "pfc_enable": "3",                                             → Enable PFC 3
        "tc_to_pg_map": "[TC_TO_PRIORITY_GROUP_MAP|AZURE]"             → Bind TC-to-Priority group map profile to Ethernet0
    },
    "Ethernet4": {
        "pfc_enable": "3",                                             → Enable PFC 3
        "tc_to_pg_map": "[TC_TO_PRIORITY_GROUP_MAP|AZURE]"             → Bind TC-to-Priority group map profile to Ethernet4
    },
    "Ethernet20": {
        "pfc_enable": "3",                                             → Enable PFC 3
        "tc_to_pg_map": "[TC_TO_PRIORITY_GROUP_MAP|AZURE]"             → Bind TC-to-Priority group map profile to Ethernet20
    }
}

Step 4. Buffer pool for each model as attached on Appendix 1 below. If you can't find it, please request this buffer pool from us by submit a ticket through support system https://support.edge-core.com/hc/en-us/requests/new. Add this buffer pool setting to config_db.json file. In this example, AS7726-32X buffer pool setting as shown below. 

"BUFFER_POOL": {                                     → This is global buffer pool setting which is used for all ports and queues
    "ingress_lossless_pool": {
        "mode": "static",
        "size": "10875072",
        "type": "ingress",
        "xoff": "4194112"
    }
},

Step 5. Configure and tune buffer setting profile. Configuration below is based on our best tuning configuration of AS7726-32X. The details of how to calculate this configuration are on below.

"BUFFER_PROFILE": {                                                                                
    "pg_300m_profile": {                                                 → This buffer setting profile will be bound to certain or/and all ports and queues (Step 4)
        "pool": "[BUFFER_POOL|ingress_lossless_pool]",
         "size": "69632",                                                                      
         "static_th": "0",
         "xoff": "13056",
         "xon": "512"
    }
},

The picture above is buffer design of one ingress-queue on a port. Xon/ Xoff uses headroom. When the profile.size is full, it will use headroom space for packet buffering. Once profile.xoff buffer is full (buffer line touch the upper limit of profile.xoff in green block), it sends 802.1Qbb pause frame to partner and requests to stop transmission. And when the buffer is less then profile.xon, it sends request to partner to resume the traffic transmission. AS7726-32X has 32 ports. In condition of 1:1 traffic, it will reach the most congested traffic with 16 ports traffic transmission to single port.

In order to achieve the best bandwitdh performance along with low latency and lossless packet, we have to calculate some buffer parameters. 

1. MTU_cell = ⌈MTU/ 256⌉ * 256                     → Cell size : 256 (This cell size is for AS7726-32X. Please check appendix 2 below to match cell size of your switch model.). Rounding up the division result of RoCE MTU size by 256. 
2. profile.size = 16 * MTU_cell                        → Tuning profile.size to maximum size. In this example, 16 is the max ports number which would let AS7726-32X reach the most congested traffic.
3. profile.xoff = 3 * MTU_cell                          → Tuning profile.xoff size to best performance. In this example, 3 is the best start constant. 
4. profile.xon = 2 * MTU_cell                          

Please check the MTU size of your NIC card. In this example, we use the maximum size of RoCE's packets is 4174 bytes. Using formula above, calculate the parameter setting for this example,

MTU_cell = ⌈4174 / 256⌉ *256 = 4352                                         

profile.size = 4352 * 16 = 69632

profile.xoff = 4352 * 3 = 13056

profile.xon = 256 * 2 = 512

Tips in tuning : if there is packet drop, try to tune profile size. If bandwidth result is low, try to tune profile.xoff larger.

Step 6. Bind buffer profile (Step 4) to port and queue number then check the result. Compare performance and latency result after tuning and before tuning (Step 1). Tune again by follow tips in tuning on step 4 till you get a best result which means the bandwidth is better than before but doesn't increase latency.

"BUFFER_PG": {
"Ethernet20|3": {                                                  → Port|PFC Queue number
"profile": "[BUFFER_PROFILE|pg_300m_profile]"                      → Bind pg_300m_profile buffer profile to Ethernet20 Queue3
},
"Ethernet4|3": {
"profile": "[BUFFER_PROFILE|pg_300m_profile]"
},
"Ethernet0|3": {
"profile": "[BUFFER_PROFILE|pg_300m_profile]"
}
},

Reference: 

Test Result Comparison Table:

Note :

1. QP : Queue Pair
2. Tuning two RoCE QPs with lossless packet can achieve bandwidth with almost approaching the average bandwidth of two single RoCE QPs bandwidth and latency. 
3. Bandwidth and latency results are average of 5 times test.

Appendix 1: 

Buffer Pool

"BUFFER_POOL": {
       "ingress_lossless_pool": {
       "size": "8361216",
       "type": "ingress",
       "mode": "static",
       "xoff": "17664"
         }
},

"BUFFER_POOL": {
        "ingress_lossless_pool": {
            "size": "32667392",
            "type": "ingress",
            "mode": "static",
            "xoff": "19712"
        }
 },

"BUFFER_POOL": {                                                                    
    "ingress_lossless_pool": {
        "mode": "dynamic",
        "size": "12766208",
        "type": "ingress",
        "xoff": "196608"
    }
},

"BUFFER_POOL": {                                                                    
    "ingress_lossless_pool": {
        "mode": "dynamic",
        "size": "12766208",
        "type": "ingress",
        "xoff": "196608"
    }
},

"BUFFER_POOL": {                                                                    
    "ingress_lossless_pool": {
        "mode": "dynamic",
        "size": "12766208",
        "type": "ingress",
        "xoff": "4625920"
    }
},

"BUFFER_POOL": {                                                                    
    "ingress_lossless_pool": {
        "mode": "dynamic",
        "size": "59001152",
        "type": "ingress",
        "xoff": "7428992"
    }
},

"BUFFER_POOL": {
     "ingress_lossless_pool": {
        "size": "65012316",
        "type": "ingress",
        "mode": "static",
        "xoff": "0"
    }
},

Appendix 2: 

Cell Size Table