[Wedge100BF-series] QSFP can't link-up Follow

Purpose:

This is the troubleshooting guide for the QSFP on Wedge100BF-series.

Model:

Wedge100BF-32X/65X/32QS

Procedure:

Step 0. Make sure we have the prior knowledge about port LED behavior

The LED select button switch the LED present "TOP", port #1, #3, #5,..., or "BOT", #2, #4, #6,...

Please refer to [Wedge100BF] LED behavior article for more detail.

Caution: The default port speed is 100G, if want to link up at other speeds required down speed first, otherwise the port won't link up.

Step 1. Specify the problem is on QSFP or Port

Use Cross Comparison tests to judge which has the problem, QSFP or Port.

Prepare several same model QSFPs to do the Cross Comparison tests.

Loop the port directly to do the Cross Comparison tests.

According to the result, decide to run the RMA process with only specific QSFP or switch.

Step 2. Test front port LED, dump QSFP info and port status

Method 1. Login Diag to test

A). Login Diag mode to ensure LED work well

To login Diag, please refer to the article: Install and start up Accton Diag

Choose Diag item 155 to check if the front port LED change color every time after press Enter or not.

Caution: It'll take few minutes at first time, thus do not use "ctrl + c" to shut down the current operation.
Please be patient till you see "Press Enter to change color".

*********************************************************
(...omitted)
155 Test front port LED(OFF->RED->GREEN->BLUE)
(...omitted)
*********************************************************
Please enter your choice (0 to quit): 155
(...omitted)
Press Enter to change color
(...omitted)

Item (155) test result: PASSED

Please choose item 179 Unload Barefoot Diagnostic Service, after test item 155.

Otherwise, you'll see the message as below while testing other item.

*********************************************************
Please enter your choice (0 to quit): 199
Barefoot diagnostic service state:
 0x10: Disable
 0x20: Enable
 0x40: Already execute and Disable
The current state (0x20) of Barefoot diagnostic service is not matched the mask (0x50) for diag item 199
Please enter your choice (0 to quit):

B). Login Diag mode to dump the log to us.

Choose Diag item 104 to show QSFP eeprom.

Please insert QSFP while doing this test.

*********************************************************
(...omitted)
104 Show QSFP EEPROM
(...omitted)
*********************************************************
Please enter your choice (0 to quit): 104
[argc] 2, [optind] 2, argv[2]=(null), optarg=(null)
Driver not running
=======================
READ QSFP port 1
=======================
/root/DIAG-V0.36/usr/local/accton/tools/cp2112_util_upper write 0xe0 0 0x02
/root/DIAG-V0.36/usr/local/accton/tools/cp2112_util_upper addr_read 0xa0 0 256
Writing 2 bytes to address 0xe0:
0000: 00 02
Read 256 bytes at device 0xa0 address 0x00:
0000: 11 05 04 00 00 00 00 00 00 00 00 00 00 00 00 00
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0060: 00 00 00 00 ff ff ff ff ff ff ff 00 00 00 08 00
0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0080: 11 00 23 80 00 00 00 00 00 00 00 05 ff 00 00 00
0090: 00 00 03 a0 41 6d 70 68 65 6e 6f 6c 20 20 20 20
00a0: 20 20 20 20 1f 78 a7 14 4e 44 41 41 46 46 2d 30
00b0: 30 30 33 20 20 20 20 20 41 20 07 0b 0d 13 55 f9
00c0: 0b 00 00 00 41 50 46 31 35 31 37 30 30 33 32 35
00d0: 4e 30 20 20 31 35 30 35 30 36 20 20 00 00 67 40
00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
Writing 2 bytes to address 0xe0:
0000: 00 00
=======================
READ QSFP port 2
=======================
(...omitted)

C). Use "swutil ps" to check interfaces status

Please connect Port 1 and 2 directly with 100G QSFP.

If you exit from Diag Utility means you had already enter these commands once and can skip to next part.

root@bfn-switch:~# cd DIAG-V0.36/
root@bfn-switch:~/DIAG-V0.36# sudo su
root@bfn-switch:~/DIAG-V0.36# source sysenv
root@bfn-switch:~/DIAG-V0.36# source sysenv2
root@bfn-switch:~/DIAG-V0.36# export ACC_MODEL_SKU=MODEL_1U    <---for 32X/32QS
root@bfn-switch:~/DIAG-V0.36# export ACC_MODEL_SKU=MODEL_2U    <---for 65X
root@bfn-switch:~/DIAG-V0.36# source ./bf_diag_start.sh
[ 124.148979] bf probe ok

Start up the driver.

root@bfn-switch:~/DIAG-V0.36# swutil -C -S --install-path=$BF_DIAG_INSTALL_DIR
(...omitted)

********************************************
 * WARNING: Authorised Access Only *
 ********************************************

bfshell> root@bfn-switch:~/DIAG-V0.36#

Note: It takes about 2 minute and stays at "bfshell>".

root@bfn-switch:~# swutil ps
Trying to connect switch driver...
Switch driver Connected.
Command sent to Switch driver.

(...omitted)

Client=> cat /tmp/mav_diag_result
Port link lns speed dlpx a-neg pause fec  txPre txMain txPost mtu   loop Intf   Ext-intf
------------------------------------------------------------------------------------------
1/0  up   4   100G  FD   Done  None  RS   2     0      0      10240 No   None   CR4_100G
2/0  up   4   100G  FD   Done  None  RS   4     0      0      10240 No   None   CR4_100G
3/0  down 4   100G  FD   None  None  RS   6     0      0      10240 No   None   Other
4/0  down 4   100G  FD   None  None  RS   6     0      0      10240 No   None   Other
5/0  down 4   100G  FD   None  None  RS   6     0      0      10240 No   None   Other
6/0  down 4   100G  FD   None  None  RS   6     0      0      10240 No   None   Other
(...omitted)

Method 2. Login Barefoot SDE to test 

Consider some NOS erased Diag after installation, this method could use SDE for test.

a). Test front ports LEDs can light at specific color

bfshell> ucli
Starting UCLI from bf-shell
Cannot read termcap database;
using dumb terminal settings.
bf-sde> 

bf-sde> bf_pltfm
bf-sde.bf_pltfm> led
bf-sde.bf_pltfm.led>
led-set  led-set <dev> <port> <chn> 0x<col> <blink>

<dev>=0
<port>=port No.
<chn>=0 LED1, =1 LED2, =2 LED3, =3 LED4

<col> color
0x00 Off
0x01 Blue
0x02 Green
0x03 Light Blue
0x04 Red
0x05 Purple
0x06 Amber
0x07 Light Purple
<blink>=0 keeps light on, =1 blinking

For example

bf-sde.bf_pltfm.led> led-set 0 1 0 0x00 0
bf-sde.bf_pltfm.led> led-set 0 1 1 0x01 0
bf-sde.bf_pltfm.led> led-set 0 1 2 0x02 0
bf-sde.bf_pltfm.led> led-set 0 1 3 0x03 0
bf-sde.bf_pltfm.led> led-set 0 3 0 0x04 0
bf-sde.bf_pltfm.led> led-set 0 3 1 0x05 0
bf-sde.bf_pltfm.led> led-set 0 3 2 0x06 0
bf-sde.bf_pltfm.led> led-set 0 3 3 0x07 0

b). Dump the QSFP EEPROM

bf-sde> bf_pltfm
bf-sde.bf_pltfm> qsfp
bf-sde.bf_pltfm.qsfp>
qsfp-read  qsfp-read <port> <page> <num-bytes> <start-offset>

bf-sde.bf_pltfm.qsfp> qsfp-read 1/0 0 255 0
QSFP:1 page 0 offset 0x0 read nbytes:255

Page   0 : offset   0 : 11 07 00 ff 00 ff 00 00 00 55 55 11 11 55 55 00 : .........UU..UU.
Page   0 : offset  16 : 00 00 00 00 00 00 17 00 00 00 7f d0 00 00 00 00 : ................
Page   0 : offset  32 : 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00 : ................
Page   0 : offset  48 : 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00 : ................
Page   0 : offset  64 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
Page   0 : offset  80 : 00 00 00 00 00 00 0f 00 00 00 00 00 00 01 00 00 : ................
Page   0 : offset  96 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
Page   0 : offset 112 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
Page   0 : offset 128 : 11 c0 23 80 00 00 00 00 00 00 00 05 ff 00 00 00 : ..#.............
Page   0 : offset 144 : 00 00 03 00 45 64 67 65 63 6f 72 65 20 20 20 20 : ....Edgecore
Page   0 : offset 160 : 20 20 20 20 00 70 72 cf 45 54 37 34 30 32 2d 31 : .pr.ET7402-1
Page   0 : offset 176 : 30 30 41 4f 43 2d 33 4d 30 31 00 00 00 00 46 95 : 00AOC-3M01....F.
Page   0 : offset 192 : 01 07 ff de 4a 31 31 38 34 30 30 30 34 30 35 31 : ....J11840004051
Page   0 : offset 208 : 20 20 20 20 31 35 30 37 31 33 20 20 00 00 68 b0 : 150713 ..h.
Page   0 : offset 224 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
Page   0 : offset 240 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    : ...............

c). Configure front ports in SDE

bf-sde> pm
bf-sde.pm> 
port-add <port_str> <speed (1G, 10G, 25G, 40G, 40G_NB, 50G, 100G,200G, 400G 40G_NON_BREAKABLE)> <fec (NONE, FC, RS)>
port-del <port_str>
port-enb <port_str>
port-dis <port_str>
show -a -p <port_str> [-d]
an-set <port_str> <0->default,1->enable,2->disable>

For <port_str>, -/- indicate all ports.

For example

bf-sde> pm
bf-sde.pm> port-add -/- 100G RS
bf-sde.pm> an-set -/- 2
bf-sde.pm> port-enb -/-

Use "show" to check interfaces status

bf-sde.pm> show
-----+----+---+----+------+----+---+---+---+----------------+----------------+-
PORT |MAC |D_P|P/PT|SPEED |FEC |RDY|ADM|OPR|FRAMES RX       |FRAMES TX       |E
-----+----+---+----+------+----+---+---+---+----------------+----------------+-
1/0  |31/0|188|1/60| 100G | RS |YES|ENB|UP |               0|               0|
2/0  |30/0|184|1/56| 100G | RS |YES|ENB|UP |               0|               0|
3/0  |29/0|180|1/52| 100G | RS |YES|ENB|UP |               0|               0|
4/0  |28/0|176|1/48| 100G | RS |YES|ENB|UP |               0|               0|
5/0  |27/0|172|1/44| 100G | RS |NO |ENB|DWN|               0|               0|
6/0  |26/0|168|1/40| 100G | RS |NO |ENB|DWN|               0|               0|
7/0  |25/0|164|1/36| 100G | RS |NO |ENB|DWN|               0|               0|
8/0  |24/0|160|1/32| 100G | RS |NO |ENB|DWN|               0|               0|