Sample show spx-mon Output Indicating Issues

The following example shows two PEs are in Ready state with no errors. There is reserved configuration for another PE. Assuming that the user has attached only two PEs, no corrections are necessary.

However, if the user expected three PEs to be attached and in Ready state, some diagnosis is required. The output includes potential CLI guidance that indicates the show spx-mon pe-join command can be used for the appropriate CB port (through which the inactive unit is attached) to diagnose the issue.

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)  <--- Reserved configuration for 1 PE
Number of PEs in Config-mismatch   :  0 
Number of PEs in Image-Mismatch    :  0


CLI: sh spx-mon pe-join <cb-cport> can be used to diagnose pe-join issues   <--- CLI help for additional
                                                                                 diagnosis

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 1 percent busy
300 sec avg 1 percent busy

PE CPU Utilization:           Normal
Spx Interface Utilization:    Very low
Spx Interface Errors:         None
PE User Port Errors:          None

Monitoring Traffic for an SPX Port

Command output for the show spx-mon command includes information on SPX interface utilization and gives an overview of CB-PE and PE-PE traffic. The following example shows that PE SPX port 17/2/1 is using 18% of its traffic capacity in both directions.

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)
Number of PEs in Config-mismatch   :  0
Number of PEs in Image-Mismatch    :  0

CLI: sh spx-mon pe-join <cb-cport> can be used to diagnose pe-join issues

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 1 percent busy
300 sec avg 1 percent busy

PE CPU Utilization:           Normal
PE 17:
Rx utilization Port 17/2/1    18%
Tx utilization Port 17/2/1    18%

Spx Interface Errors:         None
PE User Port Errors:          None

CB Active Controller with SPX LAG Issues

The following example indicates that SPX LAG port 2/1/1 has an incorrect PE database index. Command output displays the warning "Port 2/1/1 doesn't have matching PE db."

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)
Number of PEs in Config-mismatch   :  0
Number of PEs in Image-Mismatch    :  0

CLI: sh spx-mon pe-join <cb-cport> can be used to diagnose pe-join issues

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 7 percent busy
300 sec avg 3 percent busy

Port 2/1/1 doesn't have matching PE db                       <--- Error condition indicated for Port 2/1/1
 CLI sh spx debug lag all can be used for more information   <--- CLI help for additional diagnosis

PE CPU Utilization:           Normal
Spx Interface Utilization:    Very low
Spx Interface Errors:         None
PE User Port Errors:          None
ICX7750-48F Router#

The CLI help indicates that the show spx debug lag all command can be used for additional diagnosis. As shown in the example, debug command output provides the SPX LAG ID (columns 1 and 4), lists the ports in each SPX LAG (column 1), provides a database index number (column 2), and displays a port type for each port in the LAG (column 3). In the example, the database index for ports 1/1/1 and 2/1/1 do not match although they should.

ICX7750-48F Router# show spx debug lag all

spx-lag ID 3072

Port          PortExtDb Index          Port type          lag_id
1/1/1            0                       1                 3072   |<-- Database mismatch
2/1/1            f                       1                 3072   |    on 1/1/1 and 2/1/1

spx-lag ID 3073

Port          PortExtDb Index          Port type          lag_id
1/1/6            6                       1                 3073

spx-lag ID 3074

Port          PortExtDb Index          Port type          lag_id
1/1/2            ff                       1                 3074
1/1/3            ff                       1                 3074  


Using the keyword distributed derives SPX LAG information directly from remote PEs. The following example displays information on SPX LAGs derived directly from PE unit 19.

ICX7750-48F Router# show spx debug lag all dist units 19 

*********************
Response from PE 19:
*********************

spx-lag ID 1
Port          ECID          Port type          lag_id
19/1/1         236           5                   1
19/1/2         236           5                   1

spx-lag ID 2
Port          ECID          Port type          lag_id
19/2/1         235           2                   2

Standby Controller with High CPU Utilization

The following example shows high CPU utilization on the standby controller.

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)
Number of PEs in Config-mismatch   :  0
Number of PEs in Image-Mismatch    :  0

CLI: sh spx-mon pe-join <cb-cport> can be used to dignose pe-join issues

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 1 percent busy
300 sec avg 1 percent busy

Standby CPU Utilization      <--- All measured intervals show high CPU use on the standby controller.
1   sec avg 99 percent busy
5   sec avg 99 percent busy
60  sec avg 255 percent busy
300 sec avg 255 percent busy

PE CPU Utilization:           Normal
Spx Interface Utilization:    Very low
Spx Interface Errors:         None
PE User Port Errors:          None

Standby Controller with Topology Mismatch Issues

The following example shows an inconsistency in the Campus Fabric domain topology stored on the standby controller.

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)
Number of PEs in Config-mismatch   :  0
Number of PEs in Image-Mismatch    :  0

CLI: sh spx-mon pe-join <cb-cport> can be used to diagnose pe-join issues

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 1 percent busy
300 sec avg 1 percent busy

Port 2/1/1 doesn't have matching PE db
 CLI sh spx debug lag all can be used for more information
PE 17: Standby Topology and PE db don't match, sh spx debug lag all can used to get details  <--- Standby 
                                                                                                  topology
                                                                                                  mismatch

Standby CPU Utilization
1     sec avg 99 percent busy
5     sec avg 99 percent busy
60    sec avg 255 percent busy
300   sec avg 255 percent busy

PE CPU Utilization:           Normal
Spx Interface Utilization:    Very low
Spx Interface Errors:         None
PE User Port Errors:          None

PE Join Issue: No Available SPX Port

The show spx command output in the following example indicates that unit 2 is the CB active controller for the Campus Fabric domain. The active controller has only one SPX port, port 2/1/1. Any PE must join through that port. The topology shows a chain of two PE units (PE 17 and PE 23). Any new PE must join after PE 23.

The configuration shows reserve PE configuration for PE 18. If the PE had been expected to join, the show spx-mon command set can be used for troubleshooting.

ICX7750-48F Router# show spx
T=26m14.9: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment                   
1  S ICX7750-48XGF standby cc4e.2438.a580 128 remote  Ready
2  S ICX7750-48XGF active  cc4e.2438.8d80 128 local   Ready
17 S ICX7450-24G   spx-pe  cc4e.246c.e2b8 N/A remote  Ready
18 S ICX7450-48G   spx-pe  0000.0000.0000 N/A reserve 
23 S ICX7450-48G   spx-pe  cc4e.246c.ea50 N/A remote  Ready

    standby      active                                                        
     +---+        +---+                                                        
  2/4| 1 |2/1--2/1| 2 |                                                        
     +---+        +---+                                                        
            +----+        +----+                                               
  2/1/1--2/2| 17 |2/3--2/3| 23 |                                               
            +----+        +----+                                               

The show spx-mon pe-join output for CB SPX port 2/1/1 indicates that the connected PE chain does not have an SPX port from the last PE in the chain (PE 23) that a new PE unit could join.

ICX7750-48F Router# show spx-mon pe-join 2/1/1
Error! Last PE 23 in the chain has no DOWNSTREAM SPX ports
Above error(s) needs to be corrected...

PE Join Issue: No SPX Port in the Up State

The following example shows CB unit 2 as the active controller for the Campus Fabric domain. PE unit 18 has reserve configuration and has not joined the Campus Fabric domain. The problem can be diagnosed with the show spx-mon or the show spx-mon pe-join command.

ICX7750-48F Router# show spx
T=13m4.7: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment                   
1  S ICX7750-48XGF standby cc4e.2438.a580 128 remote  Ready
2  S ICX7750-48XGF active  cc4e.2438.8d80 128 local   Ready
17 S ICX7450-24G   spx-pe  cc4e.246c.e2b8 N/A remote  Ready
18 S ICX7450-48G   spx-pe  0000.0000.0000 N/A reserve 
23 S ICX7450-48G   spx-pe  cc4e.246c.ea50 N/A remote  Ready

    standby      active                                                        
     +---+        +---+                                                        
  2/4| 1 |2/1--2/1| 2 |                                                        
     +---+        +---+                                                        
            +----+        +----+                                               
  2/1/1--2/2| 17 |2/3--2/3| 23 |2/1-                                           
            +----+        +----+    

In this case, the show spx-mon pe-join command is executed on CB port 2/1/1, and the output indicates that the last PE in the chain has no active downstream SPX port.

ICX7750-48F Router# show spx-mon pe-join 2/1/1
Error! Last PE 23 in the chain has no DOWNSTREAM SPX ports in UP state
Above error(s) needs to be corrected...

When the show interface brief command is executed for the problem port (23/2/1), output confirms that the link is down. Once the link is restored, an SPX connection can be created or restored with PE unit 18.

ICX7750-48F Router# show interface brief ethernet 23/2/1

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
23/2/1     Down    None    None None  None  No  1    0   cc4e.246c.ea81         

PE Join Issue: PE Ring with ACL Issues

The show spx-mon command can be used to diagnose issues related to ACLs used to create SPX rings.

ICX7750-48F Router# show spx-mon
spx-mon is enabled

Total 2 PE(s) attached
Number of PEs in Ready State       :  2 (Id(s): 17, 23)
Number of PEs in Reserve           :  1 (Id(s): 18)
Number of PEs in Config-mismatch   :  0
Number of PEs in Image-Mismatch    :  0

CLI: sh spx-mon pe-join <cb-cport> can be used to diagnose pe-join issues

Active CPU Utilization
1   sec avg 1 percent busy
5   sec avg 1 percent busy
60  sec avg 7 percent busy
300 sec avg 3 percent busy

Port 17/2/2 has incorrect acl config  <--- ACL issues indicated on port.

PE CPU Utilization:           Normal
Spx Interface Utilization:    Very low
Spx Interface Errors:         None
PE User Port Errors:          None
ICX7750-48F Router#

In the example, PE port 17/2/2 is an SPX uplink port with an incorrect ACL configuration.