Troubleshoot Nexus 7000 with Message Transaction Services
Available Languages
Introduction
This document describes the Message and Transition Services (MTS) functionality used to troubleshoot Nexus 7000 platforms.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
Services (also known as Processes) in the NeXus Operating System (NX-OS) communicate with each other by MTS.
If the target service is physically on a different module, Inter Node Communication needs to be used. The communication to the other module is delivered by Advanced Inter Process Communication (AIPC) over the Ethernet Out-of-Band Channel (EOBC) links. The EOBC links are on the MidPlane of the Nexus 7000 chassis.
MTS Provides:
- Messages and high availability (HA) infrastructure and Application Program Interfaces (APIs)
- Buffer management
- Message delivery
AIPC Provides:
- Reliable transport across EOBC
- Delivery of ACK-based retransmission
- Fragmentation and reassembly
MTS mainly consists of destination/source Node/SAP and Opcode.
- Node - an identifier of the physical module and VDC.
- SAP - an identifier of each service. (Each service could have multiple Service Access Points (SAPs) for multiple functions.)
- Opcode - a data type which services use to communicate with other services.
Terminology
MTS : Message and Transaction Services
SAP : Service Access Point
AIPC : Advanced Inter Process Communication / Andiamo IPC
EOBC : Ethernet Out-of-Band Channel
NX-OS : NeXus Operating System
MTS Formats
The MTS destination is represented by SAP (a service) and Node (the module on which the SAP runs).
-
SAP
In most cases, the static SAP is used to track the MTS event.
SAP can be checked in general or per module (supervisor or linecard).
Note: sup stands for supervisor and lc for linecard.
Get SAP running on any module.
`show system internal sysmgr service ~~~~`
n7ka# sh system internal sysmgr service all
Name UUID PID SAP state Start count Tag Plugin ID
---------------- ---------- ------ ----- --------- ----------- ---------------- -----------
aaa 0x000000B5 6942 111 s0009 1 N/A 0
cert_enroll 0x0000012B 6941 169 s0009 1 N/A 0
ExceptionLog 0x00000050 7267 92 s0009 1 N/A 0
`show system internal mts ~~~~~`
n7ka# sh system internal mts sup sap 111 description
Below shows sap on default-VDC, to show saps on non-default VDC, run
show system internal mts node sup-<vnode-id> sap ...
AAA Daemon
Get SAP running on specific module only.
`Attach to the module`
n7ka# attach module 4
Attaching to module 4 ...
To exit type 'exit', to abort type '$.'
Last login: Tue Nov 7 15:42:35 PST 2023 from 127.1.1.2 on pts/0
`show system internal sysmgr service ~~~~`
n7ka# show system internal sysmgr service all
Name UUID PID SAP state Start count Tag Plugin ID
---------------- ---------- ------ ----- --------- ----------- ---------------- -----------
aclqo 0x0000016E 1301 190 s0009 1 N/A 0
amm 0x00000260 1130 895 s0009 1 N/A 0
bfdc 0x000002C7 1110 1008 s0009 1 N/A 0
`show system internal mts ~~~~~`
module-4# show system internal mts lc sap 190 description
Aclqos SAP
- Opcode Registry
Sometimes MTS is sent to destination SAP 0. As a result, the MTS is sent to multiple SAPs that are registered to the SAP registry (opcode registry).
As an example, Opc 8182 is registered by SAP 175 and 378, confirmed by show system internal mts sup registry persistent command. So this MTS is delivered to both SAP175 and SAP 378.
n7ka# show system internal ethpm event-history msgs
51) Event:E_MTS_RX, length:60, at 36968 usecs after Thu Sep 18 14:42:15 2014
[NOT] Opc:MTS_OPC_LINK_EVENT_DOWN(8182), Id:0X034960A1, Ret:SUCCESS
Src:0x00000102/181, Dst:0x00000609/0, Flags:None
HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:82
Payload:
0x0000: 00 00 00 07 00 11 00 21 00 00 00 04 00 12 00 04
n7ka# show system internal mts sup registry persistent | i 8182
MTS_OPC_LINK_EVENT_DOWN(8182): 175, 378
- Node ID
The Node ID is a 16-bit ID.
+ Slot ID - module slot ID. Starts with 1
+ Vnode ID - virtual node ID. VDC ID starts with 0
+ Lnode ID - logical node ID. SUP:1 , LC:2
Example: 0x805 = 0x08 + 0b0000_0101 = 0x08 + 0x1 + 0x1 (slot + Vnode + Lnode)
=> SUP module VDC 2 in slot 8
- MTS Address
The Node ID is a component of the MTS address. This MTS address indicates the module to which this MTS is to be delivered.
The MTS address (Node ID) is the left part of the output shown next.
Src:0x00000102/181, Dst:0x00000609/0, Flags:None
If MTS is unicast, the MTS address is the same as the Node ID.
Track Process Communication with MTS
ages
You can see the MTS communication history by use of event-history messages. Almost all services have this event history message capability.
This CLI is the MTS event history for eth_port_channel (ethpc).
n7ka# show port-channel internal event-history msgs
12) Event:E_MTS_RX, length:60, at 15586 usecs after Thu Sep 18 13:13:57 2014
[REQ] Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444), Id:0X00323B1E, Ret:SUCCESS
Src:0x00000601/175, Dst:0x00000601/378, Flags:None
HA_SEQNO:0X00000000, RRtoken:0x00323B1E, Sync:UNKNOWN, Payloadsize:26
Payload:
0x0000: 00 00 00 02 00 04 00 02 00 01 00 05 00 0c 00 00
The fields listed next provide additional details with regard to their purpose in the command:
Event:E_MTS_RX - indicates this MTS was received by this service, ethpc in this case. If it is E_MTS_TX, then ethpc is the sender of this MTS.
Src:0x00000601/175 - MTS_addr/SAP, represents the sender of this MTS.
Dst:0x00000601/378 - MTS_addr/SAP, represents the receiver of this MTS.
Id:0X00323B1E - MTS ID, where both the sender and receiver have the same ID for the same MTS.
Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444) - Indicates what this event type is. In other words, which opcode this MTS delivers, where 61444 is the opcode number.
Example of an MTS Event
This MTS event example is for a link down event.
n7ka# 2014 Sep 18 14:42:15 n7ka %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
As a first step, check the ethpm history because it is the comprehensive process for all I/F-related things.
n7ka# sh system internal ethpm event-history msgs
51) Event:E_MTS_RX, length:60, at 36968 usecs after Thu Sep 18 14:42:15 2014
[NOT] Opc:MTS_OPC_LINK_EVENT_DOWN(8182), Id:0X034960A1, Ret:SUCCESS
Src:0x00000102/181, Dst:0x00000609/0, Flags:None
HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:82
Payload:
0x0000: 00 00 00 07 00 11 00 21 00 00 00 04 00 12 00 04
The result shown indicates the ETHPM received the link down event from Src:0x00000102/181. This MTS address indicates the SAP 181 is on VDC 1 LC on slot 1.
Use the CLI listed next to determine what SAP 181 is on slot 1.
module-1# attach module 1
module-1# show system internal mts lc sap 181 description
Port_client SAP
Use the CLI listed next to check the port_client MTS history on slot 1.
module-1# show system internal port-client event-history msgs
49) Event:E_MTS_TX, length:60, at 298743 usecs after Thu Sep 18 14:42:14 2014
[NOT] Opc:MTS_OPC_LINK_EVENT_DOWN(8182), Id:0X034960A1, Ret:SUCCESS
Src:0x00000102/181, Dst:0x00000609/0, Flags:None
HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:82
Payload:
0x0000: 00 00 00 07 00 11 00 21 00 00 00 04 00 12 00 04
50) Event:E_MTS_RX, length:60, at 298329 usecs after Thu Sep 18 14:42:14 2014
[NOT] Opc:MTS_OPC_LC_LINK_DOWN(8185), Id:0X0349609F, Ret:SUCCESS
Src:0x00000102/536, Dst:0x00000102/0, Flags:None
HA_SEQNO:0X00000000, RRtoken:0x00000000, Sync:UNKNOWN, Payloadsize:17
Payload:
0x0000: 00 00 00 02 40 e5 00 07 02 54 1a 70 b6 00 04 84
From Id:0X034960A1, we can tell the 49) event is the one of interest. The Port_Client sent the MTS as expected. This must be invoked by the previous MTS event on Port_Client, which is 50). This MTS was sent by Src:0x00000102/536.
Check SAP 536.
module-1# sh system internal mts lc sap 536 de
Naxos FPGA
Now we know NAXOS FPGA is the source of this whole event. Use the CLI shown next to find that the local_fault is the root cause of the link down.
module-1# sh hardware internal naxos event-history port 3
15) Event E_NAXOS_ISR_DATA length:69, at 170763 usecs after Thu Sep 18 14:42:14 2014
TO NAXOS_PORT_STATUS_LINK_DOWN: reg_val 0x14
Status:SUCCESS (0x0)
16) Event E_NAXOS_ISR_DATA length:50, at 170619 usecs after Thu Sep 18 14:42:14 2014
LOCAL_FAULT: reg_val 0x14
Status:SUCCESS (0x0)
Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
09-Nov-2023 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)