- Preface
- New and Changed
- Troubleshooting Overview
- Troubleshooting APIC Crash Scenarios
- Recovering Cisco APIC Passwords and Accessing Special Logins
- Cisco APIC Troubleshooting Operations
- Using the Cisco APIC Troubleshooting Tools
- Troubleshooting Switch Discovery
- Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI
- Decommissioning and Recommissioning Switches
- Troubleshooting Steps for Endpoint Connectivity Problems
- Troubleshooting EVPN Type-2 Route Advertisement
- Performing a Rebuild of the Fabric
- Verifying IP-Based EPG Configurations
- Recovering a Disconnected Leaf
- Troubleshooting a Loopback Failure
- Determining Why a PIM Interface Was Not Created
- Confirming the Port Security Installation
- Troubleshooting QoS Policies
- Determining the Supported SSL Ciphers
- Removing Unwanted _ui_ Objects
- Troubleshooting Multipod and Multi-Site Issues
- acidiag Command
- Configuring Export Policies for Troubleshooting
- Finding the Switch Inventory
- Cisco APIC SSD Replacement
- Expected Output Errors
- Viewing CRC Error Counters
- Index
acidiag Command
To troubleshoot operations on the Cisco APIC, use the acidiag command.
![]() Caution |
This command is not intended for every day operation of ACI. Running all forms of the command can be very disruptive and cause major issues in your network if not used properly. Make sure you understand the full effect on your fabric before running them. |
Cluster Commands
Syntax Description
|
Option |
Function |
||
|---|---|---|---|
| avread | Displays
APICs
within the cluster. The
avread output includes:
|
||
| bootcurr | On the next boot, the APIC system will boot the current APIC image in the Linux partition. This option is not expected to normally be used. | ||
| bootother | On the next boot, the APIC system will boot the previous APIC image in the Linux partition. This option is not expected to normally be used. | ||
| bond0test | Disruptive test of the APIC connection to the leaf. This is used for internal Cisco testing purposes only and outside of that could cause issues with the APIC connection to the fabric. | ||
| fnvread | Displays the address and state of switch nodes registered with the fabric. | ||
| fnvreadex | Displays additional information for switch nodes registered with the fabric. | ||
| linkflap | Brings down and back up a specified APIC interface. | ||
| preservelogs | APIC will archive current logs. During a normal reboot this automatically occurs. This option can be used prior to a hard reboot. | ||
| run | Two available options are iptables-list and lldptool. The iptables-list is used to display the Linux iptables, which are controlled by the mgmt Tenant contracts. lldptool is used to display lldp information which is sent or received by the APIC. | ||
| rvread | Summarizes the data layer state. The output shows a summary of the data layer state for each service. The shard view shows replicas in ascending order. | ||
| acidiag rvread service | Displays the data layer
state for a service on all shards across all replicas.
|
||
| acidiag rvread service shard | Displays the data layer
state for a service on a specific shard across all replicas.
|
||
| acidiag rvread service shard replica | Displays the data layer
state for a service on a specific shard and replica.
|
||
| validateimage | Prior to loading an image into the firmware repository, the image can be validated. Note that this function runs as a normal part of the process of the image being added into the repository. | ||
| validateenginxconf | Validates the generated nginx configuration file on APIC to ensure nginx can start with that configuration file. This is meant for debug use, in cases where the nginx webserver is not running on APIC. |
Service IDs
The service IDs listed in the table below are also visible when entering the man acidiag command.
|
Service |
ID |
|---|---|
|
cliD |
1 |
|
controller |
2 |
|
eventmgr |
3 |
|
extXMLApi |
4 |
|
policyelem |
5 |
|
policymgr |
6 |
|
reader |
7 |
|
ae |
8 |
|
topomgr |
9 |
|
observer |
10 |
|
dbgr |
11 |
|
observerelem |
12 |
|
dbgrelem |
13 |
|
vmmmgr |
14 |
|
nxosmock |
15 |
|
bootmgr |
16 |
|
appliancedirector |
17 |
|
adrelay |
18 |
|
ospaagent |
19 |
|
vleafelem |
20 |
|
dhcpd |
21 |
|
scripthandler |
22 |
|
idmgr |
23 |
|
ospaelem |
24 |
|
osh |
25 |
|
opflexagent |
26 |
|
opflexelem |
27 |
|
confelem |
28 |
|
vtap |
29 |
|
snmpd |
30 |
|
opflexp |
31 |
|
analytics |
32 |
|
policydist |
33 |
|
plghandler |
34 |
|
domainmgr |
35 |
|
licensemgr |
36 |
|
N/A |
37 |
|
platformmgr |
38 |
|
edmgr |
39 |
|
State |
ID |
|---|---|
| COMATOSE | 0 |
| NEWLY_BORN | 1 |
| UNKNOWN | 2 |
| DATA_LAYER_DIVERGED | 11 |
| DATA_LAYER_DEGRADED_LEADERSHIP | 12 |
| DATA_LAYER_ENTIRELY_DIVERGED | 111 |
| DATA_LAYER_PARTIALLY_DIVERGED | 112 |
| DATA_LAYER_ENTIRELY_DEGRADED_LEADERSHIP | 121 |
| DATA_LAYER_PARTIALLY_DEGRADED_LEADERSHIP | 122 |
| FULLY_FIT | 255 |
System Keywords
Syntax Description
|
Option |
Function |
|---|---|
| -c | Specifies a clean install |
| -u | Specifies a URL for the APIC image. |
| imageurl | Specifies an APIC image. |
| installer | Installs a new image on the APIC, -c for clean install |
| mgmt | Specifies all services on the APIC. |
| reboot | Reboots the APIC. |
| restart | Restarts services on an APIC. |
| start | Starts services on an APIC. |
| stop | Stops services on an APIC. |
| touch [clean | setup] | Resets the
APIC
configuration.
|
| verifyapic | Displays the APIC software version. |
| xinetd | Specifies xinetd (extended internet daemon) service, which controls the ssh and telnet daemons. |
Diagnostic Keywords
Syntax Description
|
Option |
Function |
|---|---|
| crashsuspecttracker | Tracks states of a service or data subset that indicate a crash. |
| dbgtoken | Generates a token used to generate a root password. This is to be used as directed while working with the TAC as needed. |
| version | Displays the APIC ISO software version. |
Examples
The following examples show how to use the acidiag command:
apic1# acidiag version 2.2.1o
apic1# acidiag verifyapic
openssl_check: certificate details
subject= CN=ABC12345678,serialNumber=PID:APIC-SERVER-L1 SN:ABC12345678
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Sep 28 17:17:42 2016 GMT
notAfter=Sep 28 17:27:42 2026 GMT
openssl_check: passed
ssh_check: passed
all_checks: passed
apic1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 ROUTABLE IP ADDRESS=0.0.0.0 CHASSIS_ID=1009f750-adab-11e9-a044-8dbd212cd556
Cluster of 7 lm(t):1(2019-08-08T01:02:17.961-07:00) appliances (out of targeted 7 lm(t):7(2019-08-08T03:50:57.240-07:00)) with FABRIC_DOMAIN name=ACI Fabric1 set to version=apic-4.2(0.235j) lm(t):1(2019-08-17T01:09:16.413-07:00); discoveryMode=PERMISSIVE lm(t):0(1969-12-31T17:00:00.007-07:00); drrMode=OFF lm(t):0(1969-12-31T17:00:00.007-07:00); kafkaMode=OFF lm(t):0(1969-12-31T17:00:00.007-07:00)
appliance id=1 address=10.0.0.1 lm(t):1(2019-08-08T01:02:08.544-07:00) tep address=10.0.0.0/16 lm(t):1(2019-08-08T01:02:08.544-07:00) routable address=0.0.0.0 lm(t):1(zeroTime) oob address=172.23.96.10/21 lm(t):1(2019-08-08T01:02:18.218-07:00) version=4.2(0.235j) lm(t):1(2019-08-15T15:22:00.158-07:00) chassisId=1009f750-adab-11e9-a044-8dbd212cd556 lm(t):1(2019-08-15T15:22:00.158-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X7F lm(t):1(2019-08-17T01:13:46.997-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-08T01:02:18.228-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-08T01:02:18.228-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-08T01:02:18.228-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-08T01:02:18.228-07:00) cntrlSbst=(APPROVED, FCH1748V0SZ) lm(t):1(2019-08-15T15:22:00.158-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2019-08-08T01:02:08.544-07:00) commissioned=YES lm(t):1(zeroTime) registered=YES lm(t):1(2019-08-08T01:02:08.544-07:00) standby=NO lm(t):1(2019-08-08T01:02:08.544-07:00) DRR=NO lm(t):0(zeroTime) apicX=NO lm(t):1(2019-08-08T01:02:08.544-07:00) virtual=NO lm(t):1(2019-08-08T01:02:08.544-07:00) active=YES(2019-08-08T01:02:08.544-07:00) health=(applnc:255 lm(t):1(2019-08-17T01:39:26.296-07:00) svc's)
appliance id=2 address=10.0.0.2 lm(t):7(2019-08-08T03:50:55.470-07:00) tep address=10.0.0.0/16 lm(t):2(2019-07-23T17:51:38.997-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.23.96.11/21 lm(t):1(2019-08-18T23:14:28.720-07:00) version=4.2(0.235j) lm(t):2(2019-08-15T15:22:00.300-07:00) chassisId=694e6a98-adac-11e9-ad79-d1f60e3ee822 lm(t):2(2019-08-15T15:22:00.300-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X2 lm(t):2(2019-08-14T07:55:10.074-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.829-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.829-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.829-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.829-07:00) cntrlSbst=(APPROVED, FCH1748V0MS) lm(t):2(2019-08-15T15:22:00.300-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):2(2019-08-08T01:42:03.670-07:00) commissioned=YES lm(t):1(2019-08-08T01:02:17.961-07:00) registered=YES lm(t):7(2019-07-24T15:24:25.693-07:00) standby=NO lm(t):2(2019-08-08T01:42:03.670-07:00) DRR=NO lm(t):1(2019-08-08T01:02:17.961-07:00) apicX=NO lm(t):2(2019-08-08T01:42:03.670-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-13T17:02:32.983-07:00) health=(applnc:255 lm(t):2(2019-08-17T01:32:51.454-07:00) svc's)
appliance id=3 address=10.0.0.3 lm(t):7(2019-08-08T03:50:55.470-07:00) tep address=10.0.0.0/16 lm(t):3(2019-07-23T19:05:56.405-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.23.96.12/21 lm(t):1(2019-08-18T23:14:28.721-07:00) version=4.2(0.235j) lm(t):3(2019-08-15T15:21:59.893-07:00) chassisId=1f98b916-adb7-11e9-a6f8-abe00a04e8e6 lm(t):3(2019-08-15T15:21:59.893-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X4 lm(t):3(2019-08-14T07:55:22.256-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) cntrlSbst=(APPROVED, FCH1930V1X6) lm(t):3(2019-08-15T15:21:59.893-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):3(2019-08-08T02:15:20.560-07:00) commissioned=YES lm(t):2(2019-08-08T01:42:15.337-07:00) registered=YES lm(t):7(2019-07-24T15:24:25.693-07:00) standby=NO lm(t):3(2019-08-08T02:15:20.560-07:00) DRR=NO lm(t):2(2019-08-08T01:42:15.337-07:00) apicX=NO lm(t):3(2019-08-08T02:15:20.560-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-13T17:02:33.182-07:00) health=(applnc:255 lm(t):3(2019-08-15T16:08:46.119-07:00) svc's)
appliance id=4 address=10.0.0.4 lm(t):7(2019-08-08T03:50:55.470-07:00) tep address=10.0.0.0/16 lm(t):4(2019-07-23T17:46:15.545-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.23.97.231/21 lm(t):1(2019-08-18T23:14:28.717-07:00) version=4.2(0.235j) lm(t):4(2019-08-15T15:22:00.669-07:00) chassisId=3a7f38aa-adac-11e9-8869-a9e520cdc042 lm(t):4(2019-08-15T15:22:00.669-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X8 lm(t):4(2019-08-14T07:54:59.490-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.825-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.825-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.825-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.825-07:00) cntrlSbst=(APPROVED, FCH1902V1WW) lm(t):4(2019-08-15T15:22:00.669-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):4(2019-08-08T02:40:09.610-07:00) commissioned=YES lm(t):3(2019-08-08T02:15:32.613-07:00) registered=YES lm(t):7(2019-07-24T15:24:25.693-07:00) standby=NO lm(t):4(2019-08-08T02:40:09.610-07:00) DRR=NO lm(t):3(2019-08-08T02:15:32.613-07:00) apicX=NO lm(t):4(2019-08-08T02:40:09.610-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-15T15:21:59.914-07:00) health=(applnc:255 lm(t):4(2019-08-17T01:39:26.477-07:00) svc's)
appliance id=5 address=10.0.0.5 lm(t):7(2019-08-08T03:50:55.470-07:00) tep address=10.0.0.0/16 lm(t):5(2019-07-23T19:05:11.089-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.23.97.232/21 lm(t):1(2019-08-18T23:14:28.723-07:00) version=4.2(0.235j) lm(t):5(2019-08-15T15:22:00.248-07:00) chassisId=35428666-adb7-11e9-a315-1d7671b518b3 lm(t):5(2019-08-15T15:22:00.248-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X10 lm(t):5(2019-08-14T07:55:19.573-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.854-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.854-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.854-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.854-07:00) cntrlSbst=(APPROVED, FCH1902V1EG) lm(t):5(2019-08-15T15:22:00.248-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):5(2019-08-08T03:03:50.338-07:00) commissioned=YES lm(t):4(2019-08-08T02:40:15.939-07:00) registered=YES lm(t):7(2019-07-24T15:24:25.693-07:00) standby=NO lm(t):5(2019-08-08T03:03:50.338-07:00) DRR=NO lm(t):4(2019-08-08T02:40:15.939-07:00) apicX=NO lm(t):5(2019-08-08T03:03:50.338-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-15T15:21:59.756-07:00) health=(applnc:255 lm(t):5(2019-08-17T01:32:43.730-07:00) svc's)
appliance id=6 address=10.0.0.6 lm(t):7(2019-08-08T03:50:55.470-07:00) tep address=10.0.0.0/16 lm(t):6(2019-07-23T19:39:41.972-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.31.170.230/21 lm(t):1(2019-08-18T23:14:28.727-07:00) version=4.2(0.235j) lm(t):6(2019-08-15T15:22:00.562-07:00) chassisId=066c943a-adbc-11e9-bbed-257398025731 lm(t):6(2019-08-15T15:22:00.562-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X20 lm(t):6(2019-08-14T07:55:20.053-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.820-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.821-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.821-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.821-07:00) cntrlSbst=(APPROVED, WZP22350JFT) lm(t):6(2019-08-15T15:22:00.562-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=9 lm(t):6(2019-08-08T03:28:11.246-07:00) commissioned=YES lm(t):5(2019-08-08T03:03:57.387-07:00) registered=YES lm(t):7(2019-07-24T15:24:25.693-07:00) standby=NO lm(t):6(2019-08-08T03:28:11.246-07:00) DRR=NO lm(t):5(2019-08-08T03:03:57.387-07:00) apicX=NO lm(t):6(2019-08-08T03:28:11.246-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-13T17:30:37.663-07:00) health=(applnc:255 lm(t):6(2019-08-15T15:57:05.128-07:00) svc's)
appliance id=7 address=10.0.0.7 lm(t):7(2019-08-08T03:50:48.149-07:00) tep address=10.0.0.0/16 lm(t):7(2019-07-24T15:24:19.988-07:00) routable address=0.0.0.0 lm(t):0(zeroTime) oob address=172.31.172.157/21 lm(t):1(2019-08-18T23:14:28.722-07:00) version=4.2(0.235j) lm(t):7(2019-08-15T15:22:00.539-07:00) chassisId=859be4ae-ae61-11e9-9840-7d9d67698989 lm(t):7(2019-08-15T15:22:00.539-07:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X40 lm(t):7(2019-08-14T07:55:23.872-07:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-08-15T15:22:00.824-07:00) cntrlSbst=(APPROVED, FCH2051V116) lm(t):7(2019-08-15T15:22:00.539-07:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=10 lm(t):7(2019-08-08T03:50:48.149-07:00) commissioned=YES lm(t):6(2019-08-08T03:28:16.727-07:00) registered=YES lm(t):6(2019-07-24T15:27:25.518-07:00) standby=NO lm(t):7(2019-08-08T03:50:48.149-07:00) DRR=NO lm(t):6(2019-08-08T03:28:16.727-07:00) apicX=NO lm(t):7(2019-08-08T03:50:48.149-07:00) virtual=NO lm(t):0(zeroTime) active=YES(2019-08-13T17:30:45.488-07:00) health=(applnc:255 lm(t):7(2019-08-17T01:39:26.549-07:00) svc's)
---------------------------------------------
clusterTime=<diff=2817 common=2019-08-19T15:33:55.929-07:00 local=2019-08-19T15:33:53.112-07:00 pF=<displForm=0 offsSt=0 offsVlu=-25200 lm(t):7(2019-08-08T03:50:55.925-07:00)>>
---------------------------------------------
apic1# acidiag rvread 6 3 1
(6,3,1) st:6 lm(t):3(2014-10-16T08:48:20.238+00:00) le: reSt:LEADER voGr:0 cuTerm:0x19 lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x31 veFiEn:0x31 lm(t):3(2014-10-16T08:48:20.120+00:00)
lastUpdt 2014-10-16T09:07:00.214+00:00
---------------------------------------------
clusterTime=<diff=65247252 common=2014-10-16T09:07:01.837+00:00 local=2014-10-15T14:59:34.585+00:00
pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):3(2014-10-16T04:50:08.714+00:00)>>
apic1# acidiag rvread 6 3
(6,3,1) st:6 lm(t):3(2014-10-16T08:48:20.238+00:00) le: reSt:LEADER voGr:0 cuTerm:0x19 lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x31 veFiEn:0x31 lm(t):3(2014-10-16T08:48:20.120+00:00)
lastUpdt 2014-10-16T09:08:30.240+00:00
(6,3,2) st:6 lm(t):1(2014-10-16T08:47:25.323+00:00) le: reSt:FOLLOWER voGr:0 cuTerm:0x19 lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x49 veFiEn:0x49 lm(t):1(2014-10-16T08:48:20.384+00:00) lp: clSt:2
lm(t):1(2014-10-16T08:47:03.286+00:00) dbSt:2 lm(t):1(2014-10-16T08:47:02.143+00:00) stMmt:1
lm(t):0(zeroTime) dbCrTs:2014-10-16T08:47:02.143+00:00 lastUpdt 2014-10-16T08:48:20.384+00:00
(6,3,3) st:6 lm(t):2(2014-10-16T08:47:13.576+00:00) le: reSt:FOLLOWER voGr:0 cuTerm:0x19 lCoTe:0x18
lCoIn:0x1800000000001b2a veFiSt:0x43 veFiEn:0x43 lm(t):2(2014-10-16T08:48:20.376+00:00)
lastUpdt 2014-10-16T09:08:30.240+00:00
---------------------------------------------
clusterTime=<diff=65247251 common=2014-10-16T09:08:30.445+00:00 local=2014-10-15T15:01:03.194+00:00
pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):3(2014-10-16T04:50:08.714+00:00)>>

Feedback