Cisco NIR Application on Cisco APIC Troubleshooting Commands
Faults
If faults occur within the application, they can be viewed from the Warning icon at the top-right of Application GUI screen next to the Settings icon.
Property | Description |
---|---|
Creation Time |
The day and time of when the audit log, event, or fault instance occurred. |
Severity |
The current severity level of the event. The levels are:
|
Code |
The code that helps to categorize and identify different types of fault instance objects. |
Last Transition |
The day and time on which the severity last changed. If the severity has not changed, this field displays the original creation date. |
Description |
Additional descriptive information on the audit log, event or fault. |
Basic Debugging Commands
apic-ifc1# acidiag scheduler status
Scheduler status:
[True] APIC-01
[True] APIC-02
[True] APIC-03
apic-ifc1# acidiag scheduler members
ID Name Status Address OOBAddress Type Serial NodeFqdn
-----------------------------------------------------------------------------------------------------------------------------
1* apic-ifc1 active 10.0.0.1 172.1.2.3 Apic FCH1748V24D apic-ifc1.node.ifav22.apic.local
2 apic-ifc2 active 10.0.0.2 172.4.5.6 Apic FCH1809V18S apic-ifc2.node.ifav22.apic.local
3 apic-ifc3 active 10.0.0.3 172.7.8.9 Apic FCH1809V191 apic-ifc3.node.ifav22.apic.local
apic-ifc1#
apic-ifc1# acidiag scheduler appstatus
Job Type Status
-------------------------------------------------------
Cisco_NIR
`-Cisco_NIR-ClusterService service running
`-Cisco_NIR-SystemService system running
bird_kafka
`-bird_kafka-kafka system running
bird_kafkax
`-bird_kafkax-kafka system running
bird_zk
`-bird_zk-zk service running
elastic
`-elastic-systemjob system running
elasticx
`-elasticx-systemjob system running
apic-ifc1# acidiag scheduler appstatus bird_kafka
Container Group Node Status Modified Image
------------------------------------------------------------------------------------------------------------------------------------------------------
kafka bird_kafka-kafka.kafka apic-ifc3 running 0d 19h 37m 16s apic-system/kafka:0.1.0
kafka bird_kafka-kafka.kafka apic-ifc1 running 0d 19h 37m 16s apic-system/kafka:0.1.0
kafka bird_kafka-kafka.kafka apic-ifc2 running 0d 19h 37m 16s apic-system/kafka:0.1.0
apic-ifc1# acidiag scheduler appstatus elastic
Container Group Node Status Modified Image
------------------------------------------------------------------------------------------------------------------------------------------------------
es elastic-systemjob.db apic-ifc1 running 0d 19h 41m 8s apic-system/elastic:v1
es elastic-systemjob.db apic-ifc3 running 1d 13h 2m 52s apic-system/elastic:v1
es elastic-systemjob.db apic-ifc2 running 1d 13h 13m 15s apic-system/elastic:v1
apic-ifc1# acidiag scheduler appstatus Cisco_NIR
Container Group Node Status Modified Image
------------------------------------------------------------------------------------------------------------------------------------------------------
app-brain Cisco_NIR-ClusterService.brain apic-ifc2 running 0d 18h 58m 53s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/brain:v1-0-1-827
app-scheduler Cisco_NIR-ClusterService.scheduler apic-ifc1 running 0d 18h 58m 54s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/scheduler:v1-0-1-827
app-correlator Cisco_NIR-ClusterService.correlator apic-ifc3 running 0d 18h 58m 53s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/correlator:v1-0-1-827
app-predictor Cisco_NIR-ClusterService.predictor apic-ifc3 running 0d 18h 58m 53s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/predictor:v1-0-1-827
app-apicagent Cisco_NIR-ClusterService.apicagent apic-ifc2 running 0d 18h 58m 54s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apicagent:v1-0-1-827
app-logstash Cisco_NIR-SystemService.logstash apic-ifc1 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-eventcollector Cisco_NIR-SystemService.eventcollector apic-ifc3 running 0d 18h 59m 5s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827
app-eventcollector Cisco_NIR-SystemService.eventcollector apic-ifc1 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827
app-logstash Cisco_NIR-SystemService.logstash apic-ifc2 running 0d 18h 59m 5s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-apiserver Cisco_NIR-SystemService.apiserver apic-ifc2 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-apiserver Cisco_NIR-SystemService.apiserver apic-ifc1 running 0d 18h 59m 5s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-logstash Cisco_NIR-SystemService.logstash apic-ifc3 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-apiserver Cisco_NIR-SystemService.apiserver apic-ifc3 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-eventcollector Cisco_NIR-SystemService.eventcollector apic-ifc2 running 0d 18h 59m 4s local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827
apic-ifc1#
apic-ifc1# acidiag scheduler elastic members
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.0.0.3 26 99 20 4.88 4.40 3.49 mdi - apic-ifc3
10.0.0.1 26 91 19 3.04 3.75 3.56 mdi - apic-ifc1
10.0.0.2 26 88 19 0.97 1.77 2.05 mdi * apic-ifc2
apic-ifc1# acidiag scheduler elastic health
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 120,
"active_shards" : 360,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
apic-ifc1# acidiag scheduler elastic indices
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open cisco_nir-fabricnodesdb B8X8lktsSnWzCckzms8JfQ 1 2 16 0 182.7kb 61kb
green open cisco_nir-aggflowdb-2019.01.31.18.00.00 RnIB3S7fTBikO07xPhquFw 9 2 0 0 6.1kb 2kb
green open cisco_nir-sysmetrics-2019.01.31 HBP_iJgsRQyGTyOa-Horvg 7 2 1807463 0 747.1mb 249.1mb
green open cisco_nir-statsdb-000003 Sgh1bZ7CQ_et4j__AQ56Ww 5 2 9517896 0 2.9gb 998.8mb
green open cisco_nir-eventsdb tJTC02wpSmy_9Fa8p33WDg 5 2 22940 0 25.4mb 8.4mb
green open searchguard 9nSh8NeqSYKYF7w4W0eHkQ 1 2 5 2 65.1kb 21.7kb
green open cisco_nir-statsdb-000002 Zv9P247tSfyK_6o37NGkjg 5 2 9494058 0 2.9gb 999.5mb
green open cisco_nir-fault_historydb mUY-NT2lQqmP54flD44xzg 5 2 2405 0 4.3mb 1.4mb
green open cisco_nir-collectorstatsdb 6RrCkrhxT6OWz-M8eIfjrw 5 2 0 0 3.4kb 1.1kb
green open cisco_nir-sysmetrics-2019.01.30 wz3Jif_8SMOhc4Or8MEXNg 7 2 41870 0 19.2mb 6.4mb
green open cisco_nir-fabric_issuesdb tj-Y0cP4SF2OdfMkumqcqQ 2 2 0 0 1.3kb 466b
green open cisco_nir-anomalytsdb rzGukbWCTk276i2FQpRCJQ 3 2 1 0 24.9kb 8.3kb
green open cisco_nir-aggflowdb-2019.01.31.12.00.00 hVUmPx5JQJi9gtiEB4no_A 9 2 0 0 6.1kb 2kb
green open cisco_nir-resourcecollectdb kDTBYxq0RtSp0tzXkFgVWw 3 2 168380 0 38.1mb 12.7mb
green open cisco_nir-resourcescoresdb ApM3S1QEQ3m9co-UeX-tvQ 3 2 38120 0 29.4mb 9.8mb
green open cisco_nir-aggflowdb-2019.01.31.16.00.00 fdaRZvNVS2eVqEF1uRFKcg 9 2 0 0 6.1kb 2kb
green open cisco_nir-eprecordsdb JIzHooPPQwShJeFCa11GyA 5 2 0 0 3.4kb 1.1kb
green open cisco_nir-statsdb-000004 pqhaqo3OTv6E6y1zBwfwYg 5 2 1539566 0 507.6mb 170.8mb
green open cisco_nir-aggflowdb-2019.01.31.14.00.00 G6yngLSoQzyn1odMCDLIaQ 9 2 0 0 6.1kb 2kb
green open cisco_nir-licensedb 87XBQmQHRfap024AAXnEXg 1 2 1 0 10.2kb 3.4kb
green open cisco_nir-aggflowdb-2019.01.31.20.00.00 ZQdMl2yxSaaNCdGXW-4YOg 9 2 0 0 6.1kb 2kb
green open cisco_nir-aggflowdb-2019.01.31.10.00.00 btO1x9A0Teakv2AdK_6n-A 9 2 0 0 6.1kb 2kb
green open cisco_nir-anomalydb QrHtrk2LSZ-LNsS37E0btQ 3 2 1 0 23.5kb 7.8kb
apic-ifc1# acidiag scheduler elastic shards
index shard prirep state docs store ip node
cisco_nir-sysmetrics-2019.01.30 4 r STARTED 5914 924.5kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 4 p STARTED 5914 928.9kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 4 r STARTED 5914 899.8kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 1 r STARTED 6033 920.7kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 1 p STARTED 6033 954.1kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 1 r STARTED 6033 982.7kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 2 r STARTED 6070 944.1kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 2 r STARTED 6070 914.2kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 2 p STARTED 6070 951.1kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 6 p STARTED 5923 961.2kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 6 r STARTED 5923 944.4kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 6 r STARTED 5923 958.8kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 3 p STARTED 5962 954.4kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 3 r STARTED 5962 911.1kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 3 r STARTED 5962 926.3kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 5 r STARTED 6003 937.9kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 5 r STARTED 6003 931.6kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 5 p STARTED 6003 912kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30 0 p STARTED 5965 947.9kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30 0 r STARTED 5965 909.2kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30 0 r STARTED 5965 966.8kb 10.0.0.1 ifav22-ifc1
<-- SNIP LIST OF ALL OTHER RESOURCES -->
apic-ifc1#