Operating Cisco Application Centric Infrastructure

Proactive Monitoring - Tenant and Fabric Policies

Proactive monitoring is a very important piece of the network administrator's job, but is often neglected because putting out fires in the network usually takes priority. However, since the Application Policy Infrastructure Controller (APIC) makes it incredibly easy to gather statistics and perform analyses, this will save network administrators both time and frustration. Since statistics are gathered automatically and policies are used and can be re-used in other places, the human error and effort is minimal.

Statistics gathering has been a somewhat manual and even resource intensive process for ACME in the past. Even when they have used tools to gather data on layer one through seven devices, it has still been necessary to manually specify which devices are to be monitored and how they should be monitored. For example, SNMP and a third party tool may have been used to monitor the CPU of switches or bandwidth utilization on ports, but they struggled with entering correct SNMP information on each device, or often forgot to add a new device to their Network Monitoring System (NMS). Cisco Application Centric Infrastructure (ACI) provides an APIC which will do all of the statistics gathering, and provides the ability to proactively monitor your entire environment without all of the hassle of maintaining a third party monitoring tool.

The APIC, whether accessed through the GUI, CLI, or API, can be used to drill into any of the components and provides the ability to click on a Stats tab to display on-demand statistics, but more importantly it enables the setup of policies to keep persistent data to analyze trends in the environment, as well as to troubleshoot or predict any issues that may be arising. When planning to move an application from a legacy network to the ACI infrastructure, it is sensible to start by testing before going straight to production. Add test VMs to port groups on either a DVS or AVS associated with the APIC, and add physical test servers to VPCs on the leaf switches. This could also be in a testing tenant which is completely separate from the production environment. At this point the APIC is already gathering statistics for the VMM domain and the physical devices. The next step is to configure a policy for trend analysis.

There are four different scopes for statistics gathering: Common or Fabric Wide, Fabric, Tenant, or Access. A Fabric Wide policy would be created as a default policy to be applied to all tenants. However, to override that policy for a particular tenant, the tenant policy will override the Fabric policy. In the following testing example, a Tenant policy is created to gather statistics. Even if this tenant is shared with other applications, customers, test cases, it will provide a real world example of how the application will behave in a production environment.

Create Tenant Monitor Policy

To create a tenant monitoring policy:

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane, choose Tenant_Name > Monitoring Policies.
In the Work pane, choose Actions > Create Monitoring Policies.
In the Create Monitoring Policies dialog box, perform the following actions:
1. In the Name field enter a name for the Monitoring Policy.
2. Click Submit.
In the Navigation pane, choose Tenant_Name > Monitoring Policies > Policy_Name to display the following information:
- Stats Collection Policies
- Stats Export Policies
- Callhome, SNMP, and Syslog
- Event Severity Assignment Policies
- Fault Severity Assignment Policies
- Fault Lifecycle Policies

Stats Collection Policies

Clicking on Stats Collection Policies will display the default retention periods and admin states (Enabled/Disabled) for ALL Monitored Objects. Most likely the defaults will be kept, but a double click on them will change the admin state or retention periods. For example, to have it poll a component every 5 minutes, but be retained for 2 hours, just click on the policy that specifies a 5 minute granularity and change the retention period to 2 hours. It is similarly possible to change the policies for specific Monitoring Objects. A monitoring object tells the APIC which components to gather statistics about. For example, to change the information gathered for Bridge Domains, use the Bridge Domain (infra.RSOInfraBD) Monitoring Object.

To add monitoring objects:

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane choose Tenant_Name > Monitoring Policies > Monitoring Policy_Name > Stats Collection Policies
1. Click on the Pencil icon to edit the Monitoring Objects.
2. Put a checkmark next to the Monitoring Objects to be included, and remove any checkmarks next to Monitoring Objects to be left out.
3. Click Submit.

For this example, changes might be made to Monitoring Object policies for Tenant, VXLAN Pool, Leaf Port, and/or Taboo Contract. There are several options and this will all depend on what is important to monitor in the environment. Click on the pull down menu to select a monitoring object and add a retention policy to it.

To add a policy to a Monitoring Object:

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane choose Monitoring Policies > Monitoring Policy_Name > Stats Collection Policies.
In the Work pane, in the Stats Collection Policy dialog box, perform the following actions:
1. Select the Monitoring Object.
2. Click + to add the policy.
3. Select the granularity with which it is to poll.
4. Leave the state as inherited to stick with the defaults as set for ALL, or explicitly select enabled or disabled.
5. The retention policy may either be inherited or explicitly specified as enabled or disabled as well.
6. Click Update.

Stats Export Policies

It is desirable to collect these ongoing statistics as well as to see how this data behaves over time. Use the Stats Export Policies option in the left navigation pane, located under the monitoring policy. Much like the Stats Collection Policies, it is possible to create a policy for ALL monitoring objects, or select specific monitoring objects and specify where this information will be saved.

To create a Stats Export Policy:

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane choose Tenant_Name > Monitoring Policies > Monitoring Policy_Name > Stats Export Policies.
In the Work pane, in the Stats Export Policy dialog box, perform the following actions:
1. Select ALL or a specific monitoring object from the drop-down list.
2. Click + to add the policy.
3. Now define the Stats Export Policy in the wizard.
4. Choose either JSON or XML as the format. There's really no difference other than personal preference, or it may be dictated by the tool used to read it.
5. Choose to compress it using GZIP, or leave it uncompressed.
6. Click + under Export Destinations to specify a server where this information is to be collected. Another wizard will pop up to enable specification of the protocols and credentials used to connect to this server.
7. Click Ok.
Click Submit.

Diagnostics Policies Using the GUI

The diagnostics policies are in the navigation pane on the left. This feature allows the setup of diagnostics test for the Monitoring Objects that were specified in the Stats Collection Policies. Next to the Monitoring Object is the Pencil button which enables selection of the monitoring objects to be configured with diagnostics policies. There are two different kind of policies for configuration: Boot-Up diagnostics or Ongoing diagnostics.

To configure diagnostic policies:

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane choose Tenant_Name > Monitoring Policies > Diagnostics Policies.
In the Work pane, in the Diagnostic Policies dialog box, perform the following actions:

Click on the Pencil Icon and put checks next to the Monitoring Objects to which diagnostics tests are to be added.
1. Select one of the Monitoring Objects.
2. Click + to add an Object.
  1. Select either Boot-Up or Ongoing.
  2. Boot-Up runs the tests while the devices are booting, and Ongoing will run the tests as often as specified within the wizard.
  3. In the wizard give it a name and select the admin state.
  4. There are five different diagnostics tests available: ASIC, CPU, Internal Connectivity, Peripherals, and System Memory. Double-click on each to obtain the option of specifying no tests, full tests, or recommended tests.
  5. Click Submit.

The diagnostics found here can be useful in finding failed components before they cause major issues within your environment.

Call Home/SNMP/Syslog

There are a few different ways to setup notification or alert policies. The Call Home/SNMP/Syslog policy will allow alerting to be configured in a flexible manner. Cisco Call Home is a feature in many Cisco products that will provide email or web-based notification alerts in several different formats for critical events. This allows administrators to resolve issues before they turn into outages. SNMP or syslog policies can also be used with current notification systems. Different logging levels may be selected for notifications and alert levels specified for Monitoring Objects from which alerts are to be received.

Event Severity and Fault Severity Assignments

Event and fault severities can be changed for events raised by Monitoring Objects. Most likely, the default severity assignments for Events and Faults will be kept, but there are examples where an ACI administrator may decide the event or fault is more or less severe than the default value. For example, if only critical faults are being notified, but there is a major fault you'd also like to be notified about immediately, you can change the severity for that particular fault code.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane, choose Tenant_Name > Monitoring Policies > Monitoring_Policy > Fault Lifecycle Policies.
In the Work pane, in the Fault Severity Assignment Policies dialog box, perform the following actions:
1. Select a Monitoring Object, which will dictate the fault codes for which you are changing the fault severity.
2. Click + to add an Object.
3. Select the particular fault code for which severity is to be changed.
4. Select the severity: Cleared, Critical, Major, Minor, Squelched, Inherit, Warning, Info.
  
  Squelched gives it a weight of 0%, meaning it does not affect health scores.
Click Update.

The Event Severity Assignment Policies are configured in the same way.

Fault Lifecycle Policies

Fault Lifecycle is the term Cisco uses to describe the life of a fault. Once a fault is detected it is in the "soaking" state. After a certain amount of time, referred to as the "soaking interval" it will move on to the "raised" state. "Raised" means the fault is still present after the soaking interval. After the fault clears it's in a state called "raised clearing." It is only in this state briefly and moves on to the "clearing time" state. It remains in the "clearing time" state for the amount of time specified in the "clearing interval." Lastly it moves on to the "retaining" state and does not get removed until the end of the "retaining interval."

To change Fault Lifecycle Intervals:

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the Tenant_Name .
In the Navigation pane, choose Tenant_Name > Monitoring Policies > Monitoring_Policy > Fault Lifecycle Policies.
In the Work pane, in the Fault Lifecycle Policies dialog box, perform the following actions:
1. Select a Monitoring Object, which will dictate the fault codes for which you are changing the default intervals.
2. Click +.
3. Specify times for the Clearing Interval, Retention Interval, and Soaking Interval (all in seconds).
  
  Note: The default for the Clearing Interval is 120 seconds; the Retention Interval is 3600 seconds; and the Soaking Interval is 120 seconds.

At this point there will be a fully working tenant monitoring policy. ACME will have other policies to configure in the fabric as outlined in the following sections.

TCAM Policy Usage

The physical ternary content-addressable memory (TCAM) in which policy is stored for enforcement is an expensive component of switch hardware and therefore tends to lower policy scale or raise hardware costs. Within the Cisco Application Centric Infrastructure (Cisco ACI) fabric, policy is applied based on the EPG rather than the endpoint itself. This policy size can be expressed as n*m*f, where n is the number of sources, m is the number of destinations, and f is the number of policy filters. Within the Cisco ACI fabric, sources and destinations become one entry for a given EPG, which reduces the number of total entries required.

TCAM is a fabric resource that should be monitored. There is a system wide view of available TCAM resources. To view the TCAM resources, on the menu bar, choose Operations > Capacity Dashboard. The Work pane displays a table that summarizes the capacity for all nodes.

TCAM is a critical system resource in a Cisco ACI fabric and should be monitored for utilization. The architecture/design team should articulate what the assumptions were for TCAM utilization. There is a Fabric Resource Calculation tool on Github that will help with estimation of normal operating parameters: https://github.com/datacenter/FabricResourceCalculation.

As a general rule, the default monitoring policies will alert you to a resource shortage and lower overall fabric health score. If your environment has a high rate of change, or you anticipate the possibility of consistently being oversubscribed, you may want to set different thresholds.

Create TCAM Policy Monitor

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, choose Monitor Policies > d efault > Stats Collection Policies.
In the Work pane, in the Stats Collection Policies dialog box, perform the following actions:
1. Select the Monitoring Object Equipment Capacity Entity (eqptcapacity.Entity).
2. Select the Stats Type Policy Entry.
3. Click + under Config Thresholds.
4. In the Thresholds For Collection 5 Minute window, select the blue pencil icon next to policy CAM entries usage current value.

TCAM Prefix Usage

This procedure manages a TCAM Prefix Usage.

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, choose Monitor Policies > default > Stats Collection Policies.
In the Work pane, in the Stats Collection Policies dialog box, perform the following actions:
1. Select the Monitoring Object Equipment Capacity Entity (eqptcapacity.Entity).
2. Select the Stats TypeLayer3 Entry.
3. Click + under Config Thresholds.
4. In the Thresholds For Collection 5 Minute window, select the blue pencil icon next to policy CAM entries usage current value.

Health Score Evaluation Policy

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, choose Monitor Policies > Common Policies > Health Score Evaluation Policy > Health Score Evaluation Policy.
In the Work pane, in the Properties dialog box, perform the following actions:
1. In the Penalty of fault severity critical dropdown menu, select the desired %.
2. In the Penalty of fault severity major dropdown menu, select the desired %.
3. In the Penalty of fault severity minor dropdown menu, select the desired %.
4. In the Penalty of fault severity warning dropdown menu, select the desired %.
Click Submit.

Communication Policy

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, expand Pod Policies > Policies > Communication.
In the Work pane, choose Actions > Create Communication Policy.
In the Create Communication Policy dialog box, perform the following actions:
1. Enter Communication Policy Name.
2. From the HTTP Admin State dropdown menu select the desired state.
3. From the HTTP Port dropdown menu select the desired port.
4. Select the desired HTTP redirect state.
5. From the HTTPS Admin State dropdown menu select the desired state.
6. From the HTTPS Port dropdown menu select the desired port.
7. Select the desired HTTPS redirect state.
8. From the SSH Admin State dropdown menu select the desired state.
9. From the Telnet Admin State dropdown menu select the desired state.
10. From the Telnet Port dropdown menu select the desired port.
Click Submit.

CPU Utilization and Memory

GUI

The easiest way to quickly verify the health of the controllers is the APIC. When logging into the system dashboard, the health of the APICs and the health of the cluster itself are displayed right at the dashboard.

The normal state for an APIC is to be green in a "fully fit" state, implying the APICs are synchronized with each other.

A more detailed drilldown is available by clicking on System > Controllers.

REST API

Controllers provide information regarding the current status of CPU and memory utilization by creating instances of the procEntity class. procEntity is a container of processes in the system. This object holds detailed information about various processes running on the APIC. The procEntity objects contain the following useful properties:

cpuPct - CPU utilization

maxMemAlloc - The maximum memory allocated for the system

memFree - The maximum amount of available memory for the system

Sample Usage: This information can be retrieved for all APIC controllers using the following REST call:

http[s]://apic_ip/api/node/class/procEntity.xml?

CLI

The Linux Top utility also comes built into the APIC controllers and can be used for troubleshooting and/or verification.


user@apic1:~> top
top - 11:41:51 up 16:50, 4 users, load average: 4.19, 4.27, 4.29
Tasks: 354 total, 1 running, 353 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.4%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 131954932k total, 7473180k used, 124481752k free, 409540k buffers
Swap: 0k total, 0k used, 0k free, 1952656k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32102 root 20 0 556m 205m 85m S 3.3 0.2 38:11.04 svc_ifc_applian
32120 ifc 20 0 660m 343m 86m S 2.0 0.3 27:58.73 svc_ifc_dbgr.bi
32121 ifc 20 0 631m 286m 86m S 2.0 0.2 17:41.92 svc_ifc_topomgr
32105 root 20 0 659m 258m 85m S 1.7 0.2 17:08.35 svc_ifc_bootmgr
32113 ifc 20 0 1083m 721m 69m S 1.7 0.6 20:03.37 svc_ifc_observe
32128 ifc 20 0 639m 315m 69m S 1.7 0.2 16:28.34 svc_ifc_reader.
32132 ifc 20 0 657m 252m 71m S 1.7 0.2 17:13.74 svc_ifc_scripth
1291 root 20 0 834m 419m 94m S 1.3 0.3 20:35.24 nginx.bin

Disk Utilization

GUI

There are several disks and file systems present on the APICs. The GUI provides ready access to disk space utilization of all partitions on the system and can be used for monitoring this information.

The disk utilization can be viewed by clicking on System > Controllers > Apic-X > Storage

The work pane displays the utilization of all partitions in the system.

REST API

This information can be retrieved for all APIC controllers using the following REST call:


http[s]://apic-ip/api/node/class/eqptStorage.xml?

CLI


user@apic1:~> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-1 41282880 10518960 28666872 27% /
tmpfs 4194304 56456 4137848 2% /dev/shm
tmpfs 65977464 964 65976500 1% /tmp
/dev/mapper/vg_ifc0-data
41282880 10518960 28666872 27% /data
/dev/mapper/vg_ifc0-firmware
41284928 13860672 25327104 36% /firmware
/dev/mapper/vg_ifc0-data2
583149656 1281104 552246280 1% /data2

* Note that not all file systems are visible from the CLI as some require root access to reach the mount points. The GUI should be used as a single source of truth for file system utilization.

Physical and Bond Interface Statistics

APICs use a bonded interface that is typically dual-homed to two leaves for connectivity to the ACI fabric and have the ability to use a bonded interface that can be dual homed to the out-of-band management network.

Bond0 is the bond interface used to connect to the fabric itself (to connect to leaves that connect into the fabric).

Bond1 is the bond interface used to connect to the out-of-band segment (to connect to an OOB segment that allows setup of the APIC itself).

The bond interfaces rely on underlying physical interfaces and it is important to note that the GUI provides link information for both the physical and logical bond interfaces.

GUI

To view interface status for the interfaces on the APICs, navigate to System > Controllers > Apic-x > Interfaces

REST API

This information can be retrieved for all APIC controllers using the following REST call:


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1/sys.json?querytarget=
subtree&target-subtree-class=l3EncRtdIf

CLI

Both "ifconfig" and the "ip link" CLI commands can be used to verify link state. The CLI also provides information on detailed interface statistics such as RX and TX counters.

Fan Status

The following section describes methodologies to retrieve the status of the fan trays on the APICs.

GUI

To view interface status for the interfaces on the APICs, navigate to System > Controllers > Apic-x > Equipment-Fans

REST API

This information can be retrieved for all APIC controllers using the following REST call:


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1.json?querytarget=
subtree&target-subtree-class=eqptFan

CLI

The Fan status for the APICs can be monitored using the CLI on the CIMC port of the APIC. To obtain this, login to the CIMC using the credentials used for setting up the CIMC (may not be the same as the credentials used for APIC). If this has not been setup previously, the default username is admin and the default password is password.

The CIMC port is the integrated lights-out management port that can be used to recover an APIC in the event of a catastrophic failure.


user@apic1:~> ssh -l admin 172.16.176.179
Warning: Permanently added '172.16.176.179' (RSA) to the list of known hosts.
admin@172.16.176.179's password:
C220-FCH1807V02V# scope sensor
C220-FCH1807V02V /sensor # show fan
Name Sensor Reading Units Min. Max. Min. Max.
Status Warning Warning Failure Failure
----------- ------- -------- ----- --------- --------- -------- ---------
FAN1_TACH1 Normal 7490 RPM 1712 N/A 1284 N/A
FAN1_TACH2 Normal 7490 RPM 1712 N/A 1284 N/A
FAN2_TACH1 Normal 7490 RPM 1712 N/A 1284 N/A
FAN2_TACH2 Normal 7276 RPM 1712 N/A 1284 N/A
FAN3_TACH1 Normal 7704 RPM 1712 N/A 1284 N/A
FAN3_TACH2 Normal 7276 RPM 1712 N/A 1284 N/A
FAN4_TACH1 Normal 7704 RPM 1712 N/A 1284 N/A
FAN4_TACH2 Normal 7276 RPM 1712 N/A 1284 N/A
FAN5_TACH1 Normal 7704 RPM 1712 N/A 1284 N/A

Temperature Status

To monitor the temperature state of the various sensors available on the APICs use the following steps.

GUI

To view interface status for the interfaces on the APICs, navigate to System > Controllers > Apic-x > Equipment-Sensors

REST API

This information can be retrieved for all APIC controllers using the following REST call:


https://{{apic-ip}}//api/node/mo/topology/pod-1/node-1.json?querytarget=
subtree&target-subtree-class=eqptSensor

CLI


C220-FCH1807V02V /sensor # show temperature
Name Sensor Reading Units Min. Max. Min. Max.
Status Warning Warning Failure Failure
------------------ -------- -------- ------- ------- --------- -------- ---------
P1_TEMP_SENS Normal 49.5 C N/A 81.0 N/A 86.0
P2_TEMP_SENS Normal 50.5 C N/A 81.0 N/A 86.0
RISER1_INLET_TMP Normal 45.0 C N/A 60.0 N/A 70.0
RISER2_INLET_TMP Normal 41.0 C N/A 60.0 N/A 70.0
RISER1_OUTLETTMP Normal 50.0 C N/A 60.0 N/A 70.0
RISER2_OUTLETTMP Normal 41.0 C N/A 60.0 N/A 70.0
FP_TEMP_SENSOR Normal 37.0 C N/A 60.0 N/A 70.0
DDR3_P1_A1_TEMP Normal 42.0 C N/A 65.0 N/A 85.0
DDR3_P1_B1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P1_C1_TEMP Normal 44.0 C N/A 65.0 N/A 85.0
DDR3_P1_D1_TEMP Normal 44.0 C N/A 65.0 N/A 85.0
DDR3_P2_E1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P2_F1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P2_G1_TEMP Normal 42.0 C N/A 65.0 N/A 85.0
DDR3_P2_H1_TEMP Normal 41.0 C N/A 65.0 N/A 85.0
VICP81E_0_TMP3 Normal 56.0 C N/A 85.0 N/A 90.0
PSU1_TEMP Normal 37.0 C N/A 60.0 N/A 65.0
PCH_TEMP_SENS Normal 51.0 C N/A 80.0 N/A 85.0

Power Supply Status

To monitor the temperature state of the various sensors available on the APICs use the following steps.

CLI


C220-FCH1807V02V /sensor # show psu
Name Sensor Reading Units Min. Max. Min. Max.
Status Warning Warning Failure Failure
------------------ -------- -------- -------- ------- --------- -------- ---------
P1_TEMP_SENS Normal 49.5 C N/A 81.0 N/A 86.0
POWER_USAGE Normal 160 Watts N/A N/A N/A 800
PSU1_POUT Normal 136 Watts N/A 624 N/A 648
PSU1_PIN Normal 160 Watts N/A 720 N/A 744
PSU1_STATUS Normal present
PSU2_STATUS Normal absent
PSU1_PWRGD Normal good
PSU1_AC_OK Normal good

Monitoring Switch CPU Utilization

There are several methods to poll CPU utilization and trend it over different periods of time. The following sections describe a few of the methods available.

GUI

REST API

Spine and Leaf switches CPU utilization can be monitored using the following classes, based on the desired timescale and granularity.

proc:SysCPU5min - A class that represents the most current statistics for System cpu in a 5 minute sampling interval. This class updates every 10 seconds.

proc:SysCPU15min - A class that represents the most current statistics for System cpu in a 15 minute sampling interval. This class updates every 5 minutes.

proc:SysCPU1h - A class that represents the most current statistics for System cpu in a 1 hour sampling interval. This class updates every 15 minutes.

proc:SysCPU1d - A class that represents the most current statistics for System cpu in a 1 day sampling interval. This class updates every hour.

proc:SysCPU1w - A class that represents the most current statistics for System cpu in a 1 week sampling interval. This class updates every day.

proc:SysCPU1mo - A class that represents the most current statistics for System cpu in a 1 month sampling interval. This class updates every day.

proc:SysCPU1qtr - A class that represents the most current statistics for System cpu in a 1 quarter sampling interval. This class updates every day.

proc:SysCPU1year - A class that represents the most current statistics for System cpu in a 1 year sampling interval. This class updates every day.

ACME would like to see the average CPU utilization of all of the fabric switches over the last day.


http[s]://apic_ip//api/node/class/procSysCPU1d.xml?

CLI


Leaf-1# show proc cpu sort
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
4012 69510 493837 140 1.3% t2usd_tor
4065 7239 27609 262 1.3% python
4292 3841 134758 28 0.8% svc_ifc_opflexe
4391 2355 4423 532 0.4% nginx
4067 1911 206 9278 0.4% svc_ifc_policye
4302 1904 1862 1022 0.3% svc_ifc_observe
4311 1811 1018 1779 0.3% svc_ifc_confele
4123 1407 251 5606 0.3% svc_ifc_eventmg
4310 1802 689 2616 0.3% svc_ifc_dbgrele
4846 119693 36527 3276 0.2% stats_manager
3923 15406 2645 5824 0.1% pfmclnt
4864 2361 2812 839 0.1% ospfv3
4865 2402 2717 884 0.1% ospf
13606 435 211 2065 0.0% bgp
4296 6263 7413 844 0.0% snmpd
4297 6667 4542 1467 0.0% dc3_sensor
4299 8559 8225 1040 0.0% policy_mgr
4301 1860 19152 97 0.0% plog
4866 2792 3269 854 0.0% isis
5025 1611 1743 924 0.0% mcecm

In order to obtain a historical view of CPU utilization from the CLI it may be necessary to jump into an alternative shell from the switch bash prompt. This shell is called vsh (or v-shell).


Leaf-1# show processes cpu history
1 1 33 1
746554661885833055376572545534667663554785033943645665335644
100
90
80
70
60
50
40 #
30 ##
20 ##
10 # ### ####### ######## # ## ##### ## #### # # #### ##
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
# = average CPU%
32 13311134111111111311111111131 11111113 1111 11231 1111111
749513800432206328353370732175609342000769025791144192680117
100
90
80
70
60
50
40 * * * *
30 * ** ** * * * *
20 ** **** ** * * ** * * *** ** ** ** ** *
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1
440
030
100 *
90 *
80 *
70 *
60 *
50 *
40 ***
30 ***
20 ***
10 ###
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Monitoring Switch Memory Utilization

There are several methods to poll memory utilization and trend it over different periods of time. The following sections describe a few of the methods available.

Note

Switch memory utilization of 70-75% is normal, even for leaf and spine switches in a new deployment where no configuration has been pushed. Switch memory utilization in ACI mode should not be compared to utilization in standalone NX-OS mode due to the additional ACI-specific processes, which consume additional memory.

You should be concerned only if memory utilization exceeds 90%. Open a TAC case for proactive troubleshooting if memory utilization exceeds 85%.

REST API

Spine and Leaf switches memory utilization can be monitored using the following classes, based on the desired timescale and granularity.

proc:SysMem5min - A class that represents the most current statistics for System memory in a 5 minute sampling interval. This class updates every 10 seconds.

proc:SysMem15min - A class that represents the most current statistics for System memory in a 15 minute sampling interval. This class updates every 5 minutes.

proc:SysMem1h - A class that represents the most current statistics for System memory in a 1 hour sampling interval. This class updates every 15 minutes.

proc:SysMem1d - A class that represents the most current statistics for System memory in a 1 day sampling interval. This class updates every hour.

proc:SysMem1w - A class that represents the most current statistics for System memory in a 1 week sampling interval. This class updates every day.

proc:SysMem1mo - A class that represents the most current statistics for System memory in a 1 month sampling interval. This class updates every day.

proc:SysMem1qtr - A class that represents the most current statistics for System memory in a 1 quarter sampling interval. This class updates every day.

proc:SysMem1year - A class that represents the most current statistics for System memory in a 1 year sampling interval. This class updates every day.

ACME would like to monitor memory over the last day, and would use the following REST call:


http[s]://apic_ip/api/node/class/procSysMem1d.xml?

CLI


Leaf-1# show system resources
Load average: 1 minute: 1.21 5 minutes: 1.14 15 minutes: 0.80
Processes : 513 total, 2 running
CPU states : 4.1% user, 2.5% kernel, 93.4% idle
Memory usage: 16401072K total, 9054020K used, 7347052K free
Current memory status: OK

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective. See the Cisco ACI MIB Support List.

Monitoring File System Health

CLI

Currently, the CLI is only way to monitor the utilization of the file system on the leaves. It is normal to see a higher % utilization on some of the mount points in the file system hierarchy. The critical volumes to keep track of in terms of utilization are /volatile, bootflash and logflash.


Leaf-1# df
df: `/nxos/tmp': No such file or directory
df: `/var/home': No such file or directory
df: `/var/tmp': No such file or directory
df: `/nginx': No such file or directory
df: `/debugfs': No such file or directory
df: `/recovery': No such file or directory
df: `/cfg0': No such file or directory
df: `/cfg1': No such file or directory
df: `/logflash/INXOS_SYSMGR/startup-cfg': No such file or directory
df: `/mnt/plog': No such file or directory
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 512000 1064 510936 1% /
rootfs 512000 1064 510936 1% /
none 512000 1064 510936 1% /isan
none 512000 1064 510936 1% /var
none 51200 2288 48912 5% /etc
none 51200 108 51092 1% /var/log
none 3145728 336664 2809064 11% /dev/shm
none 512000 0 512000 0% /volatile
/dev/sda4 7782036 1080636 6306088 15% /bootflash
/dev/sda5 60485 5356 52006 10% /mnt/cfg/0
/dev/sda6 60485 5356 52006 10% /mnt/cfg/1
/dev/sda3 120979 13349 101384 12% /mnt/pss
/dev/sda7 512000 1064 510936 1% /logflash
/dev/sda9 15748508 591216 14357292 4% /mnt/ifc/cfg
/dev/sda8 15748508 991204 13957304 7% /mnt/ifc/log
/dev/sda8 15748508 991204 13957304 7% /var/log/dme/oldlog
/dev/sda9 15748508 591216 14357292 4% /controller
rootfs 716800 665728 51072 93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
rootfs 716800 665728 51072 93% /controller/sbin
/dev/sda8 15748508 991204 13957304 7% /data/techsupport
rootfs 716800 665728 51072 93% /bin
/dev/sda4 7782036 1080636 6306088 15% /bootflash
rootfs 716800 665728 51072 93% /data/challenge.old.plugin
/dev/sda9 15748508 591216 14357292 4% /controller
rootfs 716800 665728 51072 93% /controller/sbin
rootfs 716800 665728 51072 93% /dev
none 3145728 336664 2809064 11% /dev/shm
none 51200 2288 48912 5% /etc
none 2097152 682360 1414792 33% /isan/plugin/0/isan/utils
none 2097152 682360 1414792 33% /isan/plugin/0/lc/isan/utils
none 2097152 682360 1414792 33% /isan/plugin/0/lc/isan/lib
none 2097152 682360 1414792 33% /isan/plugin/0/isan/lib
none 2097152 682360 1414792 33% /isan/lib
none 2097152 682360 1414792 33% /isan/plugin/0/lib
none 2097152 682360 1414792 33% /isan/utils
rootfs 716800 665728 51072 93% /lc/isan/utils
rootfs 716800 665728 51072 93% /lib
rootfs 716800 665728 51072 93% /mnt/cfg
/dev/sda5 60485 5356 52006 10% /mnt/cfg/0
/dev/sda6 60485 5356 52006 10% /mnt/cfg/1
rootfs 716800 665728 51072 93% /mnt/ifc
/dev/sda9 15748508 591216 14357292 4% /mnt/ifc/cfg
rootfs 716800 665728 51072 93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
/dev/sda8 15748508 991204 13957304 7% /mnt/ifc/log
/dev/sda3 120979 13349 101384 12% /mnt/pss
rootfs 716800 665728 51072 93% /sbin
/dev/sda8 15748508 991204 13957304 7% /data/techsupport
none 1572864 39444 1533420 3% /tmp
rootfs 716800 665728 51072 93% /usr
none 51200 108 51092 1% /var/log
/dev/sda8 15748508 991204 13957304 7% /var/log/dme/oldlog
none 51200 108 51092 1% /var/log/messages
rootfs 716800 665728 51072 93% /var/log/dme
rootfs 716800 665728 51072 93% /var/log/dme/nginx
rootfs 716800 665728 51072 93% /usr/share/vim
/dev/sda7 11811760 375608 10836140 4% /var/log/dme/log
/dev/sda8 15748508 991204 13957304 7% /var/log/dme/oldlog
none 512000 1064 510936 1% /var/run/mgmt/log
none 512000 1064 510936 1% /var/run/utmp
/dev/sda7 11811760 375608 10836140 4% /var/sysmgr
none 40960 8 40952 1% /var/sysmgr/startup-cfg
/dev/sda7 11811760 375608 10836140 4% /logflash/core
rootfs 716800 665728 51072 93% /usb
none 512000 0 512000 0% /volatile

Monitoring CoPP (Control Plane Policing) Statistics

CLI

CoPP is enabled by default and the parameters cannot be changed at this time. CoPP statistics are available through the CLI.

To show the CoPP policy that is programed by the system use the following CLI command:


Leaf-1# show copp policy
COPP Class COPP proto COPP Rate COPP Burst
mcp mcp 500 500
ifc ifc 5000 5000
igmp igmp 1500 1500
nd nd 1000 1000
cdp cdp 1000 1000
pim pim 500 500
dhcp dhcp 1360 340
lacp lacp 1000 1000
ospf ospf 2000 2000
arp arp 1360 340
lldp lldp 1000 1000
acllog acllog 500 500
stp stp 1000 1000
eigrp eigrp 2000 2000
coop coop 5000 5000
traceroute traceroute 500 500
isis isis 1500 5000
icmp icmp 500 500
bgp bgp 5000 5000

To show drops against specific CoPP classes, use the following CLI command:


Leaf-1# show copp policy stats
COPP Class COPP proto COPP Rate COPP Burst AllowPkts AllowBytes DropPkts DropBytes
mcp mcp 500 500 0 0 0 0
ifc ifc 5000 5000 195072 161613961 0 0
igmp igmp 1500 1500 3 192 0 0
nd nd 1000 1000 6 564 0 0
cdp cdp 1000 1000 494 140543 0 0
pim pim 500 500 0 0 0 0
dhcp dhcp 1360 340 4 1400 0 0
lacp lacp 1000 1000 0 0 0 0
ospf ospf 2000 2000 0 0 0 0
arp arp 1360 340 1284 90068 0 0
lldp lldp 1000 1000 5029 1717208 0 0
acllog acllog 500 500 0 0 0 0
stp stp 1000 1000 0 0 0 0
eigrp eigrp 2000 2000 0 0 0 0
coop coop 5000 5000 4722 470546 0 0
traceroute traceroute 500 500 0 0 0 0
isis isis 1500 5000 17141 2167565 0 0
icmp icmp 500 500 0 0 0 0
bgp bgp 5000 5000 864 73410 0 0

Physical Interface Statistics and Link State

GUI

To access interface link state information, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Interfaces > Physical Interfaces. In the work pane, the oper state column displays the operational state of the link. Note that there are other tabs available in the work pane that reference other types of interfaces like port-channels, virtual port-channels, routed interfaces, loopbacks, etc.

To access interface statistics, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Interfaces > Physical interfaces > Eth X/Y, and then click on the Stats tab in the work pane. You can click on the check icon enables you to select additional statistics that can be graphed.

REST API

For customers that prefer the REST API interface to poll for interface statistics, several objects are available. There are several such counters that are available (e.g. RX/TX, input/output / duplex, 30 second rates, 5 minute rate, unicast packets, multicast packets, etc.). As a pointer, the parent managed object is provided below, as the children can be derived from it.

It is expected that the reader has a good understanding of the object model and is able to navigate through the model to obtain the information desiered using the example below, information provided in preceding sections and the tools described therein.

An example of the base API call for physical interface statistics is:


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/phys-[eth1/1].json

For example, to determine the total ingress bytes on Leaf 101 port Eth1/1, the ACME operator could issue the following API call:


/topology/pod-1/node-101/sys/phys-[eth1/1].json

Visore allows the operator to dig deeper into the hierarchical tree. From the prior command, the operator could see children of the interface object, such as ingress and egress bytes. One of the child objects includes the following:


/topology/pod-1/node-101/sys/phys-[eth1/1]/dbgEtherStats

CLI

The show interface eth x/y command can be used to monitor interfaces from the CLI. Other supported commands include "show interface port-channel x/y"


Leaf-1# show int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000/auto Ethernet, address: 7c69.f60f.8771 (bia 7c69.f60f.8771)
MTU 9000 bytes, BW 1000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 04:19:13
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 169328 bits/sec, 97 packets/sec
30 seconds output rate 424528 bits/sec, 115 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 644416 bps, 134 pps; output rate 365544 bps, 114 pps
RX
2474537 unicast packets 8434 multicast packets 2 broadcast packets
2482973 input packets 1686129815 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 712 input discard
0 Rx pause
TX
1673907 unicast packets 575 multicast packets 7 broadcast packets
1674489 output packets 455539518 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for interfaces. See: Cisco ACI MIB Support List.

Module Status

Even though the leaves are considered fixed switches, they have a supervisor component which refers to the CPU complex. From a forwarding perspective, there are two data plane components, viz. the NFE (Network Forwarding Engine ASIC) which provide the front panel ports, and the ALE or ALE2 (Application Leaf Engine ASIC) depending on the generation of switch hardware, which provides uplink connectivity to the spines. The following methods can be used to determine the status of the modules in the switch.

GUI

To access module status for the NFE and the CPU complex, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Chassis > Module > Supervisor modules and the status of the module is displayed in the work pane.

To access module status for the ALE/ALE2, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Chassis > Module > Line modules and the status of the module is displayed in the work pane.

REST API

The following REST API call(s) can be used to monitor the state of the supervisor and the module.


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/supslot-1/sup
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/lcslot-1/lc

CLI

The show module command can be used to obtain the status of the base module and the uplink module.


Leaf-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 48 1/10G Ethernet Module N9K-C9396PX active
GEM 12 40G Ethernet Expansion Module N9K-M12PQ ok
Mod Sw Hw
--- -------------- ------
1 11.1(0.152) 0.2050
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 7c-69-f6-0f-87-71 to 7c-69-f6-0f-87-ad SAL17267Z9U
Mod Online Diag Status
--- ------------------
1 pass

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for interfaces. See: Cisco ACI MIB Support List.

Switch Fan Status

The following section describes methodologies to retrieve the status of the fan trays on the leaf switches.

GUI

To access fan status for the leaf switch, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Chassis > Fan Tray and the status of the modules is displayed in the work pane.

REST API

The following REST API call(s) and their child objects can be used to monitor the state of the fans on a leaf switch (note that there are 3 slots on this particular switch).


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/ftslot-1
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/ftslot-2
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/ftslot-3

CLI

The following CLI command can be used to monitor the state of the fans on a leaf switch.


Leaf-1# show environment fan
Fan:
------------------------------------------------------
Fan Model Hw Status
------------------------------------------------------
Fan1(sys_fan1) N9K-C9300-FAN1-B -- ok
Fan2(sys_fan2) N9K-C9300-FAN1-B -- ok
Fan3(sys_fan3) N9K-C9300-FAN1-B -- ok
Fan_in_PS1 -- -- unknown
Fan_in_PS2 -- -- ok
Fan Speed: Zone 1: 0x5f
Fan Air Filter : Absent

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for fans. See: Cisco ACI MIB Support List.

Switch Power Supply Status

The following sections describe methodologies to retrieve the status of the power supplies on the leaf switches.

GUI

To access power supply status for the leaf switch, in the APIC GUI, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Chassis > Power Supply Units and the status of the modules is displayed in the work pane.

REST API

The following REST API call(s) and their child objects can be used to monitor the state of the fans on a leaf switch (note that there are 3 slots on this particular switch).


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-1
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-2

CLI

The following CLI commands can be used to monitor the state of the fans on a leaf switch:


Leaf-1# show environment power
Power Supply:
Voltage: 12.0 Volts
Power Actual Total
Supply Model Output Capacity Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 UCSC-PSU-650W 0 W 648 W shut
2 UCSC-PSU-650W 168 W 648 W ok
Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
-------- ------------------- ----------- ----------- --------------
1 N9K-C9396PX 168 W 456 W Powered-Up
fan1 N9K-C9300-FAN1-B N/A N/A Powered-Up
fan2 N9K-C9300-FAN1-B N/A N/A Powered-Up
fan3 N9K-C9300-FAN1-B N/A N/A Powered-Up
N/A - Per module power not available
Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) Non-Redundant(combined)
Power Supply redundancy mode (operational) Non-Redundant(combined)
Total Power Capacity (based on configured mode) 648 W
Total Power of all Inputs (cumulative) 648 W
Total Power Output (actual draw) 168 W
Total Power Allocated (budget) N/A
Total Power Available for additional modules N/A

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for power supplies. See: Cisco ACI MIB Support List.

LLDP Neighbor Status

The APIC provides a single pane of glass to query and determine all LLDP neighbors in a fabric.

GUI

To obtain a list of LLDP neighbors on an interface, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Protocols > LLDP > Neighbors > eth x/y.

A full listing of all LLDP neighbors on the interface can be obtained in the work pane.

REST API

The following rest API call can be used to obtain the same information:


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/lldp/inst/if-[eth1/1]

CLI


Leaf-1# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic2 Eth1/1 120 4c:4e:35:09:77:2f
apic3 Eth1/4 120 90:e2:ba:4b:fa:d4
5548-2 Eth1/7 120 B Eth1/3
Spine-1 Eth1/49 120 BR Eth4/1
Spine-2 Eth1/50 120 BR Eth4/1
Spine-3 Eth1/51 120 BR Eth1/3
Spine-4 Eth1/52 120 BR Eth1/3
Spine-5 Eth1/53 120 BR Eth1/5
Spine-6 Eth1/54 120 BR Eth1/5
Total entries displayed: 9

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for LLDP. See: Cisco ACI MIB Support List.

CDP Neighbor Status

The APIC provides a single pane of glass to query and determine all CDP neighbors in a fabric.

GUI

To obtain a list of LLDP neighbors on an interface, navigate to Fabric > Inventory > Pod-1 > Leaf-X > Protocols > CDP > Neighbors > eth x/y.

A full listing of all CDP neighbors on the interface can be obtained in the work pane.

In the above workflow clicking on Neighbors (instead of eth x/y) gives you a list of all LLDP neighbors on the switch.

REST API

The following rest API call can be used to obtain the same information:


https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/cdp/inst/if-[eth1/1]

CLI


Leaf-1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
Services-UCS-A(SSI15450J63)
Eth1/5 129 S I s UCS-FI-6248UP Eth1/17
5548-2(SSI154300VL)
Eth1/7 123 S I s N5K-C5548UP Eth1/3

SNMP

As mentioned in the URL for the SNMP reference guide for ACI release 1.x, the following SNMP objects are supported from an SNMP polling perspective for CDP. See: Cisco ACI MIB Support List.

GOLD Diagnostic Results

GOLD is covered in greater detail in the Hardware Replacement section.

GOLD diagnostics provide an easy and quick way for operations teams to confirm that bootup and non-disruptive tests that run during normal operations have executed properly, as well as the ability to run on demand diagnostics to isolate potential hardware at fault.

GUI

To view GOLD Diagnostic test results in the GUI for the Supervisors, click on Fabric > Inventory > Pod-1 > Leaf-1 > Chassis > Supervisor Modules > Slot-1. Then click troubleshooting in the work pane.

To view the same for modules, click on Fabric > Inventory > Pod-1 > Leaf-1 > Chassis > Line Modules > Slot-x. Then click Troubleshooting in the work pane.

CLI


Leaf-1# show diagnostic result module all
Current bootup diagnostic level: bypass
Module 1: 1/10G Ethernet Module (Active)
Test results: (. = Pass, F = Fail, I = Incomplete,
U = Untested, A = Abort, E = Error disabled)
1) bios-mem-----------------------> .
2) mgmtp-lb-----------------------> .
4) nsa-mem------------------------> .
6) fabp-prbs:
Port 1 2 3 4 5 6 7 8 9 10 11 12
-----------------------------------------
. . . . . . . . . . . .
22) cpu-cache----------------------> .
23) mem-health---------------------> .
24) ssd-acc------------------------> .
25) act2-acc-----------------------> .
26) ge-eeprom----------------------> .
29) usb-bus------------------------> .
30) cons-dev-----------------------> .
31) obfl-acc-----------------------> .
32) nvram-cksum--------------------> .
33) fpga-reg-chk-------------------> .
34) asic-scratch-------------------> .
40) rtc-test-----------------------> .
41) pcie-bus-----------------------> .

Monitoring Workload Bandwidth

ACME would like to proactively monitor connections to servers to determine whether adequate bandwidth is available to a given workload. This enables them to answer questions about whether a server needs to be upgraded from 10G to 40G, or multiple interfaces need to be bonded to provide more bandwidth to the server. The following will provide an example of how statistics policies and thresholds can be used to alert the operators when additional bandwidth is required to a server.

ACME's server team has determined that they would like to monitor link utilization over a 5 minute period, and when the average utilization is above 80%, they would like to raise a fault for the affected interface. To accomplish these tasks requires configuring a monitoring policy with two distinct types of policies. First, ACME will configure a stats collection policy to collect the average utilization of a link over the desired 5 minute interval. Next they will configure a threshold policy to generate a fault when the link utilization exceeds various thresholds according to the following table.

Create an interface monitoring policy.

We have now created a policy which will monitor associated interfaces and ingress traffic rate at 5 minute intervals. Next we will configure a threshold which will alert us if the utilization exceeds 80% and assign an default fault severity to it.

Note: In this example, we are interested in an absolute percentage, these policies can be further customized to provide a normal value, and look for deviations above or below that normal value.

In this scenario, 80% utilization does not necessarily mean that the application performance is degraded, so we will flag this as a warning. Additional levels/severities can also be specified if desired.

The set column specifies the level at which the fault will be raised, and the reset column will specify the the level at which the fault will be cleared. For example, we will be raising a warning when the utilization goes above 80, and clear the warning when the utilization falls below 75. Repeat these steps for the egress statistics as well.

Finally, we will associate the newly created policy with an interface policy group that represents the interfaces we to monitor with this policy.

For our example, we will apply the policy to the UCS-10G-PG

EPG Level Statistics

The application owner would like to be able to monitor network-related information for their application, such as the aggregate amount of traffic to a specific tier. As an example, we will monitor the amount of traffic to the web tier of a given application. In this example, the default monitoring policies are appropriate, and they are simply extracting them from the system to be consumed externally. This information is useful in scenarios such as a new release being pushed, and to make sure that no traffic anomalies are created after the push.

REST API

Additionally, this information can be gathered from the API:


http[s]://apic_ip/api/node/mo/uni/tn-mynewproject/ap-app1/epg-web-epg.xml?querytarget=
self&rsp-subtree-include=stats

Reactive Monitoring

It is crucial that the ACME operational staff are able to react to any indication of something going wrong. If there is a notification that something has gone wrong, such as a fault notification, a low health score, or a ticket/report that end-user functionality has been impacted, knowledge of the available monitoring tools is important for the identification and collection of evidence. This is especially true the box by box approach to troubleshooting gets replaced by system-wide approach. It is vital to understand the tools that are available to assist you with your troubleshooting efforts, which take a top/down approach and effectively zoom in to the problem. You can then use this evidence to identify and analyze the root cause of the problem before taking corrective action. For more information regarding faults and health scores please refer to those specific sections within this book.

A deep dive into the processes of troubleshooting is out of the scope of this book. Please refer to Troubleshooting Cisco Application Centric Infrastructure, "Analytical problem solving applied to the policy driven data center" available at: http://aci-troubleshooting-book.readthedocs.org/en/latest/

Tenant—Troubleshoot Policies

Within the Application Policy Infrastructure Controller (APIC) GUI, under each tenant you can find a Troubleshoot Policy section. This section allows configuration of policies that are specific to one tenant, and the monitoring of traffic and test connectivity between endpoints.

As seen in the image above, the following troubleshooting policies can be configured:

SPAN (Switched Port ANalyzer)—Configuration of SPAN and ERSPAN sources and destinations to be used in external monitoring of Tenant traffic flows.
Endpoint-To-Endpoint Traceroute—Configuration of a path validation tool for verifying validity of communications between Tenant endpoints in an Cisco Application Centric Infrastructure (ACI) fabric.
Endpoint-To-External-IP Traceroute—Configuration of a path validation tool for verifying validity of communications between a tenant endpoint and an external device outside of the ACI fabric.
Atomic Counters—Configuration of a set of customizable counters to collect and report on information between a definable set of objects. A policy can be configured to collect statistics between endpoints, between endpoint groups, between endpoint groups and specific IPs and other special objects, such as Any or External traffic flows.

Fabric—Troubleshoot Policies

For troubleshooting within the entire fabric, there are the following tools and policies:

SPAN (Switched Port Analyzer)—Configuration of SPAN and ERSPAN sources and destinations to be used in external monitoring of fabric traffic flows.
On-demand Diagnostics—Configuration of a policy for collection of diagnostic information that can be executed at a point in time and which will return a set of valuable output for investigation.
Leaf Nodes Traceroute—Configuration of a path validation tool for verifying validity of communications between ACI fabric nodes. The traceroute for a leaf node allows you to determine the path a packet takes to get to a destination from a given source by returning the sequence of hops the packet traversed.
Traffic Map—As of release 1.2, this tool has been moved to the Operations tab and is renamed Visualization. It still provides an at-a-glance hotspot map of node-to-node traffic flow in an ACI fabric.

Other Tools

iPing—A troubleshooting tool in the ACI fabric that can be used to verify reachability of a device connected to the fabric utilizing the fabric as the pervasive source.
Audit Logs—Audit logs are continually collected on all actions taken in an ACI fabric and can give a quick indication of which user took which actions at what time.

Reactive Monitoring Use Cases

At the end of this chapter, we are going to describe two situations where ACME ran into issues and how they made use of the tools previously described.

No access to application: An end-user calls and reports that they can no longer access a web application running within the fabric.
Users report that an application running in the fabric is slow or there is a report of slowness to the web application running within the fabric.

Operations Tab

The Operations tab provides a single location that includes several commonly used tools and outputs required for troubleshooting endpoint connectivity. The Operations tab contains the following subtabs:

Enhanced Troubleshooting Wizard (Visibility & Troubleshooting) tab—Quickly identifies connectivity issues when troubleshooting connectivity between endpoints within the fabric. The Enhanced Troubleshooting Wizard provides a single location that includes several commonly used tools and outputs required for troubleshooting end point connectivity.
Capacity Dashboard tab—Gets a summary of critical fabric resource thresholds.
ACI Optimizer tab—Enables you to enter your network requirements to determine how many leafs you will need for your network, and to learn how to deploy each application and external endpoint group on each leaf without violating any constraints.
EP Tracker tab—Allows you to enter an endpoint IP or MAC address, and quickly see the location of this endpoint, the endpoint group that it belongs to, the VLAN encapsulation used, and if any transitions (flaps) have occurred for this endpoint.
Visualization tab—Allows you to view traffic statistics for a set of spine and leaf switches.

Visibility and Troubleshooting—Enhanced Troubleshooting Wizard

The Enhanced Troubleshooting Wizard provides a visual view of the connectivity between two endpoints. In releases prior to 1.2(1x), the endpoints must reside in the same VRF and tenant space. Starting in the 1.2(1x) release, the endpoints must be in the same tenant, but the bridge domain and VRF can be in the Common tenant.

Other scenarios such as inter-tenant can also be viewed, but many of the tools that comprise the Enhanced Troubleshooting Wizard will not work, such as atomic counters and traceroute, due to hardware-based topology restrictions on those features. This is unrelated to the Enhanced Troubleshooting Wizard tool itself.

In the menu bar, choose Operations > Visibility and Troubleshooting to launch the enhanced troubleshooting wizard.
In the Name field, enter a name for the new session or choose an existing session.
In the Source field, enter the MAC or IP address for the source endpoint and click Search.
(Optional) If the endpoint is external, put a check in the check box.
Click on the endpoint once it is displayed.
In the Destination field, enter the MAC or IP address for the destination endpoint and click Search.
(Optional) If the endpoint is external, put a check in the check box.
Click on the endpoint once it is displayed.
Click Start.
Choose the Drop/Stats tab to view the next screen. You can choose the other tabs to view additional information about these endpoints. You can enter various endpoint topologies, but not all of the available tools that are contained in the Enhanced Troubleshooting Wizard will work.

Capacity Dashboard

The Capacity Dashboard tab can be used to get a summary of critical fabric resource thresholds. This allows you to see quickly how close you are to reaching the approved scalability limits. Per leaf usage is also shown, allowing you to see quickly which leaf may be hitting resource constraints.

In the menu bar, choose Operations > Capacity Dashboard to launch the Capacity Dashboard troubleshooting tool.

Note

Disabling the 5-minute interval on the fabric default monitoring policy causes less information to appear on the Capacity Dashboard.

Endpoint Tracker

The Endpoint Tracker tab allows you to enter a fabric-attached endpoint IP or MAC address and quickly see the location of this endpoint, the endpoint group it belongs to, the VLAN encapsulation used, and if any transitions (flaps) have occurred for this endpoint.

In the menu bar, click Operations > EP Tracker to launch the Endpoint Tracker troubleshooting tool
In the End Point Search field, enter the IP address or MAC address of the endpoint and click Search.
Click on the endpoint once it is displayed.

The Endpoint Tracker tool displays the date and time of each state transition along with the IP address, MAC address, owning endpoint group, action (attached or detached), physical node, Interface, and VLAN encapsulation during the event.

Switch Port Analyzer (SPAN)

SPAN is generally used in two ways:

Proactively as part of a third party or offline analysis requirement.
- Security tools (IDS/IPS, Data Loss Prevention)
- Call Recording
Troubleshooting application and networking issues within the fabric.

It may be helpful to perform a capture of some particular traffic to see what is going on within the stream of data. Looking through traffic flows will allow investigation of packet and protocol level details, such as traffic resets, misbehaving protocols, improper host requests or responses, or node level communications. This will provide deeper insight into how devices are using the network than simple traffic flow and fabric configuration review.

Switched Port ANalyzer, or SPAN, is a standard feature that allows copy and replication of traffic to a network analyzer for further decoding and investigation. It can be used to copy traffic from one or more ports, VLANs, or endpoint groups (EPGs).

The SPAN feature process is non-disruptive to any connected devices and is facilitated in hardware, which prevents any unnecessary CPU load.

SPAN sessions can be configured to monitor traffic received by the source (ingress traffic), traffic transmitted from the source (egress traffic), or both. By default, SPAN monitors all traffic, but filters can be configured to monitor only selected traffic.

Multinode SPAN

APIC traffic monitoring policies can enforce SPAN at the appropriate places to copy traffic from members of each End Point Group wherever they are connected. If a member moves, APIC automatically pushes the policy to the new leaf switch. For example, when a VMotion event relocates an Endpoint to a new leaf switch, the SPAN feature configuration automatically adjusts.

SPAN Guidelines and Restrictions

Use SPAN for troubleshooting. SPAN traffic competes with user traffic for switch resources. To minimize the load, configure SPAN to copy only the specific traffic that you want to analyze.
An l3extLIfP Layer 3 subinterface cannot be configured as a SPAN source. The entire port must be used for monitoring traffic from external sources.
Tenant and access SPAN use the encapsulated remote extension of SPAN (ERSPAN) type I, while fabric SPAN uses ERSPAN type II. For information regarding ERSPAN headers, refer to the IETF Internet Draft at this URL: https://tools.ietf.org/html/draft-foschiano-erspan-00
See the Verified Scalability Guide for Cisco ACI document for SPAN-related limits, such as the maximum number of active SPAN sessions.

Configuring a SPAN Session

This procedure shows how to configure a SPAN policy to forward replicated source packets to a remote traffic analyzer.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Policies > Troubleshoot > SPAN > SPAN Destination Groups.
In the Work pane, choose Actions > Create SPAN Destination Group.
In the Create SPAN Destination Group dialog box, perform the following actions:
1. In the Name field, enter a name for the SPAN destination group.
2. In the Destination EPG dropdowns, select the destination Tenant, Application Profile, and EPG.
3. Enter the Destination IP.
4. Enter the Source IP Prefix.
5. Optionally, modify the other fields as needed.
6. Click OK.
7. If needed, add additional destinations.
8. Click Submit.
Under SPAN, right-click SPAN Source Groups and choose Create SPAN Source Group.
In the Create SPAN Source Group dialog box, perform the following actions:
1. In the Name field, enter a name for the SPAN source group.
2. From the Destination Group drop-down list, choose the SPAN destination group that you configured previously.
3. In the Create Sources table, click the + icon to open the Create Sources dialog box.
4. In the Name field, enter a name for the source.
5. In the Direction field, choose the radio button based on whether you want to replicate and forward packets that are incoming to the source, outgoing from the source, or both incoming and outgoing.
6. From the Source EPG drop-down list, choose the EPG (identified by Tenant/ApplicationProfile/EPG) whose packets will be replicated and forwarded to the SPAN destination. Click OK to save the SPAN source.
7. Click Submit to save the SPAN source group.

Traceroute

The traceroute tool is used to discover the routes that packets actually take when traveling to their destination. Traceroute identifies the path taken on a hop-by-hop basis and includes a time stamp at each hop in both directions. You can use traceroute to test the connectivity of ports along the path between the generating device and the device closest to the destination. If the destination cannot be reached, the path discovery traces the path up to the point of failure. Traceroute is a useful feature in traditional networking. In Cisco Application Centric Infrastructure (ACI), this feature is implemented taking into account the way the fabric works.

Traceroute supports a variety of modes, including endpoint-to-endpoint, and leaf-to-leaf (tunnel endpoint, or TEP to TEP). It discovers all paths across the fabric, discovers point of exits for external endpoints, and helps to detect if any path is blocked.

A traceroute that is initiated from the tenant endpoints shows the default gateway as an intermediate hop that appears at the ingress leaf switch.

Note

If traceroute is done from the OS of a connected server or VM, it will show the hops for the leaves and spines as unknown, and will keep recording the information after the packet gets out of the fabric. For more precise information, use traceroute from the Application Policy Infrastructure Controller (APIC) (GUI or CLI).

Traceroute Guidelines and Restrictions

When the traceroute source or destination is an endpoint, the endpoint must be dynamic and not static. Unlike a dynamic endpoint (fv:CEp), a static endpoint (fv:StCEp) does not have a child object (fv:RsCEpToPathEp) that is required for traceroute.
See the Verified Scalability Guide for Cisco ACI document for traceroute-related limits.
Traceroute results will display the IP address of the remote node and interface which it came it came in on. In the APIC GUI, go to Fabric > Inventory > Fabric Management to view the IP address information for correlation.
Traceroute cannot be used for endpoints that reside in an external endpoint groups.

Performing a Traceroute Between Endpoints

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Policies > Troubleshoot > Endpoint-to-Endpoint Traceroute Policies.
In the Work pane, choose Actions > Create Endpoint-to-Endpoint Traceroute Policy.
In the Create Endpoint-to-Endpoint Traceroute Policy dialog box, perform the following actions:
1. In the Name field, enter a name for the traceroute policy.
2. In the Source End Points table, click the + icon to edit the traceroute source.
3. From the Source MAC drop-down list, choose or enter the MAC address of the source endpoint and click Update.
4. In the Destination End Points table, click the + icon to edit the traceroute destination.
5. From the Destination MAC drop-down list, choose or enter the MAC address of the destination endpoint and click Update.
6. In the State field, click the Start radio button.
7. Click Submit to launch the traceroute.
In the Navigation pane or the Traceroute Policies table, click the traceroute policy. The traceroute policy is displayed in the Work pane.
In the Work pane, click the Operational tab, click the Source End Points tab, and click the Results tab.
In the Traceroute Results table, verify the path or paths that were used in the trace.
1. More than one path might have been traversed from the source node to the destination node.
2. For readability, increase the width of one or more columns, such as the Name column.

Atomic Counters

Atomic Counters are useful for troubleshooting connectivity between endpoints, EPGs, or an application within the fabric. A user reporting application may be experiencing slowness, or atomic counters may be needed for monitoring any traffic loss between two endpoints. One capability provided by atomic counters is the ability to place a trouble ticket into a proactive monitoring mode, for example when the problem is intermittent, and not necessarily happening at the time the operator is actively working the ticket.

Atomic counters can help detect packet loss in the fabric and allow the quick isolation of the source of connectivity issues. Atomic counters require NTP to be enabled on the fabric.

Leaf-to-leaf (TEP to TEP) atomic counters can provide the following:

Counts of drops, admits, and excess packets
Short-term data collection such as the last 30 seconds, and long-term data collection such as 5 minutes, 15 minutes, or more
A breakdown of per-spine traffic (available when the number of TEPs, leaf or VPC, is less than 64)
Ongoing monitoring

Leaf-to-leaf (TEP to TEP) atomic counters are cumulative and cannot be cleared. However, because 30 second atomic counters reset at 30 second intervals, they can be used to isolate intermittent or recurring problems.

Tenant atomic counters can provide the following:

Application-specific counters for traffic across the fabric, including drops, admits, and excess packets
Modes include the following:
Endpoint to endpoint MAC address, or endpoint to endpoint IP address. Note that a single target endpoint could have multiple IP addresses associated with it.
EPG to EPG with optional drill down
EPG to endpoint
EPG to * (any)
Endpoint to external IP address

Note

Atomic counters track the amount packets of between the two endpoints and use this as a measurement. They do not take into account drops or error counters in a hardware level.

Dropped packets are calculated when there are less packets received by the destination than transmitted by the source.

Excess packets are calculated when there are more packets received by the destination than transmitted by the source.

Configuring Atomic Counters

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Troubleshooting Policies > Atomic Counter Policy.
Choose a policy (traffic topology). Traffic can be measured between a combination of endpoints, endpoint groups, external interfaces, and IP addresses.
In the Work pane, choose Actions > Add Policy_Name Policy.
In the Add Policy dialog box, perform the following actions:
1. In the Name field, enter a name for the policy.
2. Choose or enter the identifying information for the traffic source. The required identifying information differs depending on the type of source (endpoint, endpoint group, external interface, or IP address).
3. Choose or enter the identifying information for the traffic destination.
4. (Optional) In the Filters table, click + to specify filtering of the traffic to be counted. In the resulting Create Atomic Counter Filter dialog box, specify filtering by the IP protocol number (TCP=6, for example) and by source and destination IP port numbers.
5. Click Submit to save the atomic counter policy.
In the Navigation pane, under the selected topology, choose the new atomic counter policy. The policy configuration is displayed in the Work pane.
In the Work pane, choose the Operational tab and choose the Traffic subtab to view the atomic counter statistics.

Traffic Map

Note

In the 1.2 release, this was moved to the Operations tab and renamed "Visualization".

Low performance and congestion can be identified by use of Traffic maps.

Traffic maps make use of atomic counters to continuously monitor and display traffic between leaf switches to help with quick debugging and isolation of application connectivity issues.

Configuring Traffic Map

On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, choose Troubleshooting Policies > Traffic Map.
Set the drop-down menu options to view the source Leaf to destination Leaf traffic paths.
- Last interval and cumulative
- Sent, received, dropped and excess
- All spines and a specific spine switch
The percentage is shown in relative terms to all traffic by Source or received by Destination.
Clicking on a cell opens a table with all data for all trails and links.

IPing

IPing is used to test and validate connectivity within the from leaf node to endpoints within the fabric, taking into account the private network. IPing is a troubleshooting tool for network users similar to the PING command.

Using IPing

iping [ -V vrf ] [ -c count ] [ -i wait ] [ -p pattern ] [ -s packetsize ] [ -t timeout ] host

Examples

pod1-leaf1# iping -V overlay-1 10.0.59.154
PING 10.0.59.154 (10.0.59.154): 56 data bytes
64 bytes from 10.0.59.154: icmp_seq=0 ttl=55 time=0.254 ms
64 bytes from 10.0.59.154: icmp_seq=1 ttl=55 time=0.256 ms
64 bytes from 10.0.59.154: icmp_seq=2 ttl=55 time=0.245 ms
64 bytes from 10.0.59.154: icmp_seq=3 ttl=55 time=0.241 ms
64 bytes from 10.0.59.154: icmp_seq=4 ttl=55 time=0.23 ms
--- 10.0.59.154 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.23/0.245/0.256 ms

Audit Logs

At times it may be required to view changes which have taken place in the fabric. An outage reported on a host or application in the fabric may need to be tracked, or data pulled for an audit requirement.

Audit logs are records of who made a change, when the change was made, and a description of the action. Audit logs also record when users logon and logoff.

Audit logs can be found in several places within the GUI, filtered to show only those events relevant to the current GUI context. Wherever a History tab appears in the GUI Work pane, the audit log can be viewed. This procedure shows how to view tenant events as an example.

Viewing Audit Logs

This procedure shows how to view an audit log on a tenant.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose Common.
In the Navigation pane, choose Common.
In the Work pane, choose the History tab.
Under the History tab, choose the Events subtab to view the event log.
Under the History tab, choose the Audit Log subtab to view the audit log.
Double-click a log entry to view additional details about the event.

Reactive Monitoring Use Cases

This chapter shows some examples of how ACME's operations teams can use their Cisco Application Centric Infrastructure (ACI) monitoring tools to react to a few common possible scenarios.

Note

These examples assume that basic low-level investigation has been done and the issue has been isolated to an issue with traffic flows across the fabric. Cables and connectivity have been verified, hosts are up, virtual machines are running, processes are running, memory and CPU utilization has been checked, and so on.

A great way to begin in any use case is to use the Visibility and Troubleshooting Tool (Enhanced Troubleshooting Wizard), which is a built-in capability of the Application Policy Infrastructure Controller (APIC).

Loss of Connectivity to Endpoint

The ACME application owner has reported that two servers have lost connectivity. These two endpoints (EPs) belong to two different endpoint groups (EPGs), within the same subnet, bridge domain and Tenant, and each endpoint is connected to different leaf switches. The bridge domain has unicast routing enabled, therefore the default gateway for both endpoints exist on the Cisco Application Centric Infrastructure (ACI) leaf switches.

The following steps troubleshoot this situation:

Check that the endpoints have been learned by the leaf switches.
Verify that the required contracts are in place between the endpoints.

Checking that the Endpoints Have Been Learned by the Leaf Switches

You can use the Application Policy Infrastructure Controller (APIC) GUI to check that the endpoints have been learned by the leaf switches.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Application Profiles > App_Profile_Name > Application EPGs > EPG_Name.
In the Work pane, choose the Operational > Client End-Points tabs.
Verify that the endpoint is present
Repeat steps 1 to 5 for the destination endpoint group.

Alternately, you can use the End Point Tracker tool to check that the endpoints have been learned by the leaf switches.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Application Profiles > App_Profile_Name > Application EPGs > EPG_Name.
In the Work pane, choose the Operational > EP Tracker tabs.
Verify the location of the relevant endpoints and view the history for those endpoints.

Verifying that the Required Contracts are in Place Between the Endpoints

You can use the Application Policy Infrastructure Controller (APIC) GUI to verify that the required contracts are in place between the endpoints.

On the menu bar, choose Tenants > ALL TENANTS.
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Application Profiles > App_Profile_Name > Application EPGs > EPG_Name.
In the Work pane, choose the Operational > Contracts tabs.
Check for a relationship between the source endpoint group and destination endpoint group, noting the type because it states the direction, such as provider/consumer. Additionally, the type states the contract subject that is in use.
Check the egress/ingress cumulative packets for each subject and relevant filters.
Inspect the contents of each filter by examining the contract under the Security Policies folder and verifying that each filter contains the appropriate filter entries.
If the endpoints are discovered within each endpoint group and the contract relationships look correct, examine the troubleshooting policies.

The following alternate techniques are available to validate communications between endpoints within the ACI fabric:

Use the Visibility and Troubleshooting Tool.
Use endpoint-to-endpoint traceroute to show if there are paths available between those endpoints.
Inside the fabric, use the iPing tool to verify connectivity between the default gateway and the endpoints, from the CLI. Each leaf has an SVI used as the default gateway (assuming this is the configuration with a subnet defined at the bridge domain/endpoint group level). If this test is successful, the connectivity between endpoints and leaf switches is not the problem. To check the connectivity to the remote end, use iPing from each leaf to the remote endpoint, using the default gateway as the source.

If you still cannot locate the issue, you can use SPAN to verify where the traffic is entering and leaving the fabric, and make sure that the frames have the correct format.

Users Report that an Application Running in the Fabric is Slow

ACME test users report slow response of the web servers running in the fabric. This performance issue could be caused due to latency, packet drops, or intermittent connectivity loss. In this case, the end-user is trying to access the web portal from a test virtual machine. The virtual machine and the web portal belong to a different EPGs in the same bridge domain and tenant.

First the operations staff should make sure endpoints are learned by the leaf switches in a consistent way so that the EPs are visible in the operational tab under the Tenant > Application Profile > EPGs > Operational.

Once they verify endpoints have been learned, they can use EP-to-EP Traceroute to show the path validity. Atomic counters can also be deployed to check if there are any drops or irregularities between the defined devices. Looking through the Operations > Visualization tool can also show an at-a-glance view in the fabric and highlight any possible hotspots or congestion in certain elements of the fabric. Check specific counters for CRC errors on the interfaces used to transmit EP specific traffic.

If everything in the fabric looks fine, SPAN may be used to verify where the traffic is entering and leaving the fabric and make sure frames have the right format. Corrupted frames could cause drops in different points, from the EPs on a OS level to the switches.

Results

Chapter: Monitoring

Monitoring

Proactive Monitoring - Tenant and Fabric Policies

Create Tenant Monitor Policy

Stats Collection Policies

Stats Export Policies

Diagnostics Policies Using the GUI

Call Home/SNMP/Syslog

Event Severity and Fault Severity Assignments

Fault Lifecycle Policies

TCAM Policy Usage

Create TCAM Policy Monitor

TCAM Prefix Usage

Health Score Evaluation Policy

Communication Policy

CPU Utilization and Memory

GUI

REST API

CLI

Disk Utilization

GUI

REST API

CLI

Physical and Bond Interface Statistics

GUI

REST API

CLI

Fan Status

GUI

REST API

CLI

Temperature Status

GUI

REST API

CLI

Power Supply Status

CLI

Monitoring Switch CPU Utilization

GUI

REST API

CLI

Monitoring Switch Memory Utilization

REST API

CLI

SNMP

Monitoring File System Health

CLI

Monitoring CoPP (Control Plane Policing) Statistics

CLI

Physical Interface Statistics and Link State

GUI

REST API

CLI

SNMP

Module Status

GUI

REST API

CLI

SNMP

Switch Fan Status

GUI

REST API

CLI

SNMP

Switch Power Supply Status

GUI

REST API

CLI

SNMP

LLDP Neighbor Status

GUI

REST API

CLI

SNMP

CDP Neighbor Status

GUI

REST API

CLI