Troubleshoot a Slow APIC GUI

Introduction

This document describes the general methodology to troubleshoot a slow APIC GUI experience.

Quick Start

It is frequently found that slow APIC GUI issues are the result of a high rate of API Requests sourced from a script, integration, or application. The access.log of an APIC logs each processed API request. The access.log of an APIC can be quickly analyzed with the Access Log Analyzer script within the Github Datacenter group aci-tac-scripts project.

Background Information

APIC as a Webserver - NGINX

NGINX is the DME responsible for the API endpoints available on each APIC. If NGINX is down, API requests cannot be handled. If NGINX is congested, the API is congested. Each APIC runs its own NGINX process, so it is possible that only a single APIC can have NGINX problems if only that APIC is targeted by any aggressive queriers.

The APIC UI performs multiple API requests to populate each page. Similarly, all APIC 'show' commands (NXOS Style CLI) are wrappers for python scripts that perform multiple API requests, handle the response, then serve it to the user.

Relevant Logs

Methodology

Isolate Initial Trigger

What is affected?

• Which APICs are affected; one, many, or all APICs?
• Where is slowness seen; via UI, CLI commands, or both?
• Which specific UI pages or commands are slow?

How is the slowness experienced?

• Is this seen across multiple browsers for a single user?
• Do multiple users report slowness or just a single/subset of users?
• Do the affected users share a similar geographic location or network path from browser to APIC?

When was the slowness first noticed?

• Was an ACI integration or script recently added?
• Was a browser extension recently enabled?
• Was there a recent change in ACI configuration?

Check NGINX Usage and Health

Access.log Entry Format

access.log is a feature of NGINX and is, therefore, APIC agnostic. Each line represents 1 HTTP Request that the APIC received. Reference this log to understand the NGINX usage of an APIC.

The default access.log format on ACI version 5.2+:

log_format proxy_ip '$remote_addr ($http_x_real_ip) - $remote_user [$time_local]'
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

This line represents an access.log entry when a moquery -c fvTenant is performed:

127.0.0.1 (-) - - [07/Apr/2022:20:10:59 +0000]"GET /api/class/fvTenant.xml HTTP/1.1" 200 15863 "-" "Python-urllib"

Map of example access.log entry to log_format:

Access.log Behaviors

High rate request bursts over a large period of time:

• Continual bursts of 40+ requests per second can cause UI slowness
• Identify which host(s) are responsible for the queries
• Reduce or disable the source of queries to see if this improves APIC response time.

Consistent 4xx or 5xx responses:

• If found, identify the error message from nginx.bin.log

Check NGINX Resource Usage

NGINX CPU and memory usage can be checked with the top command from the APIC:

top - 13:19:47 up 29 days, 2:08, 11 users, load average: 12.24, 11.79, 12.72
Tasks: 785 total, 1 running, 383 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.5 us, 2.0 sy, 0.0 ni, 94.2 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 13141363+total, 50360320 free, 31109680 used, 49943636 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 98279904 avail Mem

PID   USER PR NI VIRT    RES   SHR   S %CPU %MEM TIME+     COMMAND 
21495 root 20 0  4393916 3.5g 217624 S 2.6  2.8  759:05.78 nginx.bin

High NGINX resource usage can directly correlate to a high rate of processed requests.

Check for Cores

An NGINX crash is not typical for Slow APIC GUI issues. However, if NGINX cores are found, attach them to a TAC SR for analysis. Refer to the ACI Techsupport guide for steps to check for cores.

Check Client to Server Latency

If rapid requests are not found but a user continues to exhibit UI slowness, the issue can be Client (browser) to Server (APIC) latency.

In these scenarios, validate the data path from the browser to the APIC (Geographic distance, VPN, etc). If possible, deploy and test access from a jump server located in the same geographic region or Data Center as the APICs to isolate. Validate if other users exhibit a similar amount of latency.

Browser Development Tools Network Tab

All Browsers have the ability to validate HTTP requests and responses via its Browser Development toolkit, typically within a Network tab.

This tool can be used to validate the amount of time it takes for each stage of browser-sourced requests as shown in the image.

Example of the Browser Waiting 1.1 minutes for the APIC to Respond

Enhancements for Specific UI Pages

Policy Group page:

Cisco bug ID CSCvx14621 - APIC GUI loads slowly on IPG policies in the Fabric tab.

Interface under Inventory page:

Cisco bug ID CSCvx90048 - Initial load of "Layer 1 Physical Interface Configuration" Operational Tab is long/induces 'freeze'.

General Recommendations for Client > Server Latency

Certain browsers, such as Firefox, allow for more web connections per host by default.

• Check if this setting is configurable on the browser version that is used
• This matters more for multi-query pages, such as the Policy Group page

VPN and distance to APIC increase overall UI slowness given client browser requests and APIC response travel time. A jump box geographically local to the APICs significantly reduces the browser to APIC travel times.

Check for Long-Web Requests

If a Webserver (NGINX on APIC) is handles a high volume of Long-Web Requests, this can affect the performance of other requests received in parallel.

This is especially true for systems which have distributed databases, such as APICs. A single API request can require additional requests and lookups sent to other nodes in the fabric which can result in expectedly longer response times. A burst of these Long-Web Requests within a small time frame can compound the amount of resources required and lead to unexpectedly longer response times. Furthermore, received requests can then time out (90 seconds) which results in unexpected system behavior from a user perspective.

System Response Time -  Enable Calculation for Server Response Time

In 4.2(1)+, a user can enable "System Performance Calculation" which tracks and highlights API requests that took along time to handle.

Calculation can be enabled from System - System Settings - System Performance

Once "Calculation" is enabled, a user can navigate to specific APICs under Controllers to view the Slowest API Requests within the last 300 seconds.

System - Controllers - Controllers Folder - APIC x - Server Response Time

System Response Time -  Monitoring

If you have a scenario where the UI response slows down randomly but there is no active monitoring to know how often it is happening then you can use one of the 2 approaches

Option 1:  A subscription approach to get the response time from the fabric on a constant basis. This step requires an external server to run the script that will send subscription request and refreshes.

Option 2:  An on demand script run on the APIC itself and the outputs massaged on LV. This maybe a easier option due to less dependencies

Option 1. Subscription approach

Step 1:  (mentioned above as well) One time global change on the APIC. Non intrusive but good to always follow standard change practices.

Go to System-->System Settings-->System Performance

Enable calculations

Set response threshold to 50000ms

Don't change the other parameters

Once "Calculation" is enabled, a user can navigate to specific APICs under Controllers to view the Slowest API Requests within the last 300 seconds.

Step 2:

• Need an external host with a python3 environment that has reachability to the APIC. You need to install the python modules request and websocket-client.
• The script will subscribe to a specific object class that tracks the response times.

    Git location for apic_query_subscription_slowui.py --> https://wwwin-github.cisco.com/CX-ACI/lv_accloganalyzer

• The terminal session on the external server needs to be logged and someone will have to check the terminal to ensure that it is getting the data constantly.

python apic_query_subscription_slowui.py -a <anyapicip> -u <username>

upload the terminal session log once every 24 hours to the TAC case 

Step 3: Collect TAC outputs and access logs for event records and complete api queries hitting last 24 hrs every day at a particular time.

For ACI Fabrics running pre 5.3/6.0(3d), there is a Collect TacOutput Script available within the aci-tac-scripts  repository which serves a similar interface as the trigger tacoutput command:

apic# /tmp/collectTacOutputs.sh
Select corresponding numbers of objects to collect. Separate numbers with commas. *Note, topSystem, fabricNode, and firmwareARunning are automatically included.
Ex: 1,2,3,4,5
1. faultInfo *collected unfiltered
2. faultRecord
3. eventRecord
4. aaaModLR
5. polDeploymentRecord
6. epRecord
7. healthRecord
8. healthInst *collected unfiltered
Enter selections: 3
Enter record start date (format: 2019-12-15T00:00:00) *default is one month prior to current date: 2019-12-25T00:00:00
Enter record end date (format: 2019-12-15T00:00:00) *default is current date: 2020-01-05T00:00:00

...script collection runs...

Extract the tacoutputxx.tgz file and upload to the case

Note: For ACI Fabrics running version 5.3/6.0(3d) and later releases, trigger tacoutput provides a simplified collection interface for Events, Faults, Audit and other troubleshooting outputs). You don't need to use the script.

## From any one APIC, .

mkdir /data/techsupport/apicname_srnumber
cd /var/log/dme/log
cp  access* /data/techsupport/apicname_srnumber
cd /data/techsupport/
tar -zcvfapicname_srnumber.tgz /data/techsupport/apicname_srnumber

uploadapicname_srnumber.tgz to the case

Lab Example for script execution from an external host:

(my-virtual-env-3.7.13) root@aci-fab1-esxi2-oob:~# python apic_query_subscription_slowui.py -a 10.122.141.98 -u rgatti
Logging into APIC 10.122.141.98
rgatti password:
********** WebSocket Subscription URIS and IDs **********

WebSocket Subscription Status Code: 101

https://10.122.141.98/api/class/commRequestData.json?subscription=yes
- Subscription ID: 72059449487065089

********** WebSocket Subscription Messages **********
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.122.141.126","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-1","method":"GET","modTs":"2024-08-06T13:15:07.857-04:00","responseSize":"2313","responseTime":"1017","rn":"","startTime":"2024-08-06T13:12:38.334-04:00","status":"modified","url":"/api/node/class/firmwareOSource.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.26.164.186","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-2","method":"GET","modTs":"2024-08-06T13:15:07.857-04:00","responseSize":"252","responseTime":"1007","rn":"","startTime":"2024-08-06T13:13:39.145-04:00","status":"modified","url":"/api/node/mo/uni/infra/nodecfgcont/node-103.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.26.161.138","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-3","modTs":"2024-08-06T13:15:07.857-04:00","responseTime":"788","rn":"","startTime":"2024-08-06T13:14:24.351-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.24.131.101","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-4","modTs":"2024-08-06T13:15:07.857-04:00","responseTime":"784","rn":"","startTime":"2024-08-06T13:14:41.169-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.25.130.136","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-5","modTs":"2024-08-06T13:15:07.857-04:00","responseTime":"783","rn":"","startTime":"2024-08-06T13:12:29.028-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-1","modTs":"2024-08-06T13:15:51.533-04:00","responseSize":"3926","responseTime":"1013","rn":"","startTime":"2024-08-06T13:12:22.387-04:00","status":"modified","url":"/api/node/class/topSystem.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-2","modTs":"2024-08-06T13:15:51.533-04:00","responseSize":"905","responseTime":"208","rn":"","startTime":"2024-08-06T13:12:59.783-04:00","status":"modified","url":"/api/node/class/dnsProv.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-3","modTs":"2024-08-06T13:15:51.533-04:00","responseSize":"905","responseTime":"114","rn":"","startTime":"2024-08-06T13:10:59.781-04:00","status":"modified","url":"/api/node/class/dnsProv.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-4","modTs":"2024-08-06T13:15:51.533-04:00","responseSize":"905","responseTime":"98","rn":"","startTime":"2024-08-06T13:14:59.782-04:00","status":"modified","url":"/api/node/class/dnsProv.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-5","modTs":"2024-08-06T13:15:51.533-04:00","responseSize":"1075","responseTime":"46","rn":"","startTime":"2024-08-06T13:14:26.565-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-2/node-3/web/apiAvgResp/reqData-1","modTs":"2024-08-06T13:16:19.490-04:00","responseTime":"2392","rn":"","startTime":"2024-08-06T13:11:44.143-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-2/node-3/web/apiAvgResp/reqData-2","modTs":"2024-08-06T13:16:19.490-04:00","responseTime":"2389","rn":"","startTime":"2024-08-06T13:15:24.604-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-2/node-3/web/apiAvgResp/reqData-3","modTs":"2024-08-06T13:16:19.490-04:00","responseTime":"2384","rn":"","startTime":"2024-08-06T13:13:38.231-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-2/node-3/web/apiAvgResp/reqData-4","modTs":"2024-08-06T13:16:19.490-04:00","responseTime":"2384","rn":"","startTime":"2024-08-06T13:13:14.402-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-2/node-3/web/apiAvgResp/reqData-5","modTs":"2024-08-06T13:16:19.490-04:00","responseTime":"2383","rn":"","startTime":"2024-08-06T13:14:24.584-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.26.161.138","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-1","method":"POST","modTs":"2024-08-06T13:20:07.951-04:00","responseSize":"0","responseTime":"784","rn":"","startTime":"2024-08-06T13:19:24.361-04:00","status":"modified","url":"/uitelemetry/clicks"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.25.130.136","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-2","method":"POST","modTs":"2024-08-06T13:20:07.951-04:00","responseSize":"0","responseTime":"783","rn":"","startTime":"2024-08-06T13:18:34.821-04:00","status":"modified","url":"/uitelemetry/clicks"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.25.130.136","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-3","modTs":"2024-08-06T13:20:07.951-04:00","responseTime":"781","rn":"","startTime":"2024-08-06T13:17:03.705-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.25.130.136","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-4","modTs":"2024-08-06T13:20:07.951-04:00","responseTime":"781","rn":"","startTime":"2024-08-06T13:18:04.530-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"10.26.161.138","dn":"topology/pod-1/node-1/web/apiAvgResp/reqData-5","modTs":"2024-08-06T13:20:07.951-04:00","responseTime":"781","rn":"","startTime":"2024-08-06T13:18:24.354-04:00","status":"modified"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-1","modTs":"2024-08-06T13:20:51.568-04:00","responseSize":"905","responseTime":"1018","rn":"","startTime":"2024-08-06T13:16:59.784-04:00","status":"modified","url":"/api/node/class/dnsProv.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-2","modTs":"2024-08-06T13:20:51.568-04:00","responseSize":"405","responseTime":"148","rn":"","startTime":"2024-08-06T13:18:01.471-04:00","status":"modified","url":"/api/node/class/proxyServer.json"}}}]}
{"subscriptionId":["72059449487065089"],"imdata":[{"commRequestData":{"attributes":{"childAction":"","clientHostname":"192.168.1.1","dn":"topology/pod-1/node-2/web/apiAvgResp/reqData-3","modTs":"2024-08-06T13:20:51.568-04:00","responseSize":"3390","responseTime":"112","rn":"","startTime":"2024-08-06T13:17:29.488-04:00","status":"modified","url":"/api/node/class/pkiFabricSelfCAEp.json"}}}]}

Option 2. On Demand Script

On the APIC CLI

bash
<paste the below directly on the APIC cli - modify the filename. The script will run for 1 hr>

let i=1
while [ $i -lt 60 ]
do
date
date >> /data/techsupport/commRequestData.txt
moquery -c commRequestData >> /data/techsupport/commRequestData_<date_time>.txt
sleep 60
let i=i+1
done

The output is not easy to read which can be massaged using the script below on LV.

Git location for commrequests_processing.py --> https://wwwin-github.cisco.com/CX-ACI/lv_accloganalyzer
Copy the script to your local scripts folder (example /users/rgatti/scripts).

How to execute:

users/rgatti/dnld_695998256/cv1/jul14_2025/commrequestdata --> This is where we saved the commRequestData_<date_time>.txt file
python3 /users/rgatti/scripts/commrequests_processing.py -f commRequestData_jul14_4pm.txt > commRequestData_jul14_4pm.txt.formatted

Example below: responseTime is used for sorting with the highest response time displayed on the top

rgatti@aci-logviewer2:~/dnld_695998256/cv1/jul14_2025/commrequestdata$ less commRequestData_jul14_4pm.txt.formatted 
Processing file: commRequestData_jul14_4pm.txt
 
clientHostname | dn                                              | method | responseCode | responseSize | responseTime | startTime                     | url                                                                                
---------------+-------------------------------------------------+--------+--------------+--------------+--------------+-------------------------------+------------------------------------------------------------------------------------
127.0.0.1      | topology/pod-2/node-5/web/apiAvgResp/reqData-1  | GET    | 200          | 14255310     | 31123        | 2025-07-14T15:08:38.598-05:00 | /api/class/fvRInfoHolder.json                                                      
127.0.0.1      | topology/pod-2/node-5/web/apiAvgResp/reqData-1  | GET    | 200          | 14255310     | 31123        | 2025-07-14T15:08:38.598-05:00 | /api/class/fvRInfoHolder.json                                                      
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-1  | GET    | 200          | 17050        | 29660        | 2025-07-14T15:08:39.262-05:00 | /api/node/class/fabricPathEp.json                                                  
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-2  | GET    | 200          | 111529       | 29658        | 2025-07-14T15:08:39.261-05:00 | /api/class/mgmtRsInBStNode.json                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-3  | GET    | 200          | 474          | 29631        | 2025-07-14T15:08:39.258-05:00 | /api/class/mgmtInstP.json                                                          
127.0.0.1      | topology/pod-2/node-4/web/apiAvgResp/reqData-1  | GET    | 200          | 13637        | 28933        | 2025-07-14T15:08:38.597-05:00 | /api/class/faultDelegate.json                                                      
127.0.0.1      | topology/pod-2/node-2/web/apiAvgResp/reqData-2  | GET    | 200          | 530          | 28243        | 2025-07-14T15:08:39.274-05:00 | /api/class/vzRsDenyRule.json                                                       
127.0.0.1      | topology/pod-2/node-2/web/apiAvgResp/reqData-1  | GET    | 200          | 30           | 28243        | 2025-07-14T15:08:39.273-05:00 | /api/class/fvRsProtBy.json                                                         
127.0.0.1      | topology/pod-2/node-2/web/apiAvgResp/reqData-2  | GET    | 200          | 530          | 28243        | 2025-07-14T15:08:39.274-05:00 | /api/class/vzRsDenyRule.json                                                       
127.0.0.1      | topology/pod-2/node-2/web/apiAvgResp/reqData-1  | GET    | 200          | 30           | 28243        | 2025-07-14T15:08:39.273-05:00 | /api/class/fvRsProtBy.json                                                         
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-1  | GET    | 200          | 18426168     | 27107        | 2025-07-14T15:40:33.874-05:00 | /api/mo/uni.xml                                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-1  | GET    | 200          | 18426168     | 27107        | 2025-07-14T15:40:33.874-05:00 | /api/mo/uni.xml                                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-4  | GET    | 200          | 18429959     | 25516        | 2025-07-14T15:08:46.841-05:00 | /api/mo/uni.xml                                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-2  | GET    | 200          | 719780       | 24794        | 2025-07-14T15:40:35.346-05:00 | /api/class/fabricPathEp.json                                                       
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-2  | GET    | 200          | 719780       | 24794        | 2025-07-14T15:40:35.346-05:00 | /api/class/fabricPathEp.json                                                       
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-5  | GET    | 200          | 18431710     | 24604        | 2025-07-14T15:08:56.167-05:00 | /api/mo/uni.xml                                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-3  | GET    | 200          | 452          | 23720        | 2025-07-14T15:40:36.395-05:00 | /api/class/mgmtOoB.json                                                            
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-3  | GET    | 200          | 452          | 23720        | 2025-07-14T15:40:36.395-05:00 | /api/class/mgmtOoB.json                                                            
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-6  | GET    | 200          | 18430138     | 23578        | 2025-07-14T15:08:49.945-05:00 | /api/mo/uni.xml                                                                    
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-1  | GET    | 200          | 14255310     | 21159        | 2025-07-14T15:38:59.233-05:00 | /api/class/fvRInfoHolder.json                                                      
127.0.0.1      | topology/pod-2/node-1/web/apiAvgResp/reqData-1  | GET    | 200          | 14255310     | 21159        | 2025-07-14T15:38:59.233-05:00 | /api/class/fvRInfoHolder.json                                                

APIC API Usage Considerations

General Pointers to Ensure That a Script Does Not Harm Nginx

• Each APIC runs its own NGINX DME.
  • Only APIC 1's NGINX processes requests to APIC 1. APIC 2 and 3's NGINX does not process those requests.
• In general, 40+ API requests per second over a long period of time debilitates NGINX.
  
  • If found, reduce the aggressiveness of requests.
  • If the Requests host cannot be modified, consider NGINX Rate Limits on the APIC.

Address Script Inefficiencies

• Do not log in/log out before each API Request.
  • The default timeout for one login session is 10 minutes. This same session can be used for multiple requests and can be refreshed to extend the validity time.
  • See Cisco APIC REST API Configuration Guide - Accessing the REST API - Authenticating and Maintaining an API Session.
• If your script queries many DNs that share a parent, instead of collapse the queries into a single logical parent query with Query Filters.
  
  • See Cisco APIC REST API Configuration Guide - Composing REST API Queries - Applying Query Scoping Filters.
• If you need updates of an object or class of object, consider websocket subscriptions instead of rapid API requests.

NGINX Request Throttle

Available in 4.2(1)+, a user can enable request throttle against HTTP and HTTPS independently.

Note: Starting from ACI version 6.1(2), the supported maximum rate for this feature was reduced to 40 requests per second (r/s) or 2400 requests per minute (r/m) from 10,000 r/m.

Fabric - Fabric Policies - Policies Folder - Management Access Folder - default

When enabled:

• NGINX is restarted to apply config file changes
  • A new zone, httpsClientTagZone, is written to nginx config
• The Throttle rate can be set in Requests per Minute (r/m) or Requests per Second (r/s).
• Request Throttle relies on the Rate Limit Implementation included in NGINX
  • API Requests against the /api/ URI use the user-defined Throttle Rate + burst= (Throttle Rate x 2) + nodelay
    • There is a non-configurable throttle (zone aaaApiHttps) for /api/aaaLogin and /api/aaaRefresh which rate-limits at 2r/s + burst=4 + nodelay
  • Request Throttle is tracked on a per-client-ip-address basis
  • API requests sourced from the APIC self-ip (UI + CLI) bypass the throttle
  • Any Client IP Address which crosses the user-defined throttle rate + burst threshold receives a 503 response from the APIC
  • These 503s can be correlated within the access logs
  • error.log will have entries indicating when throttling has been activated (zone httpsClientTagZone) and against which Client hosts

apic# less /var/log/dme/log/error.log
...
2023/04/17 20:19:14 [error] ... limiting requests, excess: 40.292 by zone "httpsClientTagZone", client: h.o.s.t, ... request: "GET /api/class/...", host: "a.p.i.c"
2023/04/17 20:19:14 [error] ... limiting requests, excess: 40.292 by zone "httpsClientTagZone", client: h.o.s.t, ... request: "GET /api/node/...", host: "a.p.i.c"

As a general rule, Request Throttle only serves to protect the server (APIC) from DDOS-like symptoms induced by query-aggresive Clients. Understand and isolate the request-aggressive Client for final solutions in the app/script logic.