Overview
This guide contains the maximum verified scalability limits for ACI parameters for the Cisco APIC Release 2.3.1e and Cisco Nexus 9000 Series ACI-Mode Switches, Release 12.3.1e. These values are based on a profile where each feature was scaled to the numbers specified in the tables. These numbers do not represent the theoretically possible ACI fabric scale.
General Scalability Limits
-
L2 Fabric: In Legacy mode there is no routing, L3 context, nor contract enabled in the L2 fabric profile. A tenant in this profile does not need to be mapped to one dedicated ACI tenant. A tenant can be represented by a set of EPGs instead. To improve the load sharing among APIC controller nodes, you must distribute EPGs and BDs across an ACI tenant.
-
L3 Fabric: The ACI L3 fabric solution provides a feature-rich highly scalable solution for public cloud and large enterprise. With this design, almost all supported features are deployed at the same time and are tested as a solution. The scalability numbers listed in this section are multi-dimensional scalability numbers. The fabric scalability numbers represent the overall number of objects created on the fabric. The per-leaf scale numbers are the objects created and presented on an individual leaf switch. The fabric level scalability numbers represent APIC cluster scalability and the tested upper limits. Some of the per-leaf scalability numbers are subject to hardware restrictions. The per-leaf scalability numbers are the maximum limits tested and supported by leaf switch hardware. This does not necessarily mean that every leaf switch in the fabric was tested with maximum scale numbers.
-
Stretched Fabric: Stretched fabric allows multiple fabrics (up to 3) distributed in multiple locations to be connected as a single fabric with a single management domain. The scale for the entire stretched fabric remains the same as for a single site fabric. For example a L3 stretched fabric will support up to 200 leafs total which is the maximum number of leafs supported on a single site fabric. Parameters only relevant to stretched fabric are mentioned in the tables below.
-
Multi-Pod: Multipod enables provisioning a more fault-tolerant fabric comprised of multiple pods with isolated control plane protocols. Also, multipod provides more flexibility with regard to the full mesh cabling between leaf and spine switches. For example, if leaf switches are spread across different floors or different buildings, multipod enables provisioning multiple pods per floor or building and providing connectivity between pods through spine switches.
Multipod uses a single APIC cluster for all the pods; all the pods act as a single fabric. Individual APIC controllers are placed across the pods but they are all part of a single APIC cluster.
NOTE: The maximum number of leaf switches overall is 400 per fabric scale and the maximum number of physical ports is 19,200 per fabric.
Feature |
L2 Fabric |
L3 Fabric |
Large L3 Fabric |
|||
---|---|---|---|---|---|---|
Number of APIC controllers |
3 |
Minimum 3 (4 also supported) |
5 |
|||
Number of leaf switches |
80 |
80 |
200 |
|||
Number of spines |
Maximum spines per pod: 6. Total spines 16. |
Maximum spines per pod: 6. Total spines 16. |
Maximum spines per pod: 6. Total spines 16. |
|||
Number of FEXs |
N/A |
20 FEXes per leaf, 320 FEX ports/leaf, 650 FEXes per fabric. |
N/A |
|||
Number of tenants |
N/A |
1000 |
3000 |
|||
Number of Layer 3 (L3) contexts |
N/A |
1000 |
3000 |
|||
Number of contracts/filters |
N/A |
|
|
|||
Number of endpoint groups (EPGs) |
21,000 (500 maximum per tenant) |
15,000 (500 maximum per tenant) |
15,000 (500 maximum per tenant) |
|||
Number of Isolation enabled EPGs |
250 |
250 |
250 |
|||
Number of endpoints (EPs) |
180,000 |
180,000 |
180,000 |
|||
Number of bridge domains (BDs) |
21,000 |
15,000 |
15,000 |
|||
Number of IP longest prefix matches (IP LPMs) (for external connection)
|
N/A |
|
|
|||
Number of BGP + number of OSPF sessions + EIGRP (for external connection) |
N/A |
1,200 |
1,200 |
|||
Number of Multicast groups |
N/A |
8000 |
8000 |
|||
Number of Multicast groups per VRF |
N/A |
8000 |
8000 |
|||
Number of static routes to a single SVI/VRF |
N/A |
5,000 |
10,000 |
|||
Number of static routes on a single leaf switch |
N/A |
5,000 |
10,000 |
|||
Number of vCenters |
N/A |
|
|
|||
Number of Service Chains |
N/A |
1000 |
1000 |
|||
Number of L4 - L7 devices |
N/A |
30 physical, 1,200 virtual (1200 maximum per fabric) |
30 physical, 1,200 virtual (1500 maximum per fabric) |
|||
Number of ESXi hosts - VDS |
N/A |
3200 |
3200 |
|||
Number of ESXi hosts - AVS |
N/A |
3200 (Only 1 AVS instance per host) |
3200 (Only 1 AVS instance per host) |
|||
Number of VMs |
N/A |
Depends upon server scale |
Depends upon server scale |
|||
Number of configuration zones per fabric |
30 |
30 |
30 |
|||
Number of BFD sessions per leaf switch |
256 Minimum BFD timer required to support this scale:
|
256 Minimum BFD timer required to support this scale:
|
256 Minimum BFD timer required to support this scale:
|
|||
Multi-Pod NOTE: * = preferred cluster size |
3* or 4 node APIC cluster,6 pods, 80 leaf switches overall |
3* or 4 node APIC cluster,6 pods, 80 leaf switches overall |
|
|||
L3 eVPN Services over Fabric WAN (with and without OpFlex) |
N/A |
1000 VRFs, 1000 L3outs, 60,000 routes in a fabric |
1000 VRFs, 60,000 routes in a fabric |
|||
Layer 3 Multicast routes |
N/A |
8,000 |
8,000 |
|||
Number of Routes in Overlay-1 VRF |
1,000 |
1,000 |
1,000 |
Fabric Topology, SPAN, Tenants, Contexts, External EPGs, Bridge Domains, Endpoints, and Contracts Scalability Limits
Configurable Options |
Per Leaf Scale |
Per Fabric Scale |
|||
---|---|---|---|---|---|
Fabric Topology |
|||||
Number of PCs, vPCs |
320 (with FEX HIF) |
N/A |
|||
Number of encaps per access port, PC, vPC (non FEX HIF) |
1750 |
N/A |
|||
Number of encaps per FEX HIF, PC, vPC |
20 |
N/A |
|||
Number of member links per PC, vPC* *vPC total ports = 16, 8 per leaf |
8 |
N/A |
|||
Number of ports x VLANS (global scope and no FEX HIF) |
64,000 168,000 (when using legacy BD mode) |
N/A |
|||
Number of ports x VLANS (FEX HIFs and/or local scope) |
For ALE v1 and v2: 9,000 For LSE and LSE2: 10,000 |
N/A |
|||
Number of static port bindings |
For ALE v1 and v2: 30,000 For LSE and LSE2: 60,000 |
400,000 |
|||
STP |
All VLANs |
N/A |
|||
Mis-Cabling Protocol (MCP) |
256 VLANs per interface 2000 logical ports (port x VLAN) per leaf |
N/A |
|||
Maximum number of endpoints (EPs) |
For ALE v1 and v2:
For LSE and LSE2:
|
180,000 |
|||
Number of MAC EPGs |
N/A |
125 |
|||
Number of Multicast Groups |
8000 |
8000 |
|||
Number of Multicast Groups per VRF |
8000 |
8000 |
|||
Number of IPs per MAC |
1024 |
1024 |
|||
SPAN |
ALE based ToRs:
LSE based ToRs:
|
N/A |
|||
Number of ports per SPAN session |
NOTE: For LSE/LSE2 only: 30 (total number of unique ports (fabric + access) across all types of span sessions) |
N/A |
|||
Number of source EPGs in tenant SPAN sessions
|
ALE-based ToR switches:
LSE-based ToR switches:
|
N/A |
|||
Common pervasive gateway |
256 virtual IPs per Bridge Domain |
N/A |
|||
Maximum number of Data Plane policers |
ALE:
LSE and LSE2:
|
N/A |
|||
Maximum number of SNMP trap receivers |
10 |
10 |
|||
Maximum number of Q-in-Q tunnels (both QinQ core and edge combined) |
1980 |
N/A |
|||
Maximum number of TEP-to-TEP atomic counters |
N/A |
1600 |
|||
Tenants |
|||||
Number of Contexts per tenant |
50 |
50 |
|||
Number of application profiles per tenant (or per Context) |
N/A |
N/A |
|||
Contexts (All numbers applicable to dual stack unless explicitly called out) |
|||||
Maximum number of Context |
400 |
N/A |
|||
Number of VRFs per tenant |
N/A |
50 |
|||
Number of BDs per VRF |
N/A |
1750 |
|||
Number of isolated EPGs |
N/A |
250 |
|||
Border Leafs per L3 Out |
N/A |
8 |
|||
Maximum number of LPM Prefixes for External EPG Classification |
1000 IPv4 |
N/A |
|||
Maximum number of vzAny Provided Contracts |
16 per Ctx |
N/A |
|||
Maximum number of vzAny Consumed Contracts |
16 per Ctx |
N/A |
|||
Number of service graphs per device cluster |
N/A |
500 |
|||
L3 Out per context |
-- |
400 |
|||
Maximum number of Routed, Routed Sub-interface, or SVIs per L3 Out |
|
|
|||
Maximum number of BGP neighbors |
400 |
2400 |
|||
Maximum number of BGP neighbors with authentication enabled |
150 |
N/A |
|||
Maximum number of OSPF neighbors |
300 (Maximum number of VRFs with an l3out where OSPF is the only routing protocol enabled, cannot exceed 142) |
N/A |
|||
Maximum number of EIGRP neighbors |
16 |
N/A |
|||
Maximum number of Static Routes |
|
|
|||
Maximum number of External Routes |
For ALE v1 and v2:
For LSE and LSE2:
|
|
|||
Maximum number of Secondary addresses per logical interface |
1 |
1 |
|||
Maximum number of L3 interfaces per Context (SVIs and sub-interfaces) |
|
|
|||
Maximum number of ARP entries for L3 Outs |
7500 |
N/A |
|||
Shared L3 Out |
|
|
|||
Configurable Options |
per Leaf scale |
per Fabric scale |
|||
Maximum number of L3 Outs |
400 (per leaf scale) |
2400 (single-stack) 1800 (dual-stack) |
|||
External EPGs |
|||||
Number of External EPGs |
600 |
2400 (single stack) |
|||
Number of External EPGs per L3 out |
250 |
400 |
|||
Bridge Domain |
|||||
Maximum number of BDs |
1750; if legacy mode, 3,500; if Multicast optimized mode then 50 |
15,000 |
|||
Maximum number of BDs with Unicast Routing per Context (VRF) |
256 |
N/A |
|||
Maximum number of subnets per BD |
512 (cannot be for all BDs) |
512 per BD |
|||
Maximum number of EPGs per BD |
3499 (cannot exceed 3,500 total) 3499 is supported in hardware but please refer to the per fabric scale for the effective software support for this release. |
3499 |
|||
Number of L2 Outs per BD |
1 |
1 |
|||
Number of BDs with Custom MAC Address |
1750 If Multicast optimized mode is used, then 50 |
1750 If Multicast optimized mode is used, then 50 |
|||
Number of Multicast groups |
8000 |
8000 |
|||
Maximum number of EPGs + L3 Outs per Multicast Group |
128 |
128 |
|||
Maximum number of BDs with L3 Multicast enabled |
1750 |
1750 |
|||
Maximum number of VRFs with L3 Multicast enabled |
64 |
64 |
|||
Maximum number of L3 Outs per BD |
4 |
N/A |
|||
DHCP relay addresses per BD across all labels |
16 |
N/A |
|||
Number of external EPGs per L2 out |
1 |
1 |
|||
Maximum number of PIM Neighbors |
1000 |
1000 |
|||
Maximum number of PIM Neighbors per VRF |
64 |
64 |
|||
Maximum number of L3Out physical interfaces with PIM enabled |
32 |
N/A |
|||
Endpoint Groups (Under App Profiles) |
|||||
Maximum amount of EPGs |
Normally 1750; if legacy mode 3500 |
15,000 |
|||
Maximum amount of encaps per EPG |
1 Static leaf binding, plus 10 Dynamic VMM |
N/A |
|||
Maximum Path encap binding per EPG |
Equals to number of ports on the leaf |
N/A |
|||
Maximum amount of encaps per EPG per port |
One (path or leaf binding) |
N/A |
|||
Maximum number of domains (physical, L2, L3) |
|
N/A |
|||
Maximum number of VMM domains |
|
N/A |
|||
Maximum amount of native encaps |
|
Applicable to each leaf independently |
|||
Maximum amount of 802.1p encaps |
|
Applicable to each leaf independently |
|||
Can encap be tagged and untagged? |
No |
N/A |
|||
Maximum number of Static endpoints per EPG |
Maximum endpoints |
N/A |
|||
Maximum number of Subnets for Inter-context access per tenant |
4000 |
N/A |
|||
Maximum number of Taboo Contracts per EPG |
2 |
N/A |
|||
IP-based EPG |
4000 |
N/A |
|||
Contracts |
|||||
Security TCAM size |
|
N/A |
|||
Approximate TCAM calculator given contracts and their use by EPGs |
Number of entries in a contract X Number of Consumer EPGs X Number of Provider EPGs X 2 |
N/A |
|||
Maximum number of EPGs providing the same contract |
25 |
25 |
|||
Maximum number of EPGs consuming the same contract |
25 |
25 |
|||
FEX VPC |
|||||
Maximum EPGs behind FEX VPC port |
20 |
N/A |
|||
FCoE |
|||||
Maximum number of VSAN |
32 |
N/A |
|||
Maximum number of VFC |
151
|
N/A |
|||
Maximum number of FDISC per port |
96 |
N/A |
|||
Maximum number of FDISC per SB |
96 |
N/A |
ALE Type |
ACI-Supported TORs |
---|---|
ALE v1 |
|
ALE v2 |
|
LSE |
N9K-C93108TC-EX + N9K-C93180YC-EX |
LSE2 |
N9K-C93108TC-FX + N9K-C93180YC-FX |
Multiple Fabric Options Scalability Limits
Configurable Options |
Per Leaf Scale |
Per Fabric Scale |
|
---|---|---|---|
Stretched Fabric |
|||
Maximum number of fabrics that can be a stretched fabric |
N/A |
3 |
|
Maximum number of Route Reflectors |
N/A |
6 |
|
Multi-Pod |
|||
Maximum number of PODs |
N/A |
10 |
|
Maximum number of nodes per POD |
N/A |
200 |
|
Maximum number of leaf switches overall |
N/A |
400 |
VMM Scalability Limits
Configurable Options |
Per Leaf Scale |
Per Fabric Scale |
|
---|---|---|---|
VMware |
|||
Number of vCenters (vDS) |
N/A |
200 (Verified with a load of 10 events/minute for each vCenter) |
|
Number of vCenters (AVS) |
N/A |
50 |
|
Datacenters in a vCenter |
N/A |
2 |
|
Total number of (VMM domain, VMM controller (vCenter/vShield)) instances |
N/A |
|
|
Number of ESX hosts per AVS |
240 |
N/A |
|
Number of EPGs per vCenter/vDS |
N/A |
5,000 |
|
Number of EPGs to VMware domans/vDS |
N/A |
5,000 |
|
Number of EPGs per vCenter/AVS |
N/A |
3,500 |
|
Number of EPGs to VMware domains/AVS |
N/A |
3,500 |
|
Number of endpoints (EPs) per AVS |
10,000 |
10,000 |
|
Number of endpoints per vDS |
10,000 |
10,000 |
|
Number of endpoints per vCenter |
10,000 |
10,000 |
|
Support RBAC for AVS |
N/A |
Yes |
|
Support RBAC for vDS |
N/A |
Yes |
|
Number of VM Attribute Tags per vCenter |
N/A |
vCenter version 6.0: 500 vCenter version 6.5: 1000 |
|
Microsegmentation/DFW with AVS |
|||
Number of ESX hosts per AVS |
100 |
N/A |
|
Number of Microsegment EPGs |
1,000 |
N/A |
|
Number of DFW flows per vEth |
10,000 |
N/A |
|
Number of DFW denied and permitted flows per ESX host |
250,000 |
N/A |
|
Number of VMM domains per Microsegment EPG |
N/A |
10 |
|
Microsoft |
|||
Number of controllers per SCVMM domain |
N/A |
5 |
|
Number of SCVMM domains |
N/A |
5 |
|
EPGs per Microsoft VMM domain |
N/A |
3,000 |
|
EPGs per all Microsoft VMM domains |
N/A |
9,000 |
|
EP/VNICs per HyperV host |
N/A |
100 |
|
EP/VNICs per SCVMM |
N/A |
3,000 |
|
Number of logical switch per host |
N/A |
1 |
|
Number of uplinks per logical switch |
N/A |
4 |
|
Number of Windows Azure Pack subscriptions |
N/A |
1,000 |
|
Number of plans per Windows Azure Pack instance |
N/A |
150 |
|
Number of users per plan |
N/A |
200 |
|
Number of subscriptions per user |
N/A |
3 |
|
VM networks per Windows Azure Pack user |
N/A |
100 |
|
VM networks per Windows Azure Pack instance |
N/A |
3,000 |
|
Number of tenant shared services/providers |
N/A |
40 |
|
Number of consumers of shared services |
N/A |
40 |
|
Number of VIPs (Citrix) |
N/A |
50 |
|
Number of VIPs (F5) |
N/A |
50 |
|
Microsoft microsegmentation |
1,000 |
N/A |
Layer 4 - Layer 7 Scalability Limits
Configurable Options (L4-L7 Configurations) |
Per Leaf Scale |
Per Fabric Scale |
|
---|---|---|---|
Maximum number of L4-L7 logical device clusters |
N/A |
1,500 |
|
Maximum number of graph instances |
N/A |
1,000 |
|
Maximum number of VIPs per graph instance |
N/A |
1 |
|
Number of device clusters per tenant |
N/A |
30 |
|
Number of interfaces per device cluster |
N/A |
Any |
|
Number of graph instances per device cluster |
N/A |
500 |
|
Deployment scenario for ASA (transparent or routed) |
N/A |
Yes |
|
Deployment scenario for Citrix - One arm with SNAT/etc. |
N/A |
Yes |
|
Deployment scenario for F5 - One arm with SNAT/etc. |
N/A |
Yes |
AD, TACACS, RBAC Scalability Limits
Configurable Options |
Per Leaf Scale |
Per Fabric Scale |
|
---|---|---|---|
Number of ACS/AD/LDAP authorization domains |
N/A |
4 tested (16 maximum /server type) |
|
Number of login domains |
N/A |
15 (can go beyond) |
|
Number of security domains/APIC |
N/A |
15 (can go beyond) |
|
Number of security domains in which the tenant resides |
N/A |
4 (can go beyond) |
|
Number of priority |
N/A |
4 tested (16 per domain) |
|
Number of shell profiles that can be returned |
N/A |
4 tested (32 domains total) |
|
Number of users |
N/A |
8,000 local / 8,000 remote |
|
Number of simultaneous logins |
N/A |
500 connections / NGNIX simultaneous REST logins |
QoS Scalability Limits
The table below shows QoS scale limits. The scale numbers depend on whether remote leafs are present in the topology as well as MPOD QoS Policy and CoS Preservation settings.
QoS Scale |
||
---|---|---|
MPOD QoS Policy enabled |
Custom QOS Policy with DSCP |
9 |
Custom QOS Policy with DSCP and Dot1P |
9 |
|
Custom QOS Policy with Dot1P |
48 |
|
Custom QOS Policy via a Contract |
48 |
|
CoS Preservation enabled |
Custom QOS Policy with DSCP |
9 |
Custom QOS Policy with DSCP and Dot1P |
9 |
|
Custom QOS Policy with Dot1P |
48 |
|
Custom QOS Policy via a Contract |
48 |