Contents

• Verified Scalability Limits
• Introduction
• Verified Scalability Limits
• Deployment Case Studies
• Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Max-Host Routing Mode)
• Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Default Routing Mode)
• Verified Scalability Limits for a Multicast System Topology

First Published:

Last Updated:

Text Part Number:

Verified Scalability Limits

---

This chapter describes the Cisco NX-OS configuration limits for the Cisco Nexus 9000 Series switches.

Introduction

The values provided in this guide should not be interpreted as theoretical system limits for Cisco Nexus 9000 Series hardware or Cisco NX-OS software. These limits refer to values that have been validated by Cisco. They can increase over time as more testing and validation is done.

Verified Scalability Limits

The tables in this section list the verified scalability limits for Cisco NX-OS Release 7.0(3)I1(2). These limits are validated with a unidimensional configuration. The values provided in these tables focus on the scalability of one particular feature at a time.

Each number is the absolute maximum currently supported by this Cisco NX-OS release for the corresponding feature. If the hardware is capable of a higher scale, future software releases might increase this verified maximum limit. Results might differ from the values listed here when trying to achieve maximum scalability with multiple features enabled.

Table 1 Cisco Nexus 2000 Series Fabric Extenders (FEX) Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit1	9300 Series Verified Limit2
Fabric Extenders and Fabric Extender server interfaces	32 and 1536	16 and 768
VLANs per Fabric Extender	2000 (across all Fabric Extenders)	2000 (across all Fabric Extenders)
VLANs per Fabric Extender server interface3	75	75
Port channels	500	500
Unique Fabric Extenders per Cisco Nexus 9500 Series supported line card	12	Not applicable

¹ Beginning with Cisco NX-OS Release 7.0(3)I1(2), the Cisco Nexus 2000 Series Fabric Extender is supported with Cisco Nexus 9500 Series switches and X9464PX and X9564PX line cards.

² The Cisco Nexus 2000 Series Fabric Extender is supported with only the Cisco Nexus 9396PX and Cisco Nexus 9372PX chassis.

³ For FEX HIF port channels, Cisco recommends that you enable STP port type edge using the spanning tree port type edge [trunk] command.

Table 2 Intelligent Traffic Director Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit4	9300 Series Verified Limit5
Nodes per ITD device group	4	4
ITD services	2	2
Ingress interfaces per ITD service	1	1
Virtual IP addresses per ITD service	8	8
Probes per ITD service	4	4

⁴ ITD is supported with the Cisco Nexus 9500 Series switches and the Cisco Nexus X9464PX, X9464TX, X9564PX, and X9564TX line cards.

⁵ ITD is supported with the Cisco Nexus 9372PX, 9372TX, 9396PX, 9396TX, 93120TX, and 93128TX switches.

Table 3 Interfaces Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
Generic routing encapsulation (GRE) tunnels	8	8
Port channel links	32	8
SVIs	490	250
vPCs	275	100 (280 with Fabric Extenders)

Table 4 Layer 2 Switching Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
MST instances	64	64
MST virtual ports	85,000	48,000
RPVST virtual ports	22,000	12,000
VLANs	4000	3900
VLANs in RPVST mode	500	500
Private VLANs (PVLANs)
Primary VLANs	16	16
Secondary VLANs	20	20
Ports in Community host mode	40	40
Ports in isolated host mode	20	40
Ports in isolated trunk host mode	22	40
Ports in promiscuous mode	48	5
Ports in promiscuous trunk mode	80	5
PVLANs allowed on a PVLAN port	16	16

Note	The number of supported VLANs per vPC should be within the MST or RPVST virtual port count specified in this table, depending on the topology.

Note	The number of supported STP VLAN port instances, for Fabric Extender host interface ports, should be less than 13,000.

Table 5 Multicast Routing Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
IPv4 multicast routes	32,000	8000
Outgoing interfaces (OIFs)	40	40

Note	The IPv4 multicast routes and the IPv4/IPv6 host routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.

Note	High availability (graceful restart and stateful switchover) is not supported when unicast or multicast aggressive timers are configured at any scale.

Table 6 Security Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
DHCP snooping bindings	2000	2000
IPv4 ingress ACLs	3072 (per network forwarding engine)	3072 (per network forwarding engine)
IPv4 egress ACLs	768 (per network forwarding engine)	768 (per network forwarding engine)
IPv6 ingress ACLs	1536 (per network forwarding engine)	1536 (per network forwarding engine)
IPv6 egress ACLs	256 (per network forwarding engine)	256 (per network forwarding engine)

Note	The ACL scalability limits also apply to policy-based ACLs (PBACLs).

Table 7 System Management Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
PTP
10G physical ports enabled for PTP	44	44
SPAN and ERSPAN
Configurable SPAN or ERSPAN sessions	32	4
Active SPAN or ERSPAN sessions6	4 to 32, based on the number of line cards and the session configuration	4
Active localized SPAN or ERSPAN sessions per line card7	4	4
Source interfaces per SPAN or ERSPAN session (Rx and Tx, Rx, or Tx)	48	48
Destination interfaces per SPAN session	1 (physical interface)	1 (physical interface)
Source VLANs per SPAN or ERSPAN session	32	32
TAP aggregation
Redirect interfaces in the redirect port list	12	12
Redirect port lists (or fan outs) per system	100	100

⁶ A single forwarding engine instance supports four SPAN or ERSPAN sessions. For Cisco Nexus 9300 Series switches, if the first three sessions have bidirectional sources, the fourth session has hardware resources only for Rx sources. This limitation might also apply to Cisco Nexus 9500 Series switches, depending on the SPAN or ERSPAN source's forwarding engine instance mappings.

⁷ The number of SPAN or ERSPAN sessions per line card reduces to two if the same interface is configured as the bidirectional source in more than one session.

Note	Beginning with Cisco NX-OS Release 7.0(3)I1(2), PTP is supported for all Cisco Nexus 9000 Series hardware except for the 100G 9408PC line card and the 100G M4PC generic expansion module (GEM).

Table 8 Unicast Routing Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
Unicast Routing
BFD sessions (echo mode)	512 (IPv4 only) 512 (IPv6 only) 256 (IPv4) + 256 (IPv6)	256 (IPv4 only) 256 (IPv6 only) 128 (IPv4) + 128 (IPv6)
BGP neighbors	2000 (IPv4 only) 2000 (IPv6 only) 1000 (IPv4) + 1000 (IPv6)	512 (IPv4 only) 512 (IPv6 only) 256 (IPv4) + 256 (IPv6)
EIGRP routes	20,000	20,000
EIGRP neighbors	360 (IPv4 only) 360 (IPv6 only) 180 (IPv4) + 180 (IPv6)	128 (IPv4 only) 128 (IPv6 only) 64 (IPv4) + 64 (IPv6)
HSRP groups	490	250
IPv4 ARP	45,000 (default system routing mode) 60,000 (max-host routing mode)	45,000
IPv4 host routes	208,000 (default system routing mode) 60,000 (max-host routing mode)	208,000 (default system routing mode) 16,000 (ALPM routing mode)
IPv6 host routes	40,000 (default system routing mode) 30,000 (max-host routing mode)	40,000 (default system routing mode) 8,000 (ALPM routing mode)
IPv6 ND	40,000	40,000
IPv4 unicast routes (LPM)	128,000 (default system routing mode) 16,000 (max-host routing mode) 128,000 with no IPv6 routes (64-bit ALPM routing mode)	12,000 (default system routing mode) 128,000 (ALPM routing mode)
IPv6 unicast routes (LPM)	20,000 (default system routing mode) 4000 (max-host routing mode) 80,000 with no IPv4 routes (64-bit ALPM routing mode)	7000 (6000 routes < /64, 1000 routes > /64) (default system routing mode) 20,000 (ALPM routing mode)
IPv4 and IPv6 unicast routes (LPM) in 64-bit ALPM routing mode	x IPv6 routes and y IPv4 routes, where 2x + y <= 128,000	Not applicable
IS-ISv4 adjacencies (either L1, L2, or sum of L1 and L2 with default timers)	255	255
IS-ISv4 BFD sessions (with default timers)	255	255
IS-ISv4 routes	10,000	10,000
IS-ISv4 network type	Point to point, broadcast	Point to point, broadcast
MAC addresses	90,000	90,000
OSPFv2 neighbors	1000	256
OSPFv3 neighbors	300	256
VRFs	1000	1000
VRRP groups per interface or I/O module	250	250
Policy-based routing (PBR)
Configured sequences per policy	256	256
Next-hop addresses per policy	32	32
IPv4 ACEs (unidimensional)	3072 (per network forwarding engine)	3072 (per network forwarding engine)
IPv6 ACEs (unidimensional)	1536 (per network forwarding engine)	1536 (per network forwarding engine)
IPv4 and IPv6s ACEs	2048 IPv4 + 256 IPv6	2048 IPv4 + 256 IPv6
Interfaces with PBR policy	512	512
VRRPv3
VRRPv3 groups per interface	255	255
VRRPv3 groups with default timers (1 s)	490	490
VRRPv3 groups with aggressive timers (100 ms)	200	200
VRRPv3 groups with relaxed timers (3 s)	490	490
Pathways with one VRRPv3 group with default timer (1 s)	489	489
VRRPv3 groups and pathways combined	490	490

Note	The IPv4/IPv6 host routes and the IPv4 multicast routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.

Note	The IPv4 and IPv6 unicast routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.

Note	High availability (graceful restart and stateful switchover) is not supported when unicast or multicast aggressive timers are configured at any scale.

Guidelines and Limitations for OSPF Verified Scalability Limits

• To achieve the highest scale, we recommend that you use a single OSPF instance instead of multiple instances.

• Each OSPFv2 and OSPFv3 scale value might vary when combined with other parameters.

• The graceful restart timeout value might need to be increased in multi-dimensional scenarios.

Table 9 VXLAN Verified Scalability Limits (Unidimensional)

Feature	9500 Series Verified Limit	9300 Series Verified Limit
VXLAN Flood and Learn
Virtual network identifiers (VNIs) or VXLAN-mapped VLANs	Not applicable	1000
Underlay multicast groups	Not applicable	128
Overlay MAC addresses	Not applicable	64,000
Remote VXLAN tunnel endpoints (VTEPs)*	Not applicable	256
Ingress replication peers	Not applicable	256
Ingress replication Layer 2 VNIs	Not applicable	1000
MAC addresses for ingress replication	Not applicable	64,000
Port VLAN translation under an interface	Not applicable	100
Port VLAN translation in a switch	Not applicable	2000
Static MAC address pointing to a remote VTEP	Not applicable	1000
VXLAN VLAN logical port VP count	Not applicable	6000
VXLAN BGP eVPN
Layer 2 VNI	1000	1000
Layer 3 VNI / VRF8	750	900
Underlay multicast groups	128	128
VTEPs*	256	256
MAC addresses	64,000	64,000
IPv4 host routes	32,000	32,000
IPv6 host routes	32,000	32,000
Overlay IPv4 LPM routes	12,000	12,000
Overlay IPv6 LPM routes	7000 (6000 routes < /64, 1000 routes > /64)	7000 (6000 routes < /64, 1000 routes > /64)
VXLAN VLAN logical port VP count	6000	6000
VXLAN BGP eVPN Ingress Replication
Layer 2 VNI	Not applicable	1000
Layer 3 VNI / VRF9	Not applicable	900
VTEPs*	Not applicable	256
MAC addresses	Not applicable	64,000
IPv4 host routes	Not applicable	32,000
IPv6 host routes	Not applicable	32,000
Overlay IPv4 LPM routes	Not applicable	12,000
Overlay IPv6 LPM routes	Not applicable	7000 (6000 routes < /64, 1000 routes > /64)
VXLAN VLAN logical port VP count	Not applicable	6000

⁸ ECMP objects are not shared across multiple VRFs.

⁹ ECMP objects are not shared across multiple VRFs.

*If multicast replication is used as the VXLAN underlay to transport overlay broadcast, unknown unicast, and multicast (BUM) traffic, each VNI can span across up to 40 VTEPs with the Cisco Nexus 9000 Series switch as a spine in a VXLAN-based fabric.

Deployment Case Studies

This section provides sample topologies for some common deployments. For each topology, the scalability numbers are the limits with all of the listed features enabled at the same time.

Attention:

These numbers are not the maximum verified values if each feature is viewed in isolation. For these numbers, see Verified Scalability Limits.

Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Max-Host Routing Mode)

This Layer 2/Layer 3 aggregation topology consists of Cisco Nexus 9508 switches as virtual port channel (vPC) aggregation pairs. These aggregation nodes are fully loaded with N9K-X9564TX, N9K-X9564PX, and N9K-X9636PQ line cards. The N9K-X9636PQ line cards are used in normal mode and breakout mode. Cisco Nexus 9396PX and 93128TX switches are used as top-of-rack units with Cisco Nexus 3000 Series switches to achieve the desired vPC scale.

The Cisco Nexus 9508 switch is also used as a core Layer 3 node that connects to a pair of vPC aggregation nodes. The focus of the topology is to test IPv4 ARP, IPv6 neighbor discovery (ND), and Layer 2 scalability and other routing, switching, and Layer 4 through Layer 7 features for management and operations. All Layer 3 interfaces are configured for dual stack, and the traffic is dual stack for all VLANs.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time. The scale numbers listed here exceed those used by most customers in their topologies. These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 10 Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Max-Host Routing Mode)

Feature	9500 Series Verified Limit (Max-Host Routing Mode)	9300 Series Verified Limit
Fully loaded chassis	6 N9636PQ line cards + 1 N9564TX line card + 1 N9564PX line card + 6 fabric modules + 2 system controllers + 2 supervisors	Not applicable
Physical interfaces enabled	300	Not applicable
Multicast S,G routes	500	Not applicable
Multicast *,G routes	500	Not applicable
IPv4 unicast routes (LPM)	2400	2200
IPv6 unicast routes (LPM)	2200	2200
IPv4 ARP	64,000	5000
IPv6 ND	40,000	5000
MAC addresses	90,000	72,000
VLANs	500 (RPVST)	3900 (MST)
vPCs*	275	100
OSPFv2 neighbors	16	200
OSPFv3 neighbors	16	200
BGP (IPv4) neighbors	64 (eBGP)	150 (iBGP)
BGP (IPv6) neighbors	64 (eBGP)	150 (iBGP)
SVIs	490	250
MST instances	Not applicable	64
HSRP VLANs (IPv4/IPv6)	490	250
Virtual ports	3000 (RPVST)	24,000 (MST)
Port channel links	32	8

* The number of VLANs per vPC supported should be within the MST or RPVST virtual port count specified in this table, depending on the topology.

Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Default Routing Mode)

This Layer 2/Layer 3 aggregation topology consists of Cisco Nexus 9516 switches as virtual port channel (vPC) aggregation pairs. These aggregation nodes are fully loaded with N9K-X9432PQ, N9K-X9464PX, N9K-X9464TX, N9K-X9536PQ, and N9K-X9564TX line cards. The chassis is fully loaded with five line cards configured for breakout mode. The Cisco Nexus 9396PX and 93128TX switches are used as top-of-rack units with Cisco Nexus 3000 Series switches to achieve the desired vPC scale. The Cisco Nexus 9516 nodes are running in default routing mode. The Cisco Nexus 3164Q switch is also used as a core Layer 3 node that connects to a pair of vPC aggregation nodes.

The focus of the topology is to test IPv4 ARP, IPv6 neighbor discovery (ND), Layer 2 scalability, IPv4 and IPv6 LPM routing, Layer 2 and Layer 3 multicast routing for IPv4, and Layer 4 through Layer 7 features for management and operations. All Layer 3 interfaces are configured for dual stack, and the traffic is dual stack for all VLANs.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time (or multidimensional scale). These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 11 Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Default Routing Mode)

Feature	9516 Series Verified Limit (Default Routing Mode)
Chassis configuration	5 N9K-X9432PQ line cards 4 N9K-X9464PX line cards 3 N9K-X9464TX line cards 3 N9K-X9536PQ line cards 1 N9K-X9564TX line card
Physical ports	735
vPCs	288
SVIs	401
VRFs	100
IPv4 ARP	20,000
IPv6 ND	10,000
Virtual ports	1500
BGP neighbors (IPv4 + IPv6)	64 + 64
IPv4 LPM routes	11,000
IPv6 LPM routes	1000
BFD (IPv4 + IPv6)	150
IGP OSPFv2 neighbors	100
IGP OSPFv3 neighbors	100
HSRP (IPv4 + IPv6)	401 + 401
IGMP groups	1680
Multicast *,G routes	1680
Tracking objects	100
VLANs	500
PIM neighbors	200
MAC addresses	30,500

Verified Scalability Limits for a Multicast System Topology

This multicast system topology consists of two multicast PIM domains. The Multicast Source Discovery Protocol (MSDP) is used to exchange multicast source information between these two domains.

Two Cisco Nexus 9508 switches are configured as vPC peers in one domain, and two Cisco Nexus 9372PX switches are configured as vPC peers in the other domain. The chassis are fully loaded with N9K-X9432PQ, N9K-X9464PX, N9K-X9536PQ, N9K-X9564PX, N9K-X9564TX, and N9K-X9636PQ line cards. eBGP routing is used to connect these two PIM domains. OSPF is used as IGP in one domain, and EIGRP is configured in the other domain. This setup is configured with multiple rendezvous points (RPs) to serve different multicast group ranges. BSR is used to advertise RP information in both of these PIM domains. PIM anycast is used in one domain, and MSDP anycast is used in the other domain for redundancy and load balancing. Static RP configuration is also used for a range of multicast groups.

The Cisco Nexus 9516 and Cisco Nexus 7000 Series switches are used as Layer 3 core routers in one domain. The Cisco Nexus 3164Q switches are used as Layer 3 core routers in the other domain. This topology also includes the Cisco Nexus 9396PX, Cisco Nexus 9372PX, and Cisco Nexus 3016/3064T switches in the access layer.

In addition to including Layer 2/Layer 3 IPv4 multicast routing, this topology also covers IPv4 and IPv6 host and LPM routing and Layer 2 unicast forwarding. All interfaces are configured for dual stack.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time (or multidimensional scale). These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 12 Verified Scalability Limits for a Multicast System Topology

Feature	9500 Series Verified Limit
9500 chassis configuration	N9K-X9432PQ, N9K-X9464PX, N9K-X9536PQ, N9K-X9564PX, N9K-X9564TX, and N9K-X9636PQ line cards
Multicast S,G routes	17,500
Multicast *,G routes	2500
Sources	2000 (2000, 200, 40, 10, 3, 2, 1)
Replications	40
ECMPs	16 (16,4,3,2 )
SVIs	200
HSRP/VRRP	200
MAC addresses	40,000
ARP	20,000
Unicast LPM IPv4 routes	20,000
Unicast LPM IPv6 routes	10,000
IPv4 ARP	20,000
IPv6 ND	4000
MSDP peers (fully mesh)	4
Anycast RPs (MSDP and PIM anycast)	4

Copyright © 2015, Cisco Systems, Inc. All rights reserved.