Verified Scalability Limits

This chapter describes the Cisco NX-OS configuration limits for the Cisco Nexus 9000 Series switches.

Introduction

The values provided in this guide should not be interpreted as theoretical system limits for Cisco Nexus 9000 Series hardware or Cisco NX-OS software. These limits refer to values that have been validated by Cisco. They can increase over time as more testing and validation is done.

Verified Scalability Limits

The tables in this section list the verified scalability limits for Cisco NX-OS Releases 7.0(3)I1(1), 7.0(3)I1(1a), and 7.0(3)I1(1b). These limits are validated with a unidimensional configuration. The values provided in these tables focus on the scalability of one particular feature at a time.

Each number is the absolute maximum currently supported by this Cisco NX-OS release for the corresponding feature. If the hardware is capable of a higher scale, future software releases might increase this verified maximum limit. Results might differ from the values listed here when trying to achieve maximum scalability with multiple features enabled.

Table 1. Cisco Nexus 2000 Series Fabric Extenders (FEX) Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

Fabric Extenders and Fabric Extender server interfaces

Not applicable

16 and 768

VLANs per Fabric Extender

Not applicable

2000 (across all Fabric Extenders)

VLANs per Fabric Extender server interface

Not applicable

75

Port channels

Not applicable

500


Note

The Cisco Nexus 2000 Series Fabric Extender is supported only with the Cisco Nexus 9396PX and Cisco Nexus 9372PX chassis.

Table 2. Interfaces Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

Generic routing encapsulation (GRE) tunnels

8

8

Port channel links

32

8

SVIs

490

250

vPCs

275

100 (280 with Fabric Extenders)

Table 3. Layer 2 Switching Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

MST instances

64

64

MST virtual ports

85,000

48,000

RPVST virtual ports

22,000

12,000

VLANs

4000

3900

VLANs in RPVST mode

500

500

Private VLANs (PVLANs)

Primary VLANs

16

16

Secondary VLANs

20

20

Ports in Community host mode

40

40

Ports in isolated host mode

20

40

Ports in isolated trunk host mode

22

40

Ports in promiscuous mode

48

5

Ports in promiscuous trunk mode

80

5

PVLANs allowed on a PVLAN port

16

16

Note

The number of supported VLANs per vPC should be within the MST or RPVST virtual port count specified in this table, depending on the topology.


Note

The number of supported STP VLAN port instances, for Fabric Extender host interface ports, should be less than 13,000.

Table 4. Multicast Routing Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

IPv4 multicast routes

32,000

8000

Outgoing interfaces (OIFs)

40

40


Note

The IPv4 multicast routes and the IPv4/IPv6 host routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.


Note
High availability (graceful restart and stateful switchover) is not supported when unicast or multicast aggressive timers are configured at any scale.
Table 5. Security Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

DHCP snooping bindings

2000

2000

IPv4 ingress ACLs

3072 (per network forwarding engine)

3072 (per network forwarding engine)

IPv4 egress ACLs

768 (per network forwarding engine)

768 (per network forwarding engine)

IPv6 ingress ACLs

1536 (per network forwarding engine)

1536 (per network forwarding engine)

IPv6 egress ACLs

256 (per network forwarding engine)

256 (per network forwarding engine)


Note

The ACL scalability limits also apply to policy-based ACLs (PBACLs).

Table 6. System Management Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

PTP

10G physical ports enabled for PTP

44 1

44 2

SPAN and ERSPAN

Configurable SPAN or ERSPAN sessions

32

4

Active SPAN or ERSPAN sessions3

4 to 32, based on the number of line cards and the session configuration

4

Active localized SPAN or ERSPAN sessions per line card4

4

4

Source interfaces per SPAN or ERSPAN session (Rx and Tx, Rx, or Tx)

48

48

Destination interfaces per SPAN session

1 (physical interface)

1 (physical interface)

Source VLANs per SPAN or ERSPAN session

32

32

TAP aggregation

Redirect interfaces in the redirect port list

12

12

Redirect port lists (or fan outs) per system

100

100

1 PTP is supported only on these Cisco Nexus 9500 Series switches: Cisco Nexus 9504 or 9508 switch with all of the PTP ports on a single X9636PQ line card.
2 PTP is supported only on the Cisco Nexus 9332PQ switch (except for the last six physical interfaces), the Cisco Nexus 9396PX switch, and the Cisco Nexus 93128TX switch [beginning with Cisco NX-OS Release 7.0(3)I1(1a)].
3 A single forwarding engine instance supports four SPAN or ERSPAN sessions. For Cisco Nexus 9300 Series switches, if the first three sessions have bidirectional sources, the fourth session has hardware resources only for Rx sources. This limitation might also apply to Cisco Nexus 9500 Series switches, depending on the SPAN or ERSPAN source's forwarding engine instance mappings.
4 The number of SPAN or ERSPAN sessions per line card reduces to two if the same interface is configured as the bidirectional source in more than one session.
Table 7. Unicast Routing Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

Unicast Routing

BFD sessions (echo mode)

512 (IPv4 only)

512 (IPv6 only)

256 (IPv4) + 256 (IPv6)

256 (IPv4 only)

256 (IPv6 only)

128 (IPv4) + 128 (IPv6)

BGP neighbors

2000 (IPv4 only)

2000 (IPv6 only)

1000 (IPv4) + 1000 (IPv6)

512 (IPv4 only)

512 (IPv6 only)

256 (IPv4) + 256 (IPv6)

EIGRP routes

20,000

20,000

EIGRP neighbors

360 (IPv4 only)

360 (IPv6 only)

180 (IPv4) + 180 (IPv6)

128 (IPv4 only)

128 (IPv6 only)

64 (IPv4) + 64 (IPv6)

HSRP groups

490

250

IPv4 ARP

45,000 (default system routing mode)

60,000 (max-host routing mode)

45,000

IPv4 host routes

208,000 (default system routing mode)

60,000 (max-host routing mode)

208,000 (default system routing mode)

16,000 (ALPM routing mode)

IPv6 host routes

40,000 (default system routing mode)

30,000 (max-host routing mode)

40,000 (default system routing mode)

8,000 (ALPM routing mode)

IPv6 ND

40,000

40,000

IPv4 unicast routes (LPM)

128,000 (default system routing mode)

16,000 (max-host routing mode)

128,000 with no IPv6 routes (64-bit ALPM routing mode)

12,000 (default system routing mode)

128,000 (ALPM routing mode)

IPv6 unicast routes (LPM)

20,000 (default system routing mode)

4000 (max-host routing mode)

80,000 with no IPv4 routes (64-bit ALPM routing mode)

7000 (6000 routes < /64, 1000 routes > /64) (default system routing mode)

20,000 (ALPM routing mode)

IPv4 and IPv6 unicast routes (LPM) in 64-bit ALPM routing mode

x IPv6 routes and y IPv4 routes, where 2x + y <= 128,000

Not applicable

MAC addresses

90,000

90,000

OSPFv2 neighbors

1000

256

OSPFv3 neighbors

300

256

VRFs

1000

1000

VRRP groups per interface or I/O module

250

250

Policy-based routing (PBR)

Configured sequences per policy

256

256

Next-hop addresses per policy

32

32

IPv4 ACEs (unidimensional)

3072 (per network forwarding engine)

3072 (per network forwarding engine)

IPv6 ACEs (unidimensional)

1536 (per network forwarding engine)

1536 (per network forwarding engine)

IPv4 and IPv6s ACEs

2048 IPv4 + 256 IPv6

2048 IPv4 + 256 IPv6

Interfaces with PBR policy

512

512

VRRPv3

VRRPv3 groups per interface

255

255

VRRPv3 groups with default timers (1 s)

490

490

VRRPv3 groups with aggressive timers (100 ms)

200

200

VRRPv3 groups with relaxed timers (3 s)

490

490

Pathways with one VRRPv3 group with default timer (1 s)

489

489

VRRPv3 groups and pathways combined

490

490


Note

The IPv4/IPv6 host routes and the IPv4 multicast routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.


Note

The IPv4 and IPv6 unicast routes share the same hardware table. Limits are provided for both the default line card mode and the max host line card mode.


Note
High availability (graceful restart and stateful switchover) is not supported when unicast or multicast aggressive timers are configured at any scale.

Guidelines and Limitations for OSPF Verified Scalability Limits

  • To achieve the highest scale, we recommend that you use a single OSPF instance instead of multiple instances.

  • Each OSPFv2 and OSPFv3 scale value might vary when combined with other parameters.

  • The graceful restart timeout value might need to be increased in multi-dimensional scenarios.

Table 8. VXLAN Verified Scalability Limits (Unidimensional)

Feature

9500 Series Verified Limit

9300 Series Verified Limit

VXLAN Flood and Learn

Virtual network identifiers (VNIs) or VXLAN-mapped VLANs

Not applicable

1000

Underlay multicast groups

Not applicable

128

Overlay MAC addresses

Not applicable

64,000

Remote VXLAN tunnel endpoints (VTEPs)*

Not applicable

256

Ingress replication peers

Not applicable

16

Ingress replication Layer 2 VNIs

Not applicable

1900

MAC addresses for ingress replication

Not applicable

60,000

VXLAN BGP eVPN

Layer 2 VNI

Not applicable

1000

Layer 3 VNI

Not applicable

900

Underlay multicast groups

Not applicable

128

VTEPs*

Not applicable

256

MAC addresses

Not applicable

64,000

IPv4 host routes

Not applicable

32,000

IPv6 host routes

Not applicable

32,000

Overlay IPv4 LPM routes

Not applicable

12,000

Overlay IPv6 LPM routes

Not applicable

7000 (6000 routes < /64, 1000 routes > /64)


Note

* If multicast replication is used as the VXLAN underlay to transport overlay broadcast, unknown unicast, and multicast (BUM) traffic, each VNI can span across up to 40 VTEPs with the Cisco Nexus 9000 Series switch as a spine in a VXLAN-based fabric.

Deployment Case Studies

This section provides sample topologies for some common deployments. For each topology, the scalability numbers are the limits with all of the listed features enabled at the same time.


Attention

These numbers are not the maximum verified values if each feature is viewed in isolation. For these numbers, see Verified Scalability Limits.

Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Max-Host Routing Mode)

This Layer 2/Layer 3 aggregation topology consists of Cisco Nexus 9508 switches as virtual port channel (vPC) aggregation pairs. These aggregation nodes are fully loaded with N9K-X9564TX, N9K-X9564PX, and N9K-X9636PQ line cards. The N9K-X9636PQ line cards are used in normal mode and breakout mode. Cisco Nexus 9396PX and 93128TX switches are used as top-of-rack units with Cisco Nexus 3000 Series switches to achieve the desired vPC scale.

The Cisco Nexus 9508 switch is also used as a core Layer 3 node that connects to a pair of vPC aggregation nodes. The focus of the topology is to test IPv4 ARP, IPv6 neighbor discovery (ND), and Layer 2 scalability and other routing, switching, and Layer 4 through Layer 7 features for management and operations. All Layer 3 interfaces are configured for dual stack, and the traffic is dual stack for all VLANs.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time. The scale numbers listed here exceed those used by most customers in their topologies. These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 9. Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Max-Host Routing Mode)

Feature

9500 Series Verified Limit (Max-Host Routing Mode)

9300 Series Verified Limit

Fully loaded chassis

6 N9636PQ line cards + 1 N9564TX line card + 1 N9564PX line card + 6 fabric modules + 2 system controllers + 2 supervisors

Not applicable

Physical interfaces enabled

300

Not applicable

Multicast S,G routes

500

Not applicable

Multicast *,G routes

500

Not applicable

IPv4 unicast routes (LPM)

2400

2200

IPv6 unicast routes (LPM)

2200

2200

IPv4 ARP

60,000

5000

IPv6 ND

30,000

5000

MAC addresses

90,000

72,000

VLANs

500 (RPVST)

3900 (MST)

vPCs*

275

100

OSPFv2 neighbors

16

200

OSPFv3 neighbors

16

200

BGP (IPv4) neighbors

64 (eBGP)

150 (iBGP)

BGP (IPv6) neighbors

64 (eBGP)

150 (iBGP)

SVIs

490

250

MST instances

Not applicable

64

HSRP VLANs (IPv4/IPv6)

490

250

Virtual ports

3000 (RPVST)

24,000 (MST)

Port channel links

32

8

* The number of VLANs per vPC supported should be within the MST or RPVST virtual port count specified in this table, depending on the topology.

Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Default Routing Mode)

This Layer 2/Layer 3 aggregation topology consists of Cisco Nexus 9516 switches as virtual port channel (vPC) aggregation pairs. These aggregation nodes are fully loaded with N9K-X9432PQ, N9K-X9464PX, N9K-X9464TX, N9K-X9536PQ, and N9K-X9564TX line cards. The chassis is fully loaded with five line cards configured for breakout mode. The Cisco Nexus 9396PX and 93128TX switches are used as top-of-rack units with Cisco Nexus 3000 Series switches to achieve the desired vPC scale. The Cisco Nexus 9516 nodes are running in default routing mode. The Cisco Nexus 3164Q switch is also used as a core Layer 3 node that connects to a pair of vPC aggregation nodes.

The focus of the topology is to test IPv4 ARP, IPv6 neighbor discovery (ND), Layer 2 scalability, IPv4 and IPv6 LPM routing, Layer 2 and Layer 3 multicast routing for IPv4, and Layer 4 through Layer 7 features for management and operations. All Layer 3 interfaces are configured for dual stack, and the traffic is dual stack for all VLANs.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time (or multidimensional scale). These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 10. Verified Scalability Limits for a Layer 2/Layer 3 Aggregation Topology (Default Routing Mode)

Feature

9516 Series Verified Limit (Default Routing Mode)

Chassis configuration

5 N9K-X9432PQ line cards

4 N9K-X9464PX line cards

3 N9K-X9464TX line cards

3 N9K-X9536PQ line cards

1 N9K-X9564TX line card

Physical ports

735

vPCs

288

SVIs

401

VRFs

100

IPv4 ARP

20,000

IPv6 ND

10,000

Virtual ports

1500

BGP neighbors (IPv4 + IPv6)

64 + 64

IPv4 LPM routes

11,000

IPv6 LPM routes

1000

BFD (IPv4 + IPv6)

150

IGP OSPFv2 neighbors

100

IGP OSPFv3 neighbors

100

HSRP (IPv4 + IPv6)

401 + 401

IGMP groups

1680

Multicast *,G routes

1680

Tracking objects

100

VLANs

500

PIM neighbors

200

MAC addresses

30,500

Verified Scalability Limits for a Multicast System Topology

Two Cisco Nexus 9508 switches are configured as vPC peers in one domain, and two Cisco Nexus 9372PX switches are configured as vPC peers in the other domain. The chassis are fully loaded with N9K-X9432PQ, N9K-X9464PX, N9K-X9536PQ, N9K-X9564PX, N9K-X9564TX, and N9K-X9636PQ line cards. eBGP routing is used to connect these two PIM domains. OSPF is used as IGP in one domain, and EIGRP is configured in the other domain. This setup is configured with multiple rendezvous points (RPs) to serve different multicast group ranges. BSR is used to advertise RP information in both of these PIM domains. PIM anycast is used in one domain, and MSDP anycast is used in the other domain for redundancy and load balancing. Static RP configuration is also used for a range of multicast groups.

The Cisco Nexus 9516 and Cisco Nexus 7000 Series switches are used as Layer 3 core routers in one domain. The Cisco Nexus 3164Q switches are used as Layer 3 core routers in the other domain. This topology also includes the Cisco Nexus 9396PX, Cisco Nexus 9372PX, and Cisco Nexus 3016/3064T switches in the access layer.

In addition to including Layer 2/Layer 3 IPv4 multicast routing, this topology also covers IPv4 and IPv6 host and LPM routing and Layer 2 unicast forwarding. All interfaces are configured for dual stack.

In the following table, the Verified Limit column lists the verified scaling capabilities with all listed features enabled at the same time (or multidimensional scale). These numbers are not the maximum verified values if each feature is viewed in isolation.

Table 11. Verified Scalability Limits for a Multicast System Topology

Feature

9500 Series Verified Limit

9500 chassis configuration

N9K-X9432PQ, N9K-X9464PX, N9K-X9536PQ, N9K-X9564PX, N9K-X9564TX, and N9K-X9636PQ line cards

Multicast S,G routes

17,500

Multicast *,G routes

2500

Sources

2000 (2000, 200, 40, 10, 3, 2, 1)

Replications

40

ECMPs

16 (16,4,3,2 )

SVIs

200

HSRP/VRRP

200

MAC addresses

40,000

ARP

20,000

Unicast LPM IPv4 routes

20,000

Unicast LPM IPv6 routes

10,000

IPv4 ARP

20,000

IPv6 ND

4000

MSDP peers (fully mesh)

4

Anycast RPs (MSDP and PIM anycast) 5

4

5 This multicast system topology consists of two multicast PIM domains. The Multicast Source Discovery Protocol (MSDP) is used to exchange multicast source information between these two domains.