Chapter 1. Verified Scalability for Cisco Nexus 5600 Series NX-OS Release 7.2(0)N1(1)

Overview of Verified Scalability

Verified Scalability for a Layer 2 Switching Deployment

Verified Scalability for a Layer 2 Switching and Layer 3 Routing Deployment

Verified Scalability for a Layer 3 Aggregation Routing Deployment

Verified Scalability for a Layer 2 Switching and Unified Fabric (FCoE) Deployment

Verified Scalability for Multicast Routing

Verified Scalability for Unicast Routing

Verified Scalability for VxLAN VPC Pair

Preface

Preface

Audience

Document Conventions

Related Documentation

Documentation Feedback

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. Verified Scalability for Cisco Nexus 5600 Series NX-OS Release 7.2(0)N1(1)

This chapter contains the following sections:

- Overview of Verified Scalability

- Verified Scalability for a Layer 2 Switching Deployment

- Verified Scalability for a Layer 2 Switching and Layer 3 Routing Deployment

- Verified Scalability for a Layer 3 Aggregation Routing Deployment

- Verified Scalability for a Layer 2 Switching and Unified Fabric (FCoE) Deployment

- Verified Scalability for Multicast Routing

- Verified Scalability for Unicast Routing

- Verified Scalability for VxLAN VPC Pair

Overview of Verified Scalability

This document lists the Cisco verified scalability limits.

In the following tables, the Verified Topology column lists the verified scaling capabilities with all listed features enabled at the same time. The numbers listed here exceed those used by most customers in their topologies. The scale numbers listed here are not the maximum verified values if each feature is viewed in isolation.

The Verified Maximum column lists the maximum scale capability tested for the corresponding feature individually. This number is the absolute maximum currently supported by the Cisco NX-OS Release software for the corresponding feature. If the hardware is capable of a higher scale, future software releases may increase this verified maximum limit.

Verified Scalability for a Layer 2 Switching Deployment

This table lists the verified scalability for a Layer 2 switching deployment.

Verified Scalability for a Layer 2 Switching Deployment

Feature

Verified Topology

Verified Maximum

Active VLANs/VSANs per switch

4000

4013 (31 are reserved for VSANs, and the remaining are for VLANs.)

VLAN/VSAN ID Space

4013

4013 Unreserved Space

Virtual Ports

[1]

Verified Value

Mode

48000

Rapid PVST+

96000

MST

VLAN ACLs (VACLs)

128 (10 Unique VACLs)

1024 (512 unique VACLs with up to 1024 ACE entries across all VACLs)

Maximum interfaces per EtherChannel

16

16

IGMP Snooping Groups

4000 (in FEX deployments)

8000 (in non-FEX deployments)

4000 (in FEX deployments)

16,000 (in non-FEX deployments)

Maximum FEXs per Switch

24

24

48 (for Cisco Nexus 5696Q Switch)

48 (for Cisco Nexus 5696Q Switch)

Maximum FEXs dual-homed to a vPC Switch Pair

24

24

48 (for Cisco Nexus 5696Q Switch)

48 (for Cisco Nexus 5696Q Switch)

MAC Table Size (Entries)

64,000

115,000

Number of Switchport Etherchannels

48

The following three values apply to the Cisco Nexus 5672:

- 6 (Single member port-channel for 40G ports)

- 72 (Single member port-channel for 10G ports)

- 36 (Multi member port-channel)

The following three values apply to the Cisco Nexus 56128:

- 8 (Single member port-channel for 40G ports)

- 128 (Single member port-channel for 10G ports)

- 64 (Multi member port-channel)

Number of HIF FEX port channels/vPCs (across the maximum number of FEXs)

576

576

SPAN Sessions

4 active sessions

32 source VLANs as a RX source

16 active sessions

32 source VLANs as a RX source

FabricPath VLANs

4000[2]

4000

FabricPath Switch IDs

500

500

FabricPath Multicast Trees

1

1

Number of FabricPath Topologies

2

2

Number of FabricPath Core Port-Channels

4 (4 member ports per link)

16

FEX Host Interface Storm Control

1936 [3]

1936

Segmentation ID

3000 (1000 global segments, 2000 local segments)

3000 (1000 global segments, 2000 local segments)

[1] Virtual ports are a product of the number of VLANs times the number of physical ports.

[2] FabricPath VLANs are verified in the unified fabric topology

[3] This is the target maximum number that HIF-SC can support. Beyond this number, NIF-SC is recommended for deployment.

Verified Scalability for a Layer 2 Switching and Layer 3 Routing Deployment

This table contains the verified scalability for a Layer 2 switching and Layer 3 routing deployment.

Note

The currently tested values do not provide an indication for the maximum scalability of the control plane. These numbers vary based on the load of the system in terms of routing protocols, timers settings, and other values. Proof of concept testing should be used to determine the scalability of a given feature for your environment.

Verified Scalability for a Layer 2 Switching and Layer 3 Routing Deployment

Feature

Verified Topology

Verified Maximum

Active VLANs/VSANs per Switch

1000

4013 (31 are set reserved for VSANs and the remaing are for VLANs)

VLAN/VSAN ID Space

4013 Unreserved space

4013 Unreserved space

STP Instances

16,000

16,000

Maximum Interfaces per EtherChannel

16

16

IGMP Snooping Groups

4000 (in FEX deployments)

8000 (in non-FEX deployments)

4000 (in FEX deployments)

16,000 (in non-FEX deployments)

Maximum FEXs per Switch

24

24

32 (for Cisco Nexus 5696 switch)

32 (for Cisco Nexus 5696 switch)

Maximum FEXs Dual-homed to a vPC Switch Pair

24

24

32 (for Cisco Nexus 5696 switch)

32 (for Cisco Nexus 5696 switch)

MAC Table Size (Entries)

32,000

64,000

[1]

Number of FEX Port Channels/vPCs (across the maximum number of FEXs)

500

768

SPAN Sessions

4 active sessions

32 source VLANs as an RX source

16 active sessions

32 source VLANs as an RX source

Number of SVIs

564

564

Dynamic IPv4 Routes

[2]

8000

24,000

Dynamic IPv6 Routes

[3]

4000

[4]

6000

Multicast IPv4 Routes

[5]

4000

8000

ARPs (IPv4 Hosts)

[6]

32,000

64,000

IPv6 Hosts

16,000

32,000

VRFs

25

1000

RACLs

64 Ingress RACLs with up to 1152 ACE entries across all the RACLs

896 Ingress RACLs with up to 1152 ACE entries across all the RACLs

HSRP Groups

[7]

256

500

VRRP Groups

[8]

256

500

BFD sessions over Layer 3 interface for CE mode

8 sessions (250ms interval, 750ms dead-interval)

30 (250ms interval, 750ms dead-interval)

BFD sessions over SVI for L2MP mode

64 (250ms intvl, 750ms dead-intvl)

64 (250ms intvl, 750ms dead-intvl)

PBR IPv4

95

95

PBR IPv6

95

95

[1] 128,000 entries are reserved for unicast MAC entries and 128,000 entries are reserved for IP host routes.

[2] The maximum number of entries that can be supported is 24,000. This table is shared between IPv4 and IPv6. An IPv4 route takes up one entry in the table and an IPv6 route takes up four entries.

[3] The maximum number of entries that can be supported is 24,000. This table is shared between IPv4 and IPv6. An IPv4 route takes up one entry in the table and an IPv6 route takes up four entries.

[4] With no IPv4, the number increases to 6000.

[5] This includes (*,G) entries, (S,G) entries, and the entries required for vPC with bind-vrf configured. When bind-vrf is configured, each (*,G) and (S,G) entry is replicated.

[6] The maximum number of hosts supported is listed under ARPs. This includes IPv4 and IPv6 hosts. IPv4 hosts take up one entry and IPv6 hosts take up 2 entries in hardware. So, for the verified limit, the switch supports one of the following: 32,000 IPv4 hosts and 0 IPv6 hosts, 64,000 IPv6 hosts and 0 IPv4 hosts, or a combination of IPv4 and IPv6 hosts.

[7] The limit of the table that holds the Router MAC and Virtual MAC entries that determines whether the packet needs to be bridged or routed is 500 entries. The Virtual MAC entries can be shared across Layer 3 interfaces. So, we recommend that you configure the same group ID across all or multiple Layer 3 interfaces/SVIs. If multiple group IDs are configured on an Layer 3 interface, then we recommend that you configure the same set of group IDs across all or multiple Layer 3 interfaces. This way, HSRP/VRRP can be supported on more interfaces. Please refer to the unicast configuration guide for more information.

[8] The limit of the table that holds the Router MAC and Virtual MAC entries for determining packet routing or switching is 500 entries. The Virtual MAC entries can be shared across Layer 3 interfaces. So, we recommend that you configure the same group ID across all or multiple Layer 3 interfaces/SVIs. If multiple group IDs are configured on an Layer 3 interface, then we recommend that you configure the same set of group IDs across all or multiple Layer 3 interfaces. This way, HSRP/VRRP can be supported on more interfaces. Please refer to the unicast configuration guide for more information.

Verified Scalability for a Layer 3 Aggregation Routing Deployment

This table lists the verified scalability for a Layer 3 aggregation routing deployment.

Verified Scalability for a Layer 3 Aggregation Routing Deployment

Feature

Verified Topology

Verified Maximum

Active VLANs/VSANs per Switch

4000

4000

VLAN/VSAN ID Space

4013 unreserved space

4013 unreserved space

STP Instances

64,000

64,000

Maximum Interfaces per EtherChannel

16

16

IGMP Snooping Groups

8000 (in non-FEX deployments)

16,000 (in non-FEX deployments)

MAC Table Size

64,000

[1]

115,000

[2]

SPAN Sessions

4 active sessions

32 source VLANs as an RX source

16 active sessions

32 source VLANs as an RX source

SVIs

1000

4000

Dynamic IPv4 Routes

[3]

8000

24,000

Dynamic IPv6 Routes

[4]

4000

[5]

6000

Multicast IPv4 Routes

[6]

8000

32,000

RACLs

64 ingress RACLs with up to 1,152 ACE entries across all of the RACLs.

512 ingress RACLs with up to 1,152 ACE entries across all of the RACLs.

VRFs

25

1,000

ARPs (IPv4 Hosts)

[7]

64,000

64,000

IPv6 Hosts

[8]

10,000

[9]

32,000

IGP Peers

64

100

HSRP Groups

[10]

1721 (911 IPv4, 810 IPv6)

1721 (911 IPv4, 810 IPv6)

VRRP Groups

[11]

500

500

FabricPath Switch IDs

500

500

FabricPath Multicast Trees

2

2

Number of FabricPath Topologies

2

2

Number of FabricPath Core Links

32

32

PBR IPv4

110

110

PBR IPv6

110

110

BFD sessions over Layer 3 interface for CE Mode

100 sessions (150ms interval, 750ms dead-interval)

100 sessions (250ms interval, 750ms dead-interval)

BFD sessions over SVI for L2MP mode

64 sessions (150ms interval, 750ms dead-interval)

64 sessions (250ms interval, 750ms dead-interval)

[1] 128,000 entries are reserved for Unicast MAC entries, and 128,000 entries are reserved for IP host routes.

[2] 128,000 entries are reserved for Unicast MAC entries, and 128,000 entries are reserved for IP host routes.

[3] The maximum number of entries that can be supported is 24,000. This table is shared between IPv4 and IPv6. An IPv4 route takes up one entry in the table and an IPv6 route takes up four entries.

[4] The maximum number of entries that can be supported is 24,000. This table is shared between IPv4 and IPv6. An IPv4 route takes up one entry in the table and an IPv6 route takes up four entries.

[5] Entries shared between IPv4, IPv6 network routes.

[6] This includes (*,G) entries, (S,G) entries, and the entries required for vPC with bind-vrf configured. When bind-vrf is configured, each (*,G) and (S,G) entry is replicated. This includes (*,G) entries, (S,G) entries, and IGMP-snooping entries combined.

[7] The maximum number of hosts supported is listed under ARPs. This includes IPv4 and IPv6 hosts. IPv4 hosts take up one entry and IPv6 hosts take up 2 entries in hardware. So, for the verified limit, the switch supports one of the following: 64,000 IPv4 hosts and 0 IPv6 hosts, 16,000 IPv6 hosts and 0 IPv4 hosts, or a combination of IPv4 and IPv6 hosts.

[8] The maximum number of hosts supported is listed under ARPs. This includes IPv4 and IPv6 hosts. IPv4 hosts take up one entry and IPv6 hosts take up 2 entries in hardware. So, for the verified limit, the switch supports one of the following: 64,000 IPv4 hosts and 0 IPv6 hosts, 16,000 IPv6 hosts and 0 IPv4 hosts, or a combination of IPv4 and IPv6 hosts.

[9] Entries shared between IPv4 multicast, IPv4, IPv6 host routes .

[10] The limit of the table that holds the Router MAC and Virtual MAC entries that determine whether the packet needs to be bridged or routed is 500 entries. The Virtual MAC entries can be shared across Layer 3 interfaces. So we recommend that you configure the same group ID across all or multiple Layer 3 interfaces/SVIs. If multiple group IDs are configured on a Layer 3 interface, then we recommend that you configure the same set of group IDs across all or multiple Layer 3 interfaces. This way, HSRP/VRRP can be supported on more interfaces. Please refer to the Unicast Routing Configuration Guide for more information.

[11] The limit of the table that holds the Router MAC and Virtual MAC entries for determining packet routing or switching is 500 entries. The Virtual MAC entries can be shared across Layer 3 interfaces. So we recommend that you configure the same group ID across all or multiple Layer 3 interfaces/SVIs. If multiple group IDs are configured on a Layer 3 interface, then we recommend that you configure the same set of group IDs across all or multiple Layer 3 interfaces. This way, HSRP/VRRP can be supported on more interfaces. Please refer to the Unicast Routing Configuration Guide for more information.

Verified Scalability for a Layer 2 Switching and Unified Fabric (FCoE) Deployment

This table lists the verified scalability for a Layer 2 switching and unified fabric (FCoE) deployment.

Verified Scalability for a Layer 2 Switching and Unified Fabric (FCoE) Deployment

Feature

Verified Topology

Verified Maximum

Active VLANs/VSANs per switch

4,013

4,013 (32 are set reserved for VSANs and the remaining are for VLANs)

VLAN/VSAN ID Space

4,013 Unreserved Space

4,013 Unreserved Space

Logical Interfaces

[1]

32,000

32,000

IGMP Snooping Groups

4,000 (in FEX deployments)

8,000 (in non-FEX deployments)

4,000 (in FEX deployments)

16,000 (in non-FEX deployments)

Maximum FEXs per Cisco Nexus 5600 Series Switch

17 per DUT

24

Maximum FEXs Dual Homed to a vPC Cisco Nexus 5600 Series Switch Pair

34

34

MAC Table Size

32,000 unicast entries

[2]

115,000 entries

[3]

Number of Switchport EtherChannels

8

384

SPAN Sessions

4 active sessions

32 source VLANs as a RX source

16 active sessions

32 source VLANs as a RX source

Number of FEX Port Channels/vPCs (across the maximum number of FEXs)

544

768

FabricPath VLANs

4,000

4,000

FabricPath Switch IDs

500

500

FabricPath Multicast Trees

2

2

Number of FabricPath Topologies

2

2

Number of FabricPath Core Links

2

16

FLOGIs or FDISCs per NPV Port Group

512

512

Zone Sets per Switch

32

500

Zone Members per Physical Fabric (includes all VSANs)

16,000

16,000

Zones per Switch (includes all VSANs)

8000

8000

Maximum Diameter of a SAN Fabric

7 hops

12

FSPF Interface Instances per Switch

192

3,072

ISL Instances per Switch

6

96

VFC Interfaces

544

544

Maximum FCIDs Allocated

1024

4000

Fibre Channel Flows

32

32

FLOGIs or FDISCs per switch

1024

4000

The maximum number of vFCs that can be bound to a port-channel

48

48

[1] Logical interfaces are a product of the number of VLANs times the number of ports. This parameter reflects the load of handling port programming, and is not dependent on the spanning-tree mode or configuration.

[2] 128,000 entries are reserved for unicast MAC entries and 128,000 entries are reserved for IP host routes.

[3] 128,000 entries are reserved for unicast MAC entries and 128,000 entries are reserved for IP host routes.

Verified Scalability for Multicast Routing

This table lists the verified scalability for multicast routing.

Verified Scalability for Multicast Routing

Feature

Parameter

Verified Maximum

Protocol Independent Multicast (PIM)

Number of neighbors

500

Number of neighbors/total routes per system with aggressive hello timers (5 seconds)

16/4,000

Multicast Source Discovery Protocol (MSDP)

Number of MSDP Source-Active (SA) cache entries

6,000

Note

- In vPC setup, TCAM exhaustion failure will lead to some routes not getting programmed in the hardware. Hence, there might exist a condition where mrib will show the route exists but mfib may not have it programmed.

- In bind-vrf configuration, for every mroute, additional mroute is programmed in the hardware and this could lead to TCAM exhaustion. Hence, ensure that the mroute count does not exceed ((max-limit/2) - 4) default routes.

For example: If the hardware profile multicast max-limit is 8000, then mroute count should not exceed ((8000/2)-4) default routes.

Verified Scalability for Unicast Routing

Guidelines and Limitations for Unicast Routing

- You can have up to four instances of OSPFv2.

- You can have up to four instances of OSPFv3.

This table lists the verified scalability for unicast routing.

Verified Scalability for Unicast Routing

Feature

Parameter

Verified Maximum

OSPFv2

Number of active interfaces

256

Number of passive interfaces

256

Number of neighbors/total routes with aggressive timers (1 sec/ 3 sec)

16/6,000

OSPFv3

Number of active interfaces

256

Number of passive interfaces

256

EIGRP

Number of active interfaces

50

BGP

Number of peers (iBGP and eBGP, active)

256

Number of AS path entries

512

Number of prefix-list entries in a single prefix-list

10,000

HSRP

Number of groups with aggressive timers (1 sec/3 sec)

500

L3 ISIS

Number of adjacencies

100

Unicast Adjacencies

Number of regular adjacencies

16374

Number of ECMP adjacencies. The regular adjacencies are also part of the maximum ECMP adjacencies

28470

Verified Scalability for VxLAN VPC Pair

Verified Scalability for VxLAN VPC Pair

Feature

Verified Maximum

VTEP Peers 1K

872 remote peers and 128 peers.

Local VTEP Peers (hypervisors)

128

Multicast Distribution Groups

200

Number of VNIs

650

SVIs

900

MAC addresses

64000

L3 Routes

8000

Multicast Groups

4000

VRFs

125

FEXes

24

Preface

The preface contains the following sections:

- Preface

Preface

This preface describes the audience, organization, and conventions of the Book Title. It also provides information on how to obtain related documentation.

This chapter includes the following topics:

- Audience

- Document Conventions

- Related Documentation

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Audience

This publication is for experienced network administrators who configure and maintain Cisco NX-OS on Cisco Nexus 5000 Series Platform switches.

Document Conventions

Note

- As part of our constant endeavor to remodel our documents to meet our customers' requirements, we have modified the manner in which we document configuration tasks. As a result of this, you may find a deviation in the style used to describe these tasks, with the newly included sections of the document following the new format.

- The Guidelines and Limitations section contains general guidelines and limitations that are applicable to all the features, and the feature-specific guidelines and limitations that are applicable only to the corresponding feature.

Command descriptions use the following conventions:

Convention

Description

bold

Bold text indicates the commands and keywords that you enter literally as shown.

Italic

Italic text indicates arguments for which the user supplies the values.

[x]

Square brackets enclose an optional element (keyword or argument).

[x | y]

Square brackets enclosing keywords or arguments separated by a vertical bar indicate an optional choice.

{x | y}

Braces enclosing keywords or arguments separated by a vertical bar indicate a required choice.

[x {y | z}]

Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element.

variable

Indicates a variable for which you supply values, in context where italics cannot be used.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

Examples use the following conventions:

Convention

Description

screen font

Terminal sessions and information the switch displays are in screen font.

boldface screen font

Information you must enter is in boldface screen font.

italic screen font

Arguments for which you supply values are in italic screen font.

< >

Nonprinting characters, such as passwords, are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

This document uses the following conventions:

Note

Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Caution

Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Related Documentation

Documentation for Cisco Nexus 5000 Series Switches is available at:

- Configuration Guides

Click Here

- Command Reference Guides

Click Here

- Release Notes

Click Here

- Install and Upgrade Guides

Click Here

- Licensing Guide

Click Here

Documentation for Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders is available at:

Click Here

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to nexus5k-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation.

To receive new and revised Cisco technical content directly to your desktop, you can subscribe to the What's New in Cisco Product Documentation RSS feed. RSS feeds are a free service.

Full Cisco Trademarks with Software License

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)