Cisco MDS NX-OS Configuration Limits
This document discusses the configuration and scalability limits for various SAN switching parameters in Cisco MDS NX-OS Release 8.x:
Change History
Date | Description |
---|---|
July 1, 2020 |
Cisco MDS 9718 switches support 8000 FLOGIs per switch. |
May 1, 2020 | The following scale values were updated:
|
Switch-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches
System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.
The following table lists the switch-level configuration limits for Cisco MDS 9000 Series switches:
Feature Name |
Parameter |
MDS 9700 Series |
MDS 9250i |
MDS 9148S |
MDS 9396S |
MDS 9132T |
MDS 9148T |
MDS 9396T |
---|---|---|---|---|---|---|---|---|
Fabric Logins |
FLOGIs or FDISCs per port |
2561 |
2561 |
2561 |
2561 |
2561 |
2561 |
2561 |
FLOGIs or FDISCs per module |
10002
20003
|
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
|
FLOGIs or FDISCs per switch |
40004
80005
|
400 |
1000 |
1000 |
2000 |
2000 |
20006
40007
|
|
N-Port Virtualization (NPV) |
NPV switches per NPIV core switch |
105 |
8 |
8 |
8 |
8 |
8 |
8 |
Inter-Switch Links (ISLs) |
ISL instances per switch 8
|
Up to 200 ISLs, each ISL with 16 VSANs, for a total number of 3200-port VSAN instances |
40 ISLs, each with 29 VSANs, for a total number of 1160 port VSAN instances |
48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances |
96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances |
32 ISLs, each with 29 VSANs, for a total number of 928 port VSAN instances |
48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances |
96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances |
Port channels (PC)9
|
Fibre Channel port channels |
256 |
40 |
48 |
96 |
32 |
48 |
96 |
Members in each Fibre Channel port channel |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
|
Ethernet port channels |
128 |
8 |
None |
None |
None |
None |
None |
|
Members in each Ethernet port channel |
16 |
8 |
None |
None |
None |
None |
None |
|
SSH 10
|
Simultaneous SSH sessions per switch |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
SAN Analytics and SAN Telemetry Streaming |
Active Initiator Target LUNs (ITLs) or Initiator Target Namespace ID (ITNs) per sampling interval per switch |
20,00011 40,00012
|
None |
None |
None |
20,000 |
20,000 |
20,000 |
Active ITLs or ITNs per sampling interval per module |
DS-X9648-1536K9: 20,000 |
None |
None |
None |
20,000 |
20,000 |
20,000 |
|
Telemetry receivers per switch |
2 |
None |
None |
None |
2 |
2 |
2 |
This is the recommended limit. FLOGIs or FDISCs are not rejected until 512 logins have been accepted on the port.
Supported in Cisco MDS 9706 and 9710 switches.
Supported in Cisco MDS 9718 switches.
Supported in Cisco MDS 9706 switches.
Supported in Cisco MDS 9710 switches in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.
Supported in Cisco MDS 9710 switches with Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2) and later releases.
Supported in Cisco MDS 9718 switches with Cisco MDS 9700 Series Supervisor-1 Module (DS-X97-SF1E-K9) or Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2a) and later releases.
Supported in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.
Supported from Cisco MDS NX-OS Release 8.4(2) and later releases.
ISLs can be an individual FC link, VFC link, FC port channel, or VFC port channel. One port channel interface with up to 16 member links is considered as one ISL instance.
A system message appears in the user’s session when the maximum supported limit of port channels and member ports is exceeded.
A system message appears on the user’s session when the maximum supported number of SSH sessions is exceeded.
Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.
When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.
Fabric-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches
The following table lists the configuration limits for Cisco MDS 9000 fabrics.
Feature Name |
Parameter |
MDS 9700-only Fabric (without MDS 9500, 9300, 9200, and 9100) |
MDS Mixed Fabric (mix of MDS 9700, 9500, 9300, 9200, 9100) |
---|---|---|---|
VSANs |
VSANs per physical fabric |
80 |
3213
|
Logins and aliases |
FCNS entries in fabric |
20000`14
|
1000015
130004
|
Device alias entries in fabric |
20000 |
8000 |
|
Domains and hops |
Domains per physical fabric |
80 |
80 |
Switch hops from server to storage |
7 |
7 |
|
Zones and Smart Zones |
Active zone sets |
1000 |
1000 |
Active zones |
16000 17
|
800018
104004 |
|
Active zone members19
|
32000 20 |
1600021
20800 4 |
|
Active zone database size |
3.8 MB 22
|
2 MB 3.8 MB4 |
|
Inter-VSAN Routing (IVR) |
Active IVR zone sets |
32 |
32 |
Active IVR zones |
200011
|
200011 |
|
Active IVR zone members |
4000 |
4000 |
|
Active IVR service groups |
16 |
16 |
|
CFS |
CFS peers12
|
80 |
80 |
CFS static peers over IP12 |
100 |
100 |
In a mixed fabric comprising Cisco MDS 9700 and 9500 director class switches and Cisco MDS 9148 and 9148S fabric switches, the fabric can have all the switches in the first 32 VSANs. The remaining VSANs include only those switches that support more than 32 VSANs.
An error message appears in your session if the number of FCNS entries exceeds 20000.
For platforms other than Cisco MDS 9700, a warning appears in your session if the number of FCNS entries exceeds 10000.
This value applies to Cisco MDS 9500 with Supervisor 2A. Thus, this limit applies to a fabric comprising of these switches and Cisco MDS 9700.
An error message and syslog appear in your session if the total number of zones exceeds 16000.
For platforms other than Cisco MDS 9700, a warning appears in your session if the number of zones exceeds 8000. However, further configurations are not blocked.
Without Smart Zones and for optimal hardware resource usage, configure 1 initiator and 1 target device per zone. The recommended maximum limit is 50 members per zone.
From Cisco MDS NX-OS Release 8.4(2), Smart Zones support a maximum of 250 members per zone only when the zone members are configured as initiators or targets.
For platforms other than Cisco MDS 9700, a warning appears in your session if the number of unique zone members exceeds 16000.
An error message appears on your session if the zone database size exceeds 3.8 MB.
2000 IVR zones are available across all VSANs. However, a single VSAN must not have more than 1000 IVR zones.
A system message appears in your session when the supported configuration limit of CFS peers is exceeded.
![]() Note |
If there are any switches in the fabric running Cisco MDS NX-OS 6.2(5) or earlier releases, the previous 2-MB zone database limit prevails and the new zone scale enhancements are not available. We recommend that the 2-MB zone database limit does not exceed, unless all the switches in the fabric run on Cisco MDS NX-OS Release 6.2(7) or later. |
Example: Checking the Size of the Zone Database
The following example shows how to check the size of the zone database in VSAN 310:
(config)# show zone status vsan 310 | Inc "Db Size" P 1
Full Zoning Database :
Db Size: 1040524 Bytes
--
Active Zoning Database :
Db Size: 962156 Bytesc
Now, Add 962156 with 1040524 = 2002680
2002680 / (1024 * 1024) = 1.9 MB.
![]() Note |
This database size does not include the pending changes in a session. |
Configuration Limits for Port Line Rate Throughput on Cisco MDS 9700 40 Gbps 24-Port FCoE Modules
For ports to operate at full-line rate on Cisco MDS 9706, MDS 9710, and MDS 9718 switches, the switch should have a specific configuration. The following table lists the maximum number of line rate ports for given Cisco MDS 9700 switch configurations.
No. of Fabric Cards |
Front Panel FCoE Bandwidth/Slot |
Maximum No. of Ports/Speed |
---|---|---|
3 |
660 Gbps |
16 ports/40 Gbps |
4 |
880 Gbps |
20 ports/40 Gbps |
5 |
1100 Gbps |
24 ports/40 Gbps |
![]() Note |
If the number of ports that you configure exceeds the maximum limit for the number of ports, it results in all ports operating in oversubscribed mode. |
Fibre Channel Topology Combinations and Scaling
From Cisco MDS NX0-OS Release 6.2(7), the following configurable features that support scale enhancements are supported:
-
Fibre Channel Name Server (FCNS) bulk notification
-
Coalesce switch Registered State Change Notification (SW-RSCN)
For more information about the FCNS and SW-RSCN features, see the Cisco MDS 9000 Series NX-OS Fabric Configuration Guide and the Cisco MDS 9000 Family Command Reference documents.
![]() Note |
In multidimensional scale configurations, supervisors with 1-GB memory may experience the system manager hap-reset error. |
The following table provides the configuration limits with and without the FCNS and SW-RSCN optimizations.
Scale Topology |
FCNS Bulk Notification |
Coalesce SW-RSCN |
Configuration Limits |
Scale Optimizations |
---|---|---|---|---|
Cisco MDS NX-OS Release 6.2(9) and later on Cisco MDS 9700 Series-only Fabric |
On |
On |
MDS NX-OS Release 6.2(7) and later |
Enabled |
FLOGI per module: 1000 FLOGI node: 4000 FCNS: 20000 Zone: 16000 Zone members: 32000 |
||||
Off |
Off |
MDS NX-OS Release 6.2(5) and earlier |
Disabled (Disabled by default in Cisco MDS NX-OS Release 6.2(7)) |
|
FLOGI per module: 500 FLOGI node: 2500 FCNS: 10000 Zone: 8000 Zone members: 16000 |
||||
Cisco MDS NX-OS Release 6.2(9) and later on MDS Mixed Fabric25 (Cisco MDS 9700 Series, 9500, 9250i, 9222i, and 9148)
|
On or off |
On or off |
FLOGI per module: 400 FLOGI node: 2000 FCNS: 10000, 13000 26
Zone: 8000, 10400 27
Zone members: 16000, 20800 28
|
Enabled or disabled |
The scale enhancements introduced in Cisco MDS NX-OS 6.2(7) and later are available on Cisco MDS 9700 Series Switches (only networks with Cisco MDS NX-OS Release 6.2(7) and later, running on all the switches and the configurable optimizations): FCNS bulk notification and RSCN coalesce enabled. These enhancements are not available in mixed fabrics regardless of whether FCNS and RSCN are enabled. In Cisco MDS NX-OS Release 6.2(9) and later, FCNS bulk notification is enabled by default. (To disable FCNS bulk notification, use the fcns no-bulk-notify command.) Coalesce SW-RSCN is disabled by default.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
Fibre Channel Scale Limitations
The various Fibre Channel scale limitations are described in this section.
Limitation 1
From Cisco MDS NX-OS Release 6.2(7), the fcns bulk-notify and rscn coalesce swrscn vsan commands are available on all Cisco MDS switches. We recommend that you use these commands only on Cisco MDS 9700 Series switches with 48-port 16-Gbps Fibre Channel switching module because the higher configuration limits are currently not supported on any other MDS platform.
![]() Note |
The FCNS bulk notification and Coalesce SW-RSCN features are supported only in NX-OS Release 6.2(7) and later releases. Enable the Coalesce SW-RSCN only if all the switches in a fabric are Cisco MDS switches that are running Cisco MDS NX-OS Release 6.2(7) or later. |
Limitation 2
The maximum zone database size is increased from 2 to 3.8 MB in Cisco MDS Release 6.2(7) and later for all MDS 9700 Series switches to enable zone scale enhancements. The new limit of 16000 zones is supported only on the Cisco MDS 9700 only fabric. Fabrics with Cisco MDS 9500, 9200, or 9100 Series switches continue to have the 2-MB zone database limit, supporting only up to 8000 zones.
Limitation 3
In a three-node serial topology, traffic imbalance may occur if the number of port channels that are configured between the switches are the same. We recommend that you have a single port channel between two switches with any number of member ports. If you configure more than one port channel, ensure that the count of port channels between the switches varies.
Limitation 4
The maximum latency (round-trip time) and the packet-drops that are supported on FCIP links are 250 ms and 0.05 percent receptively.
![]() Note |
The maximum limit is the same regardless whether latency and packet drop conditions exist together or only one of the conditions exists. |
Syslog and Warnings for Fibre Channel Network Scale
The following table lists the syslogs and warnings for the Fibre Channel network.
Syslog and Warnings |
Cisco MDS 9700 Series |
Cisco MDS 9500, 9200, and 9100 |
---|---|---|
Zone limit |
16000 zones – Hard limit: Syslog warning that states no more zones can be configured:
|
8000 zones – Hard limit: Syslog warning that states no more zones can be configured:
10400 zones – Hard limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning that states no more zones can be configured:
|
FCNS limit |
20000 FCNS entries – Hard limit: Syslog warning stating that no more name server entries are supported:
|
10000 FCNS entries – Soft limit: Syslog warning about validated limit:
13000 FCNS entries – Soft limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning about validated limit:
20000 FCNS entries – Hard limit: Syslog warning:
|
Switch-Level Configuration Limits for Fibre Channel over Ethernet
System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.
The following table lists the switch-level configuration limits for Fibre Channel over Ethernet (FCoE) on Cisco MDS.
Feature Name |
MDS 9710 with 48-Port 10-Gigabit FCoE |
---|---|
FLOGIs per Port |
256 |
FLOGIs per Module |
1000 |
FLOGIs per Switch |
4000 |
VSAN |
80 |
VSAN-VLAN Mapping |
80 |
Virtual Fibre Channel Port Channel and Member Ports |
128 virtual fibre channel port channel and a maximum number of 16 members in one virtual fibre channel port channel |
Fabric-Level Configuration Limits for Fibre Channel over IP
The following table lists the fabric-level configuration limits for Fibre Channel over IP (FCIP):
Feature Name |
Cisco MDS 9250i Multiservice Fabric Switch |
Maximum latency (round-trip time) and packet drops supported on FCIP links |
250-ms round trip and 0.05% packet drop |
Feature Name |
Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module |
Maximum latency (round-trip time) and packet drops supported on FCIP links |
250-ms round trip and 0.05% packet drop |
Fabric-Level Configuration Limits for Fibre Channel over Ethernet (FCoE) on MDS 9700 Series
The following table lists the fabric-level configuration limits for the Fibre Channel over Ethernet (FCoE) on MDS 9700 Series.
Feature Name |
MDS 9700 Network with 48-Port 10-Gigabit Fibre Channel over Ethernet Module |
---|---|
Zones |
16000 |
Zone members |
32000 |
Zone sets |
1000 |
Zone database size |
3.8 MB 29
|
Number of FCNS entries in a network |
20000 |
Device alias |
12000 |
An error message appears in your console if the zone database size exceeds 3.8 MB.
![]() Note |
|
Example: Checking the Size of the Zone Database
The following example shows how to check the size of the zone database in VSAN 310:
(config)# show zone status vsan 310 | Inc "Db Size" P 1
Full Zoning Database :
Db Size: 1040524 Bytes
--
Active Zoning Database :
Db Size: 962156 Bytesc
Now, Add 962156 with 1040524 = 2002680
2002680 / (1024 * 1024) = 1.9 MB.
![]() Note |
This database size does not include the pending changes in a session. |
Cisco MDS 9000 I/O Accelerator Configuration Limits
The following table lists the Cisco MDS 9000 I/O Accelerator configurations and the corresponding limits.
Parameter |
MSM-18/4 or SSN-16 Module on MDS 9222i and MDS 9500 Modular Chassis and MDS 9250i Fabric Switch, 24/10-Port SAN Extension Module on MDS 9710 Chassis |
||
---|---|---|---|
Number of switches in a cluster |
4 |
||
Number of clusters per switch |
16 |
||
Number of switches in a SAN fabric for FC-Redirect |
34 |
||
Number of hosts per target |
128 |
||
Number of concurrent flows per Cisco MDS 9000 I/O Accelerator service engine |
128 |
||
Number of flows per Cisco MDS 9000 I/O Accelerator service engine (hard limit) |
128 - Release 4.2(1) on MDS 9222i and MDS 9500 512 - Release 4.2(7) or later on MDS 9222i and MDS 9500 512 - Release 6.2(5) or later on MDS 9250i 512 - Release 8.2(1) or later on MDS 9710 |
||
Number of flows per Cisco MDS 9000 I/O Accelerator service engine (soft limit) |
64 - Release 4.2(1) on MDS 9222i/MDS 9500 256 - Release 4.2(7) or later on MDS 9222i/MDS 9500 256 - Release 6.2(5) or later on MDS 9250i
256 - Release 8.2(1) or later on MDS 9710 |
||
Number of flows in a cluster |
1024 - Release 4.2(7d) 1248 - Release 5.2(6b) or later |
Fibre Channel Flow Configuration Limits
The following table lists the Fibre Channel flow configurations limits.
Cisco MDS Device |
Aggregate Flow and Flow Statistics Limit |
Flow Statements per Module |
---|---|---|
Generation 1 Modules |
1000 |
1024 |
Generation 2 Modules |
2000 |
2048 |
Generation 3 Modules |
512 |
512 |
Generation 4 Modules |
512 |
512 |
Generation 5 Modules |
512 |
512 |