Cisco MDS NX-OS Configuration Limits

This document discusses the configuration and scalability limits for various SAN switching parameters in Cisco MDS NX-OS Release 8.x:

Change History

Date Description
July 1, 2020

Cisco MDS 9718 switches support 8000 FLOGIs per switch.
May 1, 2020 The following scale values were updated:

  • Cisco MDS 9710 switches support 8000 FLOGIs per switch.

  • Cisco MDS 9396T switches support 4000 FLOGIs per switch.

  • Smart Zones support a maximum of 250 members per zone.

Switch-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level configuration limits for Cisco MDS 9000 Series switches:

Table 1. Switch-Level Configuration Limits

Feature Name

Parameter

MDS 9700 Series

MDS 9250i

MDS 9148S

MDS 9396S

MDS 9132T

MDS 9148T

MDS 9396T

Fabric Logins

FLOGIs or FDISCs per port

2561

2561

2561

2561

2561

2561

2561

FLOGIs or FDISCs per module

10002
20003

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

FLOGIs or FDISCs per switch
40004
80005

400

1000

1000

2000

2000
20006
40007

N-Port Virtualization (NPV)

NPV switches per NPIV core switch

105

8

8

8

8

8

8

Inter-Switch Links (ISLs)

ISL instances per switch 8

Up to 200 ISLs, each ISL with 16 VSANs, for a total number of 3200-port VSAN instances

40 ISLs, each with 29 VSANs, for a total number of 1160 port VSAN instances

48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

32 ISLs, each with 29 VSANs, for a total number of 928 port VSAN instances

48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

Port channels (PC)9

Fibre Channel port channels

256

40

48

96

32

48

96

Members in each Fibre Channel port channel

16

16

16

16

16

16

16

Ethernet port channels

128

8

None

None

None

None

None

Members in each Ethernet port channel

16

8

None

None

None

None

None
SSH 10

Simultaneous SSH sessions per switch

16

16

16

16

16

16

16

SAN Analytics and SAN Telemetry Streaming

Active Initiator Target LUNs (ITLs) or Initiator Target Namespace ID (ITNs) per sampling interval per switch

20,00011

40,00012

None

None

None

20,000

20,000

20,000

Active ITLs or ITNs per sampling interval per module

DS-X9648-1536K9: 20,000

None

None

None

20,000

20,000

20,000

Telemetry receivers per switch

2

None

None

None

2

2

2
1

This is the recommended limit. FLOGIs or FDISCs are not rejected until 512 logins have been accepted on the port.

2

Supported in Cisco MDS 9706 and 9710 switches.

3

Supported in Cisco MDS 9718 switches.

4

Supported in Cisco MDS 9706 switches.

Supported in Cisco MDS 9710 switches in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.

5

Supported in Cisco MDS 9710 switches with Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2) and later releases.

Supported in Cisco MDS 9718 switches with Cisco MDS 9700 Series Supervisor-1 Module (DS-X97-SF1E-K9) or Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2a) and later releases.

6

Supported in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.

7

Supported from Cisco MDS NX-OS Release 8.4(2) and later releases.

8

ISLs can be an individual FC link, VFC link, FC port channel, or VFC port channel. One port channel interface with up to 16 member links is considered as one ISL instance.

9

A system message appears in the user’s session when the maximum supported limit of port channels and member ports is exceeded.

10

A system message appears on the user’s session when the maximum supported number of SSH sessions is exceeded.

11 Supported in Cisco MDS NX-OS Release 8.3(1) and earlier releases.
12

Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.

When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.

Fabric-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches

The following table lists the configuration limits for Cisco MDS 9000 fabrics.

Table 2. Fabric-Level Configuration Limits

Feature Name

Parameter

MDS 9700-only Fabric

(without MDS 9500, 9300, 9200, and 9100)

MDS Mixed Fabric

(mix of MDS 9700, 9500, 9300, 9200, 9100)

VSANs

VSANs per physical fabric

80

3213

Logins and aliases

FCNS entries in fabric

20000`14
1000015
130004

Device alias entries in fabric

20000

8000

Domains and hops

Domains per physical fabric

80

80

Switch hops from server to storage

7

7

Zones and Smart Zones

Active zone sets

1000

1000

Active zones

16000 17
800018

104004
Active zone members19

32000 20

1600021

20800 4

Active zone database size

3.8 MB 22

2 MB

3.8 MB4

Inter-VSAN Routing (IVR)

Active IVR zone sets

32

32

Active IVR zones

200011

200011

Active IVR zone members

4000

4000

Active IVR service groups

16

16

CFS

CFS peers12

80

80

CFS static peers over IP12

100

100

13

In a mixed fabric comprising Cisco MDS 9700 and 9500 director class switches and Cisco MDS 9148 and 9148S fabric switches, the fabric can have all the switches in the first 32 VSANs. The remaining VSANs include only those switches that support more than 32 VSANs.

14

An error message appears in your session if the number of FCNS entries exceeds 20000.

15

For platforms other than Cisco MDS 9700, a warning appears in your session if the number of FCNS entries exceeds 10000.

16

This value applies to Cisco MDS 9500 with Supervisor 2A. Thus, this limit applies to a fabric comprising of these switches and Cisco MDS 9700.

17

An error message and syslog appear in your session if the total number of zones exceeds 16000.

18

For platforms other than Cisco MDS 9700, a warning appears in your session if the number of zones exceeds 8000. However, further configurations are not blocked.

19

Without Smart Zones and for optimal hardware resource usage, configure 1 initiator and 1 target device per zone. The recommended maximum limit is 50 members per zone.

From Cisco MDS NX-OS Release 8.4(2), Smart Zones support a maximum of 250 members per zone only when the zone members are configured as initiators or targets.

20 An error message and syslog appear in your session if the total number of the unique zone members exceeds 32000.
21

For platforms other than Cisco MDS 9700, a warning appears in your session if the number of unique zone members exceeds 16000.

22

An error message appears on your session if the zone database size exceeds 3.8 MB.

23

2000 IVR zones are available across all VSANs. However, a single VSAN must not have more than 1000 IVR zones.

24

A system message appears in your session when the supported configuration limit of CFS peers is exceeded.


Note

If there are any switches in the fabric running Cisco MDS NX-OS 6.2(5) or earlier releases, the previous 2-MB zone database limit prevails and the new zone scale enhancements are not available. We recommend that the 2-MB zone database limit does not exceed, unless all the switches in the fabric run on Cisco MDS NX-OS Release 6.2(7) or later.

Example: Checking the Size of the Zone Database

The following example shows how to check the size of the zone database in VSAN 310: 


(config)# show zone status vsan 310 | Inc "Db Size" P 1

Full Zoning Database :
    Db Size: 1040524 Bytes
--
Active Zoning Database :
    Db Size: 962156 Bytesc

Now, Add 962156 with 1040524 = 2002680

2002680 / (1024 * 1024) = 1.9 MB.


Note

This database size does not include the pending changes in a session.

Configuration Limits for Port Line Rate Throughput on Cisco MDS 9700 40 Gbps 24-Port FCoE Modules

For ports to operate at full-line rate on Cisco MDS 9706, MDS 9710, and MDS 9718 switches, the switch should have a specific configuration. The following table lists the maximum number of line rate ports for given Cisco MDS 9700 switch configurations.

Table 3. Line Rate Limitation for Cisco MDS 9700 Series Switches Using Cisco MDS 9700 40 Gbps 24-Port FCoE Module

No. of Fabric Cards

Front Panel FCoE Bandwidth/Slot

Maximum No. of Ports/Speed

3

660 Gbps

16 ports/40 Gbps

4

880 Gbps

20 ports/40 Gbps

5

1100 Gbps

24 ports/40 Gbps


Note

If the number of ports that you configure exceeds the maximum limit for the number of ports, it results in all ports operating in oversubscribed mode.

Fibre Channel Topology Combinations and Scaling

From Cisco MDS NX0-OS Release 6.2(7), the following configurable features that support scale enhancements are supported:

  • Fibre Channel Name Server (FCNS) bulk notification

  • Coalesce switch Registered State Change Notification (SW-RSCN)

For more information about the FCNS and SW-RSCN features, see the Cisco MDS 9000 Series NX-OS Fabric Configuration Guide and the Cisco MDS 9000 Family Command Reference documents.


Note
In multidimensional scale configurations, supervisors with 1-GB memory may experience the system manager hap-reset error.

The following table provides the configuration limits with and without the FCNS and SW-RSCN optimizations.

Table 4. Fibre Channel Topology Combination and Scaling

Scale Topology

FCNS Bulk Notification

Coalesce SW-RSCN

Configuration Limits

Scale Optimizations

Cisco MDS NX-OS Release 6.2(9) and later on Cisco MDS 9700 Series-only Fabric

On

On

MDS NX-OS Release 6.2(7) and later

Enabled

FLOGI per module: 1000

FLOGI node: 4000

FCNS: 20000

Zone: 16000

Zone members: 32000

Off

Off

MDS NX-OS Release 6.2(5) and earlier

Disabled (Disabled by default in Cisco MDS NX-OS Release 6.2(7))

FLOGI per module: 500

FLOGI node: 2500

FCNS: 10000

Zone: 8000

Zone members: 16000

Cisco MDS NX-OS Release 6.2(9) and later on MDS Mixed Fabric25 (Cisco MDS 9700 Series, 9500, 9250i, 9222i, and 9148)

On or off

On or off

FLOGI per module: 400

FLOGI node: 2000

FCNS: 10000, 13000 26
Zone: 8000, 10400 27
Zone members: 16000, 20800 28

Enabled or disabled

25

The scale enhancements introduced in Cisco MDS NX-OS 6.2(7) and later are available on Cisco MDS 9700 Series Switches (only networks with Cisco MDS NX-OS Release 6.2(7) and later, running on all the switches and the configurable optimizations): FCNS bulk notification and RSCN coalesce enabled. These enhancements are not available in mixed fabrics regardless of whether FCNS and RSCN are enabled. In Cisco MDS NX-OS Release 6.2(9) and later, FCNS bulk notification is enabled by default. (To disable FCNS bulk notification, use the fcns no-bulk-notify command.) Coalesce SW-RSCN is disabled by default.

26

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

27

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

28

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

Fibre Channel Scale Limitations

The various Fibre Channel scale limitations are described in this section.

Limitation 1

From Cisco MDS NX-OS Release 6.2(7), the fcns bulk-notify and rscn coalesce swrscn vsan commands are available on all Cisco MDS switches. We recommend that you use these commands only on Cisco MDS 9700 Series switches with 48-port 16-Gbps Fibre Channel switching module because the higher configuration limits are currently not supported on any other MDS platform.


Note
The FCNS bulk notification and Coalesce SW-RSCN features are supported only in NX-OS Release 6.2(7) and later releases. Enable the Coalesce SW-RSCN only if all the switches in a fabric are Cisco MDS switches that are running Cisco MDS NX-OS Release 6.2(7) or later.

Limitation 2

The maximum zone database size is increased from 2 to 3.8 MB in Cisco MDS Release 6.2(7) and later for all MDS 9700 Series switches to enable zone scale enhancements. The new limit of 16000 zones is supported only on the Cisco MDS 9700 only fabric. Fabrics with Cisco MDS 9500, 9200, or 9100 Series switches continue to have the 2-MB zone database limit, supporting only up to 8000 zones.

Limitation 3

In a three-node serial topology, traffic imbalance may occur if the number of port channels that are configured between the switches are the same. We recommend that you have a single port channel between two switches with any number of member ports. If you configure more than one port channel, ensure that the count of port channels between the switches varies.

Limitation 4

The maximum latency (round-trip time) and the packet-drops that are supported on FCIP links are 250 ms and 0.05 percent receptively.


Note

The maximum limit is the same regardless whether latency and packet drop conditions exist together or only one of the conditions exists.

Syslog and Warnings for Fibre Channel Network Scale

The following table lists the syslogs and warnings for the Fibre Channel network.

Table 5. Syslogs and Warnings for the Fibre Channel Network

Syslog and Warnings

Cisco MDS 9700 Series

Cisco MDS 9500, 9200, and 9100

Zone limit

16000 zones – Hard limit: Syslog warning that states no more zones can be configured: 

“Maximum configurable zone 
limit of 16,000 reached.
Creation of any more zones 
is not supported.”

8000 zones – Hard limit: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 8,000 reached. 
Creation of any more zones 
is not supported.”

10400 zones – Hard limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 10,400 reached. 
Creation of any more zones 
is not supported.”

FCNS limit

20000 FCNS entries – Hard limit: Syslog warning stating that no more name server entries are supported: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

10000 FCNS entries – Soft limit: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit 
of 10,000. Any more entries could 
potentially destabilize the fabric.”

13000 FCNS entries – Soft limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit of 13,000. 
Any more entries could potentially 
destabilize the fabric.”

20000 FCNS entries – Hard limit: Syslog warning: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

Switch-Level Configuration Limits for Fibre Channel over Ethernet

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level configuration limits for Fibre Channel over Ethernet (FCoE) on Cisco MDS.

Table 6. Switch-Level Configuration Limits for FCoE

Feature Name

MDS 9710 with 48-Port 10-Gigabit FCoE

FLOGIs per Port

256

FLOGIs per Module

1000

FLOGIs per Switch

4000

VSAN

80

VSAN-VLAN Mapping

80

Virtual Fibre Channel Port Channel and Member Ports

128 virtual fibre channel port channel and a maximum number of 16 members in one virtual fibre channel port channel

Fabric-Level Configuration Limits for Fibre Channel over IP

The following table lists the fabric-level configuration limits for Fibre Channel over IP (FCIP):

Table 7. Fabric-Level Configuration Limits for FCIP on Cisco MDS 9250i Multiservice Fabric Switch

Feature Name

Cisco MDS 9250i Multiservice Fabric Switch

Maximum latency (round-trip time) and packet drops supported on FCIP links

250-ms round trip and 0.05% packet drop
Table 8. Fabric-Level Configuration Limits for FCIP on Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Feature Name

Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Maximum latency (round-trip time) and packet drops supported on FCIP links

250-ms round trip and 0.05% packet drop

Fabric-Level Configuration Limits for Fibre Channel over Ethernet (FCoE) on MDS 9700 Series

The following table lists the fabric-level configuration limits for the Fibre Channel over Ethernet (FCoE) on MDS 9700 Series.

Table 9. Fabric-Level Configuration Limits for FCoE on MDS 9700 Series Switches

Feature Name

MDS 9700 Network with 48-Port 10-Gigabit Fibre Channel over Ethernet Module

Zones

16000

Zone members

32000

Zone sets

1000

Zone database size

3.8 MB 29

Number of FCNS entries in a network

20000

Device alias

12000

29

An error message appears in your console if the zone database size exceeds 3.8 MB.


Note

  • A warning appears in your console if the number of zones exceeds 8000. However, further configurations are not blocked.

  • An error message appears in your console if the total number of the unique zone members exceed 16000.

  • From Cisco MDS NX-OS Release 6.2(7), a zone's database size is increased from 2 to 3.8 MB, to provide zone scale enhancements. However, if there are any switches in the fabric that are running Cisco MDS NXOS Release 6.2(5) or earlier, the previous 2-MB zone database limit prevails and the new zone scale enhancements are not available. We recommend that the 2-MB zone database limit does not exceed, unless all the switches in the fabric run Release 6.2(7) or later.

  • An error message appears in your console if the number of FCNS entries exceeds 10000.

Example: Checking the Size of the Zone Database

The following example shows how to check the size of the zone database in VSAN 310: 


(config)# show zone status vsan 310 | Inc "Db Size" P 1

Full Zoning Database :
    Db Size: 1040524 Bytes
--
Active Zoning Database :
    Db Size: 962156 Bytesc

Now, Add 962156 with 1040524 = 2002680

2002680 / (1024 * 1024) = 1.9 MB.


Note

This database size does not include the pending changes in a session.

Cisco MDS 9000 I/O Accelerator Configuration Limits

The following table lists the Cisco MDS 9000 I/O Accelerator configurations and the corresponding limits.

Table 10. Cisco I/O Accelerator Configuration Limits

Parameter

MSM-18/4 or SSN-16 Module on MDS 9222i and MDS 9500 Modular Chassis and MDS 9250i Fabric Switch, 24/10-Port SAN Extension Module on MDS 9710 Chassis

Number of switches in a cluster

4

Number of clusters per switch

16

Number of switches in a SAN fabric for FC-Redirect

34

Number of hosts per target

128

Number of concurrent flows per Cisco MDS 9000 I/O Accelerator service engine

128

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (hard limit)

128 - Release 4.2(1) on MDS 9222i and MDS 9500

512 - Release 4.2(7) or later on MDS 9222i and MDS 9500

512 - Release 6.2(5) or later on MDS 9250i

512 - Release 8.2(1) or later on MDS 9710

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (soft limit)

64 - Release 4.2(1) on MDS 9222i/MDS 9500

256 - Release 4.2(7) or later on MDS 9222i/MDS 9500

256 - Release 6.2(5) or later on MDS 9250i

Note 

If initiators or targets participating in Cisco MDS 9000 I/O Accelerator are present on MDS 9250i switches, the limit is 203 for tape and 160 for disk.

256 - Release 8.2(1) or later on MDS 9710

Number of flows in a cluster

1024 - Release 4.2(7d)

1248 - Release 5.2(6b) or later

Fibre Channel Flow Configuration Limits

The following table lists the Fibre Channel flow configurations limits.

Table 11. Fibre Channel Flow Configuration Limits

Cisco MDS Device

Aggregate Flow and Flow Statistics Limit

Flow Statements per Module

Generation 1 Modules

1000

1024

Generation 2 Modules

2000

2048

Generation 3 Modules

512

512

Generation 4 Modules

512

512

Generation 5 Modules

512

512