Cisco MDS NX-OS Configuration Limits

This document discusses the configuration and scalability limits for various SAN switching parameters in Cisco MDS NX-OS Release 8.x, and contains the following topics:

Switch-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level Fibre Channel configuration limits for Cisco MDS 9000 Series switches:

Table 1. Switch-Level Fibre Channel Configuration Limits

Feature Name

Parameter

MDS 9700 Series

MDS 9250i

MDS 9148S

MDS 9396S

MDS 9132T

MDS 9148T

MDS 9396T

Logins (FLOGI or FDISC)

FLOGIs or FDISCs per port

256 1

256 1

256 1

256 1

256 1

256 1

256 1
FLOGIs or FDISC per switch

40002

80004

400

1000

1000

2000

2000

2000

FLOGIs or FDISCs per module

10003
20004

—

—

—

—

—

—

N-Port Virtualization (NPV)

NPV switches per NPIV core switch

105

8

8

8

8

8

8

Inter-Switch Links (ISLs)

ISL instances per switch

Up to 200 ISLs, each ISL with 16 VSANs, for a total number of 3200 port VSAN instances

Up to 40 ISLs, each with 29 VSANs, for a total number of 1160 port VSAN instances

Up to 48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

Up to 96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

Up to 32 ISLs, each with 29 VSANs, for a total number of 928 port VSAN instances

Up to 48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

Up to 96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

Port channels

Port channels and member ports in port channels

256 port channels with a maximum of 16 members in each port channel

40 port channels with a maximum of 16 members in each port channel and 8 Ethernet port channels with a maximum of 8 members in each Ethernet port channels

48 port channels with a maximum of 16 members in each port channel

96 port channels with a maximum of 16 members in each port channel

32 port channels with a maximum of 16 members in each port channel

48 port channels with a maximum of 16 members in each port channel

96 port channels with a maximum of 16 members in each port channel

SSH

Maximum simultaneous SSH sessions per switch

16

16

16

16

16

16

16

SAN Analytics and SAN Telemetry Streaming

Maximum active Initiator-

Target-LUNs (ITLs) or Initiator-

Target-Namspace ID (ITNs) per sampling interval per switch

8.3(2) and later releases: 40,000

5

8.3(1) and earlier releases: 20,000

—

—

—

20,000

20,000

20,000

Maximum active ITLs or ITNs per sampling interval per module

DS-X9648-1536K9: 20,000

—

—

—

20,000

20,000

20,000

Maximum number of telemetry receivers

2

—

—

—

2

2

2
1

When FLOGIs or FDISCs reach 512, the FLOGIs or FDISCs are rejected with the syslog "Message FLOGI-1-MSG_FDISC_REJECT_TOO_MANY_LOGINS: [VSAN [dec], Interface [chars]] FDISC rejected - this interface has reached the max num of N-Ports (512)".

2

The number of FCIDs supported per switch and per fabric is the same as the FLOGI limit.

3

Applicable to Cisco MDS 9706 and 9710 switches only.

4

Applicable to Cisco MDS 9718 switches only, starting from Cisco MDS NX-OS Release 8.1(1).

5 When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.

Note

  • A system message appears in the user’s session when the maximum supported limit of port channels and member ports is exceeded.

  • A system message appears on the user’s session when the maximum supported number of SSH sessions is exceeded.

Fabric-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches

The following table lists the fibre channel configuration limits for the Cisco MDS 9000 fibre channel fabric network.

Table 2. Fabric-Level Fibre Channel Configuration Limits

Feature Name

Parameter

MDS 9700-only Network (without MDS 9500, 9200, and 9100)

MDS Mixed Fabric Network

(MDS 9700, 9500, 9200, 9148, 9148S, and 9396S)

VSANs

Number of VSANs per physical fabric

80

80

Logins and aliases

Number of FCNS entries in fabric

200006
100007
130008

Number of device alias entries in fabric

20000

8000

Domains and hops

Number of domains per physical fabric

80

80

Number of switch hops from server to storage

7

7

Zones/Smart Zones

Note 
The preferred number of members per zone is 2, and the maximum recommended limit is 50.

Zone sets

1000

1000

Zones

16000 9
800010
10400 11

Zone members

32000 12

1600013
2080014

Zone DB size

3.8 MB 15

2 MB

3.8 MB16

IVR

IVR zone sets

32

32

IVR zones

2000

2000

IVR zone member

4000

4000

IVR service groups

16

16

CFS

CFS peers

80

80

CFS static peers over IP

100

100

6

An error message appears in the user’s session if the number of FCNS entries exceeds 20000.

7

For platforms other than Cisco MDS 9700, a warning appears in the user’s session if the number of FCNS entries exceeds 10000.

8

This increase in scale is applicable only to Cisco MDS 9500 Series Supervisor-2A Module.

9

An error message and syslog appear in a user’s session if the total number of zones exceeds 16000.

10

For platforms other than Cisco MDS 9700, a warning appears in a user’s session if the number of zones exceeds 8000. However, further configurations are not blocked.

11

This increase in scale is applicable only to Cisco MDS 9500 Series Supervisor-2A Module.

12 An error message and syslog appear in a user’s session if the total number of the unique zone members exceeds 32000.
13

For platforms other than Cisco MDS 9700, a warning appears in a user’s session if the number of unique zone member exceeds 16000.

14

This increase in scale is applicable only to Cisco MDS 9500 Series Supervisor-2A Module.

15

An error message appears in a user’s console if the zone database size exceeds 3.8 MB.

16

This increase in scale is applicable only to Cisco MDS 9500 Series Supervisor-2A Module.


Note

  • In a mixed fabric comprising Cisco MDS 9700 and 9500 director class switches and Cisco MDS 9148 and 9148S fabric switches, the fabric can have all the switches in the first 32 VSANs. The remaining VSANs include only those switches that support more than 32 VSANs.

  • From Cisco MDS NX-OS Release 6.2(11), the number of domains per physical fabric is increased to 80.

  • A system message appears in a user's session when the supported configuration limit of CFS peers is exceeded.

  • From Cisco MDS NX-OS Release 6.2(7), a zone's database size is increased from 2 MB to 3.8 MB, to provide zone scale enhancements. However, if there are any switches in the fabric running Cisco MDS NX-OS 6.2(5) or earlier releases, the previous 2-MB zone database limit prevails and the new zone scale enhancements are not available. We recommend that the 2-MB zone database limit not be exceeded, unless all the switches in the fabric run on Cisco MDS NX-OS Release 6.2(7) or later.

  • 2000 IVR zones are available across all VSANs. However, a single VSAN must not have more than 1000 IVR zones.

Example: Calculating a Zone's Database size

The following example shows how to calculate a zone's database size: 


(config)# show zone status vsan 310 | Inc "Db Size" P 1

Full Zoning Database :
    Db Size: 1040524 Bytes
--
Active Zoning Database :
    Db Size: 962156 Bytesc 
Now, Add 962156 with 1040524 = 2002680
       2002680 / (1024 * 1024) = 1.9 MB.

Note

This database size does not include the pending changes in a session.

Line Rate Limitation for Cisco MDS 9700 Series Switches Using Cisco MDS 9700 40-Gbps 24-Port FCoE Module

For a fabric card to operate at full-line rate on Cisco MDS 9706, 9710, and 9718 switches, the port and bandwidth should have specific configurations. The following table lists the port-level configuration limits and bandwidth limitations for the Cisco MDS 9700 Series Switches.

Table 3. Line Rate Limitation for Cisco MDS 9700 Series Switches Using Cisco MDS 9700 40-Gbps 24-Port FCoE Module

No. of Fabric Cards

Front Panel FCoE Bandwidth/Slot

Maximum No. of Ports/Speed

3

660 Gbps

16 ports/40 Gbps

4

880 Gbps

20 ports/40 Gbps

5

1100 Gbps

24 ports/40 Gbps


Note

If the number of ports configured exceed the maximum limit for number of ports then it will result in all ports operating in oversubscribed mode.

Fibre Channel Topology Combinations and Scaling

From Cisco MDS NX0-OS Release 6.2(7), the following configurable features that support scale enhancements are supported:

  • Fibre Channel Name Server (FCNS) bulk notification

  • Coalesce switch Registered State Change Notification (SW-RSCN)

For more information about the FCNS and SW-RSCN features, see the Cisco MDS 9000 Series NX-OS Fabric Configuration Guide and the Cisco MDS 9000 Family Command Reference documents.


Note
In multidimensional scale configurations, supervisors with 1-GB memory may experience the system manager hap-reset error.

The following table provides the configuration limits with and without the FCNS and SW-RSCN optimizations.

Table 4. Fibre Channel Topology Combination and Scaling

Scale Topology

FCNS Bulk Notification

Coalesce SW-RSCN

Configuration Limits

Scale Optimizations

Cisco MDS NX-OS Release 6.2(9) and later on Cisco MDS 9700 Series-only Fabric

ON

ON

MDS NX-OS Release 6.2(7) and later

Enabled

FLOGI per module: 1000

FLOGI node: 4000

FCNS: 20000

Zone:16000

Zone members: 32000

OFF

OFF

MDS NX-OS Release 6.2(5) and earlier

Disabled (Disabled by default in Cisco MDS NX-OS Release 6.2(7))

FLOGI per module: 500

FLOGI node: 2500

FCNS: 10000

Zone: 8000

Zone members: 16000

Cisco MDS NX-OS Release 6.2(9) and later on MDS Mixed Fabric17 (Cisco MDS 9700 Series, 9500, 9250i, 9222i, and 9148)

ON/OFF

ON/OFF

FLOGI per module: 400

FLOGI node: 2000

FCNS: 10000, 13000 18
Zone: 8000, 10400 19
Zone members: 16000, 20800 20

Enabled/Disabled

17

The scale enhancements introduced in Cisco MDS NX-OS 6.2(7) and later are available on Cisco MDS 9700 Series Switches (only networks with Cisco MDS NX-OS Release 6.2(7) and later, running on all the switches and the configurable optimizations): FCNS bulk notification and RSCN coalesce enabled. These enhancements are not available in mixed fabrics regardless of whether FCNS and RSCN are enabled. In Cisco MDS NX-OS Release 6.2(9) and later, FCNS bulk notification is enabled by default. (To disable FCNS bulk notification, use the fcns no-bulk-notify command.) Coalesce SW-RSCN is disabled by default.

18

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

19

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

20

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

Fibre Channel Scale Limitations

The various fibre channel scale limitations are described in this section.

Limitation 1

From Cisco MDS NX-OS Release 6.2(7), the fcns bulk-notify and rscn coalesce swrscn vsan commands are available on all Cisco MDS switches. We recommend that you use these commands only on the Cisco MDS 9700 Series Switches having with the 48-port 16-Gbps fibre channel switching module because the higher configuration limits are currently not supported on any other MDS platform.


Note
The FCNS bulk notification and Coalesce SW-RSCN features are supported only in NX-OS Release 6.2(7) and later releases. The Coalesce SW-RSCN should be enabled only if all the switches in the fabric are Cisco MDS switches that are running MDS NX-OS Release 6.2(7) or later.

Limitation 2

The maximum zone database size has been increased from 2 MB to 3.8 MB in Cisco MDS Release 6.2(7) and later for all MDS 9700 Series switches to enable zone scale enhancements. The new limit of 16000 zones is supported only on the Cisco MDS 9700-only fabric. Fabrics with Cisco MDS 9500, 9200, or 9100 Series Switches continue to have the 2-MB zone database limit, supporting only up to 8000 zones.

Limitation 3

In a three-node serial topology, traffic imbalance may occur if the number of port channels configured between the switches are the same. We recommended that you have a single port channel between two switches with any number of member ports. If more than one port channel is configured, ensure that the count of port channels between the switches varies.

Limitation 4

The maximum latency (round-trip time) and the packet-drops supported on FCIP links are 250 ms and 0.05 percent receptively.


Note

The maximum limit is the same regardless whether latency and packet drop conditions exist together or only one of them exists.

Syslog and Warnings for Fibre Channel Network Scale

The following table lists the syslogs and warnings for the Fibre Channel network.

Table 5. Syslogs and Warnings for the Fibre Channel Network

Syslog and Warnings

Cisco MDS 9700 Series

Cisco MDS 9500, 9200, and 9100

Zone limit

16000 zones – Hard limit: Syslog warning that states no more zones can be configured: 

“Maximum configurable zone 
limit of 16,000 reached.
Creation of any more zones 
is not supported.”

8000 zones – Hard limit: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 8,000 reached. 
Creation of any more zones 
is not supported.”

10400 zones – Hard limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 10,400 reached. 
Creation of any more zones 
is not supported.”

FCNS limit

20000 FCNS entries – Hard limit: Syslog warning that states no more name server entries are supported: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

10000 FCNS entries – Soft limit: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit 
of 10,000. Any more entries could 
potentially destabilize the fabric.”

13000 FCNS entries – Soft limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit of 13,000. 
Any more entries could potentially 
destabilize the fabric.”

20000 FCNS entries – Hard limit: Syslog warning: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

Switch-Level Configuration Limits for Fibre Channel over Ethernet

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level configuration limits for Fibre Channel over Ethernet (FCoE) on Cisco MDS.

Table 6. Switch-Level Configuration Limits for FCoE

Feature Name

MDS 9710 with 48-Port 10-Gigabit FCoE

FLOGIs per Port

256

FLOGIs per Module

1000

FLOGIs per Switch

4000

VSAN

80

VSAN-VLAN Mapping

80

vFC port channel and Member Ports

128 vFC port channel and a maximum number of 16 members in one vFC port channel

Fabric-Level Configuration Limits for Fibre Channel over IP

The following table lists the fabric-level configuration limits for fibre channel over IP (FCIP):

Table 7. Fabric-Level Configuration Limits for FCIP on Cisco MDS 9250i Multiservice Fabric Switch

Feature Name

Cisco MDS 9250i Multiservice Fabric Switch

Maximum latency (round-trip time) and packet drops supported on FCIP links

250 ms round trip and 0.05% packet drop
Table 8. Fabric-Level Configuration Limits for FCIP on Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Feature Name

Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Maximum latency (round-trip time) and packet drops supported on FCIP links

250 ms round trip and 0.05% packet drop

Fabric-Level Configuration Limits for Fibre Channel over Ethernet (FCoE) on MDS 9700 Series

The following table lists the fabric-level configuration limits for the Fibre Channel over Ethernet (FCoE) on MDS 9700 Series.

Table 9. Fabric-Level Configuration Limits for FCoE on MDS 9700 Series Switches

Feature Name

MDS 9700 Network with 48-Port 10-Gigabit Fibre Channel over Ethernet Module

Zones

16000

Zone members

32000

Zone sets

1000

Zone DB size

3.8 MB 21

Number of FCNS entries in network

20000

Device alias

12000

21

An error message appears in a user’s console if the zone database size exceeds 3.8 MB.


Note

  • A warning appears in a user’s console if the number of zones exceeds 8000. However, further configurations are not blocked.

  • An error message appears in a user’s console if the total number of the unique zone members exceed 16000.

  • From Cisco MDS NX-OS Release 6.2(7), a zone's database size is increased from 2 MB to 3.8 MB, to provide zone scale enhancements. However, if there are any switches in the fabric that are running on Cisco MDS NXOS Release 6.2(5) or earlier, the previous 2-MB zone database limit will prevail and the new zone scale enhancements will not be available. We recommend that the 2-MB zone database limit not be exceeded unless all the switches in the fabric run Release 6.2(7) or later.

  • An error message appears in a user’s console if the number of FCNS entries exceeds 10000.

Example: Calculating a Zone's Database size

The following example shows how to calculate a zone's database size: 


(config)# show zone status vsan 310 | Inc "Db Size" P 1

Full Zoning Database :
    Db Size: 1040524 Bytes
--
Active Zoning Database :
    Db Size: 962156 Bytesc 
Now, Add 962156 with 1040524 = 2002680
       2002680 / (1024 * 1024) = 1.9 MB.

Note

This database size does not include the pending changes in a session.

Cisco MDS 9000 I/O Accelerator Configuration Limits

The following table lists the Cisco MDS 9000 I/O Accelerator configurations and the corresponding limits.

Table 10. Cisco I/O Accelerator Configuration Limits

Parameter

MSM-18/4 or SSN-16 Module on MDS 9222i and MDS 9500 Modular Chassis and MDS 9250i Fabric Switch, 24/10-Port SAN Extension Module on MDS 9710 Chassis

Number of switches in a cluster

4

Number of clusters per switch

16

Number of switches in a SAN fabric for FC-Redirect

34

Number of hosts per target

128

Number of concurrent flows per Cisco MDS 9000 I/O Accelerator service engine

128

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (hard limit)

128 - Release 4.2(1) on MDS 9222i and MDS 9500

512 - Release 4.2(7) or later on MDS 9222i and MDS 9500

512 - Release 6.2(5) or later on MDS 9250i

512 - Release 8.2(1) or later on MDS 9710

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (soft limit)

64 - Release 4.2(1) on MDS 9222i/MDS 9500

256 - Release 4.2(7) or later on MDS 9222i/MDS 9500

256 - Release 6.2(5) or later on MDS 9250i

Note 

If initiators or targets participating in Cisco MDS 9000 I/O Accelerator are present on MDS 9250i switches, the limit is 203 for tape and 160 for disk.

256 - Release 8.2(1) or later on MDS 9710

Number of flows in a cluster

1024 - Release 4.2(7d)

1248 - Release 5.2(6b) or later

Fibre Channel Flow Configuration Limits

The following table lists the Fibre Channel flow configurations limits.

Table 11. Fibre Channel Flow Configuration Limits

Cisco MDS Device

Aggregate Flow and Flow Statistics Limit

Flow Statements per Module

Generation 1 Modules

1000

1024

Generation 2 Modules

2000

2048

Generation 3 Modules

512

512

Generation 4 Modules

512

512

Generation 5 Modules

512

512