Cisco MDS 9000 Family Fabric Manager Configuration Guide
Configuring Generation 2 Switching Modules
Downloads: This chapterpdf (PDF - 893.0KB) The complete bookPDF (PDF - 48.18MB) | Feedback

Configuring Generation 2 and Generation 3 Switching Modules

Table Of Contents

Configuring Generation 2 and Generation 3 Switching Modules

About Generations of Modules and Switches

Port Groups and Port Rate Modes

Port Groups

Port Rate Modes

Dedicated Rate Mode

Shared Rate Mode

Dedicated Rate Mode Configurations for the 8-Gbps Modules

Reserving Bandwidth Quickly for the 8-Gbps Modules

Dynamic Bandwidth Management

Out-of-Service Interfaces

Buffer Credit Allocation

Buffer Pools

BB_Credit Buffers for Switching Modules

48-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

24-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

4/44-Port 8-Gbps Host-Optimized Fibre Channel Module BB_Credit Buffers

48-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

24-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

18-Port Fibre Channel/4-Port Gigabit Ethernet Multiservice Module BB_Credit Buffers

12-Port 4-Gbps Switching Module BB_Credit Buffers

4-Port 10-Gbps Switching Module BB_Credit Buffers

BB_Credit Buffers for Fabric Switches

Cisco MDS 9134 Fabric Switch BB_Credit Buffers

Cisco MDS 9124 Fabric Switch BB_Credit Buffers

Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers

Extended BB_Credits

Combining Generation 1, Generation 2, and Generation 3 Modules

Port Indexes

PortChannels

Configuring Module Interface Shared Resources

Configuration Guidelines for 48-Port, 24-Port, and 4/44-Port 8-Gbps Fibre Channel Switching Modules

Migrating from Shared Mode to Dedicated Mode

Migrating from Dedicated Mode to Shared Mode

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

Migrating from Shared Mode to Dedicated Mode

Migrating from Dedicated Mode to Shared Mode

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

Configuring Port Speed

Configuring Rate Mode

Configuring Oversubscription Ratio Restrictions

Disabling Restrictions on Oversubscription Ratios

Enabling Restrictions on Oversubscription Ratios

Configuring Bandwidth Fairness

Enabling Bandwidth Fairness

Disabling Bandwidth Fairness

Upgrade or Downgrade Scenario

Taking Interfaces Out of Service

Releasing Shared Resources in a Port Group

Displaying SFP Diagnostic Information

Default Settings


Configuring Generation 2 and Generation 3 Switching Modules


Cisco MDS 9000 Family hardware modules and switches are categorized into generations based on the time of introduction, capabilities, features, and compatibilities:

Generation 1—Modules and switches with a maximum port speed of 2 Gbps.

Generation 2—Modules and switches with a maximum port speed of 4 Gbps.

Generation 3—Modules and switches with a maximum port speed of 8 Gbps.

This chapter describes how to configure these modules and switches, including the following sections:

About Generations of Modules and Switches

Port Groups and Port Rate Modes

Buffer Credit Allocation

Combining Generation 1, Generation 2, and Generation 3 Modules

Configuring Module Interface Shared Resources

Displaying SFP Diagnostic Information

Default Settings

About Generations of Modules and Switches

In addition to supporting Generation 2 modules, the Cisco MDS 9500 Series Switches and the Cisco MDS 9222i Switch support another set of modules called Generation 3 modules. Similar to Generation 2, each Generation 3 module can have one or more ports in port groups that share common resources such as bandwidth and buffer credits.

Generation 3 Fibre Channel modules are supported on the Cisco MDS 9506 and 9509 switches with Supervisor-2 modules, and on the MDS 9513 Director with Fabric 2 modules. The MDS 9222i switch supports the 4/44-port Host-Optimized Fibre Channel Switching module. MDS NX-OS Release 4.1(1) is required to support the Generation 3 modules.

Table 22-1 identifies the modules supported by the Cisco MDS 9500 Series switches and Cisco MDS 9216A and Cisco MDS 9216i switches, as well as the Fabric switches.

Table 22-1 Fibre Channel Modules and Fabric Switches 

Part Number
Product Name/Description
Generation 3 Modules

DS-X9248-96K9

48-port 8-Gbps Fibre Channel switching module

DS-X9224-96K9

24-port 8-Gbps Fibre Channel switching module

DS-X9248-48K9

4/44-port 8-Gbps Host-Optimized Fibre Channel switching module

DS-13SLT-FAB2

Fabric 2 module that enables the 48-port 8-Gbps Fibre Channel switching module to use the full 96-Gbps backplane bandwidth with any-to-any connectivity without reducing bandwidth utilization when one fabric card goes down on the MDS 9513 Director.

Generation 2 Modules

DS-X9148

48-port 4-Gbps Fibre Channel switching module

DS-X9124

24-port 4-Gbps Fibre Channel switching module

DS-X9304-18K9

18-port 4-Gbps Fibre Channel switching module with 4-Gigabit Ethernet ports

DS-X9112

12-port 4-Gbps Fibre Channel switching module

DS-X9704

4-port 10-Gbps Fibre Channel switching module

DS-X9530-SF2-K9

Supervisor-2 module for Cisco MDS 9500 Series switches.

Generation 2 Fabric Switches

DS-C9134-K9

Cisco MDS 9134 Fabric switch

32-port 4-Gbps Fabric switch with 2 additional 10-Gbps ports

DS-C9124

Cisco MDS 9124 Fabric switch

24-port 4-Gbps Fabric switch

DS-C9222i-K9

Cisco MDS 9222i Multiservice Modular switch

18-port 4-Gbps switch with 4-Gigabit Ethernet IP storage services ports, and a modular expansion slot to host Cisco MDS 9000 Family Switching and Services Modules



Note Generation 2 Fibre Channel switching modules are not supported on the Cisco MDS 9216 switch; however, they are supported by both the Supervisor-1 module and the Supervisor-2 module.


For detailed information about the installation and specifications for these modules and switches, refer to the hardware installation guide for your switch.

Port Groups and Port Rate Modes

This section includes the following topics:

Port Groups

Port Rate Modes

Dedicated Rate Mode Configurations for the 8-Gbps Modules

Reserving Bandwidth Quickly for the 8-Gbps Modules

Dynamic Bandwidth Management

Out-of-Service Interfaces

Port Groups

Each module or switch can have one or more ports in port groups that share common resources such as bandwidth and buffer credits. Port groups are defined by the hardware consisting of sequential ports. For example, ports 1 through 12, ports 13 through 24, ports 25 through 36, and ports 37 through 48 are the port groups on the 48-port 4-Gbps Fibre Channel switching modules.

Table 22-2 shows the port groups for the Generation 2 and Generation 3 Fibre Channel modules, and Generation 2 Fabric switches.

Table 22-2 Bandwidth and Port Groups for the Fibre Channel Modules and Fabric Switches 

Part Number
Product Name/ Description
Number of Ports Per Port Group
Bandwidth Per Port Group (Gbps)
Maximum Bandwidth Per Port (Gbps)
Generation 3 Modules

DS-X9248-96K9

48-port 8-Gbps Fibre Channel switching module

6

12.8

8 Gbps

DS-X9224-96K9

24-port 8-Gbps Fibre Channel switching module

3

12.8

8 Gbps

DS-X9248-48K9

4/44-port 8-Gbps Host-Optimized Fibre Channel switching module

12

12.8

8/4 Gbps1

Generation 2 Modules

DS-X9148

48-port 4-Gbps Fibre Channel switching module

12

12.8

4 Gbps

DS-X9124

24-port 4-Gbps Fibre Channel switching module

6

12.8

4 Gbps

DS-X9304-18K9

(MSM-18/4 Multiservice module)

18-port 4-Gbps Fibre Channel switching module with 4-Gigabit Ethernet ports

6

12.8

4 Gbps

DS-X9112

12-port 4-Gbps Fibre Channel switching module

3

12.8

4 Gbps

DS-X9704

4-port 10-Gbps Fibre Channel switching module

1

10

10 Gbps

Generation 2 Fabric Switches

DS-C9134-K9

(Cisco MDS 9134 Fabric switch)

32-port 4-Gbps Fabric switch

4

16

4 Gbps

2-port 10-Gbps Fabric switch

1

10

10 Gbps

DS-C9124K9 (Cisco MDS 9124 Fabric switch)

24-port 4-Gbps Fabric switch

4

16

4 Gbps

DS-C9222i-K9

(Cisco MDS 9222i Multiservice Modular switch)

18-port 4-Gbps

6

12.8

4 Gbps

1 A maximum of 4 ports (one per port group) in a 4/44-port 8-Gbps switching module can operate at 8 Gbps bandwidth in dedicated or shared mode. All the other ports can operate at a maximum of 4 Gbps in shared mode or dedicated mode.


Port Rate Modes

In Generation 2 and Generation 3 modules, you can configure the port rate modes. The port rate mode configuration is used to determine the bandwidth allocation for ports in a port group. Two port rate modes are supported:

Dedicated Rate Mode—A port is allocated required fabric bandwidth to sustain line traffic at the maximum operating speed configured on the port. For more information, see the "Dedicated Rate Mode" section

Shared Rate Mode—Multiple ports in a port group share data paths to the switch fabric and share bandwidth. For more information, see the "Shared Rate Mode" section.


Note In Generation 1 modules, you cannot configure the port rate modes. The mode is determined implicitly based on the port mode and line card type.



Note Port rate modes are not supported on the Cisco Fabric Switch for HP c-Class BladeSystem, and the Cisco Fabric Switch for IBM BladeCenter.


Table 22-3 shows the modules that support dedicated, shared, and the default rate modes.

Table 22-3 Port Rate Mode Support on Generation 2 and Generation 3 Modules and Switches 

Part Number
Product Name/
Description
Supports Dedicated Rate Mode
Supports Shared Rate Mode
Default Speed Mode and Rate Mode on All Ports
Generation 3 Modules

DS-X9248-96K9

48-Port 8-Gbps Fibre Channel switching module

Yes

Yes1

Auto, Shared

DS-X9224-96K9

24-Port 8-Gbps Fibre Channel switching module

Yes

Yes1

Auto, Shared

DS-X9248-48K9

4/44-Port 8-Gbps Host-Optimized Fibre Channel switching module

Yes

Yes1

Auto Max 4 Gbps, Shared

Generation 2 Modules

DS-X9148

48-port 4-Gbps Fibre Channel switching module2

Yes

Yes

Auto, Shared

DS-X9124

24-port 4-Gbps Fibre Channel switching module

Yes

Yes

Auto, Shared

DS-X9304-18K9

(MSM-18/4 Multiservice module)

18-port 4-Gbps Fibre Channel switching module with 4-Gigabit Ethernet ports

Yes

Yes

Auto, Shared

DS-X9112

12-port 4-Gbps Fibre Channel switching module

Yes

No

Auto, Dedicated

DS-X9704

4-port 10-Gbps Fibre Channel switching module

Yes

No

Auto, Dedicated

Generation 2 Switches

DS-C9134-K9

(Cisco MDS 9134 Fabric switch)

32-port 4-Gbps Fabric switch

Yes

Yes

Auto, Shared

2-port 10-Gbps Fabric switch

Yes

No

Auto, Dedicated

DS-C9124

(Cisco MDS 9124 Fabric switch)

24-port 4-Gbps Fabric switch3

Yes

No

Auto, Dedicated

DS-C9222i-K9

(Cisco MDS 9222i Multiservice Modular switch)

18-port 4-Gbps Fibre Channel switch with 4-Gigabit Ethernet IP storage services ports, and a modular expansion slot to host Cisco MDS 9000 Family Switching and Services Modules

Yes

Yes

Auto, Shared

1 Shared rate mode is supported on Fx ports only and no ISLs.

2 All ports in a 48-port 4-Gbps switching module can operate in dedicated rate mode with a 1-Gbps operating speed. However, if you configure one or more ports to operate in 2-Gbps or 4-Gbps dedicated rate mode, some of the other ports in the port group would have to operate in shared mode.

3 All ports in a 24-port 4-Gbps switching module can operate in dedicated rate mode with a 2-Gbps operating speed. However, if you configure one or more ports to operate in 4-Gbps dedicated rate mode, some of the other ports in the port group would have to operate in shared mode


Dedicated Rate Mode

When port rate mode is configured as dedicated, a port is allocated required fabric bandwidth and related resources to sustain line rate traffic at the maximum operating speed configured for the port. In this mode, ports do not use local buffering and all receive buffers are allocated from a global buffer pool (see the "Buffer Pools" section).

Table 22-4 shows the bandwidth provided by the various port speed configurations on the 8-Gbps Fibre Channel switching modules.

Table 22-4 Bandwidth Reserved for the Port Speeds on Generation 3 Switching Modules 

Configured Speed
Reserved Bandwidth

Auto

8 Gbps

8-Gbps

Auto with 4-Gbps maximum

4 Gbps

4-Gbps

Auto with 2-Gbps maximum

2 Gbps

2-Gbps

1-Gbps

1 Gbps


Table 22-5 show the amount of bandwidth reserved for a configured port speed on 4-Gbps switching modules.

Table 22-5 Bandwidth Reserved for the Port Speeds on Generation 2 Switching Modules 

Configured Speed
Reserved Bandwidth

Auto

4 Gbps

4-Gbps

Auto with 2-Gbps maximum

2 Gbps

2-Gbps

1-Gbps

1 Gbps



Note The 4-Port 10-Gbps Fibre Channel module ports in auto mode only support auto speed mode at 10 Gbps.


Shared Rate Mode

When port rate mode is configured as shared, multiple ports within a port group share data paths to the switch fabric so that fabric bandwidth and related resources are shared. Often, the available bandwidth to the switch fabric may be less than the negotiated operating speed of a port. Ports in this mode use local buffering for the BB_credit buffers.

All ports in 4-Gbps Fibre Channel switching modules where bandwidth is shared support 1-Gbps, 2-Gbps, or 4-Gbps traffic. However, it is possible to configure one or more ports in a port group to operate in dedicated rate mode with 1-Gbps, 2-Gbps or 4-Gbps operating speed.

All ports in the 48-Port and 24-Port 8-Gbps Fibre Channel switching modules where bandwidth is shared support 1-Gbps, 2-Gbps, 4-Gbps, or 8-Gbps traffic.

In the 4/44-Port 8-Gbps Host-Optimized Fibre Channel switching module, all the ports where bandwidth is shared support 1-Gbps, 2-Gbps, 4-Gbps in a maximum of 44 ports, or 8 Gbps in a maximum of 4 ports.

Dedicated Rate Mode Configurations for the 8-Gbps Modules

Table 22-2 shows the maximum possible dedicated rate mode configuration scenarios for the Generation 3 Fibre Channel modules.

Table 22-6 Dedicated Rate Mode Bandwidth Reservation for Generation 3 Fibre Channel Modules 

Part Number
Product Name/ Description
Dedicated Bandwidth per Port
Maximum Allowed Ports that can come up
Ports in Shared Mode

DS-X9248-96K9

48-port 8-Gbps Fibre Channel switching module

8 Gbps

8 Ports

All the remaining ports are 8 Gbps shared.

4 Gbps

24 Ports

2 Gbps

48 Ports

DS-X9224-96K9

24-port 8-Gbps Fibre Channel switching module

8 Gbps

8 Ports

All the remaining ports are 8 Gbps shared.

4 Gbps

24 Ports

DS-X9248-48K9

4/44-port 8-Gbps Host-Optimized Fibre Channel switching module

8 Gbps

4 Ports

All the remaining ports are 4 Gbps shared (8 Gbps of bandwidth can be provided only to one port per port group in Dedicated or Shared rate mode).

4 Gbps

12 Ports

2 Gbps

24 Ports

1 Gbps

48 Ports


Reserving Bandwidth Quickly for the 8-Gbps Modules

To quickly reserve bandwidth for all the ports in the port groups on the Generation 3 Fibre Channel modules using the Device Manager, follow these steps:


Step 1 On the Device Manager window, right-click the 8-Gbps Fibre Channel module.

Figure 22-1 Device Manager - 8 Gbps Module - Pop-Up Menu

Step 2 From the pop up menu, select Bandwidth Reservation Config...

Step 3 In the Bandwidth Reservation Configuration dialog box that is displayed, choose a bandwidth reservation scheme. (Figure 22-2).

Figure 22-2 RateMode Configuration Dialog Box

Table 22-7 describes the default RateMode configuration schemes available in the Bandwidth Reservation Configuration dialog box for the 8-Gbps modules.

Table 22-7 RateMode Configuration Schemes

Module
Available RateMode Config Macros

DS-X9248-96K9

48-Port 8-Gbps Fibre Channel module

Dedicated 4 Gbps on the first port of each group and the remaining ports 8 Gbps shared

Dedicated 8 Gbps on the first port of each group and the remaining ports 8 Gbps shared

Shared 8 Gbps on all ports (initial & default settings)

DS-X9224-96K9

24-Port 8-Gbps Fibre Channel module

Dedicated 8 Gbps on the first port of each group and the remaining ports 8G shared

Shared Auto1 on all ports (initial & default settings)

DS-X9248-48K9

4/44-Port 8-Gbps Host-Optimized Fibre Channel module

Dedicated 2 Gbps on the first port of each group and the remaining ports 4 Gbps shared

Dedicated 8 Gbps on the first port of each group and the remaining ports 4 Gbps shared

Shared Auto with Maximumu of 4 Gbps on all ports (initial & default settings)

1 Auto is 8 Gbps.


Step 4 Click Apply.


Dynamic Bandwidth Management

On port switching modules where bandwidth is shared, the bandwidth available to each port within a port group can be configured based on the port rate mode and speed configurations. Within a port group, some ports can be configured in dedicated rate mode while others operate in shared mode.

Ports configured in dedicated rate mode are allocated the required bandwidth to sustain a line rate of traffic at the maximum configured operating speed, and ports configured in shared mode share the available remaining bandwidth within the port group. Bandwidth allocation among the shared mode ports is based on the operational speed of the ports. For example, if four ports operating at speeds 1 Gbps, 1 Gbps, 2 Gbps, and 4 Gbps share bandwidth of 8 Gbps, the ratio of allocation would be 1:1:2:4.

Out-of-Service Interfaces

On supported modules and fabric switches, you might need to allocate all the shared resources for one or more interfaces to another interface in the port group or module. You can take interfaces out of service to release shared resources that are needed for dedicated bandwidth. When an interface is taken out of service, all shared resources are released and made available to the other interface in the port group or module. These shared resources include bandwidth, rate mode, BB_credits, and extended BB_credits. All shared resource configurations are returned to their default values when the interface is brought back into service. Corresponding resources must be made available in order for the port to be successfully returned to service.


Caution If you need to bring an interface back into service, you might disrupt traffic if you need to release shared resources from other interfaces in the same port group.

Buffer Credit Allocation

This section describes how buffer credits are allocated to switches and modules, and includes the following topics:

Buffer Pools

BB_Credit Buffers for Switching Modules

BB_Credit Buffers for Fabric Switches

Extended BB_Credits

Buffer Pools

In the architecture of Generation 2 and Generation 3 modules, receive buffers shared by a set of ports are called buffer groups. The receive buffer groups are organized into global and local buffer pools.

The receive buffers allocated from the global buffer pool to be shared by a port group are called a global recieve buffer pool. Global receive buffer pools include the following buffer groups:

Reserved internal buffers

Allocated BB_credit buffers for each Fibre Channel interface (user configured or assigned by default)

Common unallocated buffer pool for BB_credits, if any, to be used for additional BB_credits as needed

Performance buffers (only used on 12-port 4-Gbps and 4-port 10-Gbps switching modules)


Note The 48-Port and 24-Port 8-Gbps modules have dual global buffer pools. Each buffer pool in the 48-port modules support 24 ports and in the 24-port modules each buffer pool supports 12 ports.


Figure 22-3 shows the allocation of BB_credit buffers on linecards (24-port and 48-port 4-Gbps line cards).

Figure 22-3 Receive Buffers for Fibre Channel Ports in a Global Buffer Pool

Figure 22-4 shows the default BB_credit buffer allocation model for 48-port 8-Gbps switching modules. The minimum BB_credits required to bring up a port is two buffers.

Figure 22-4 BB_Credit Buffer Allocation in 48-port 8-Gbps Switching Modules

Figure 22-5 shows the default BB_credit buffer allocation model for 24-port 8-Gbps switching modules. The minimum BB_credits required to bring up a port is two buffers.

Figure 22-5 BB_Credit Buffer Allocation in 24-port 8-Gbps Switching Modules

Figure 22-6 shows the default BB_credit buffer allocation model for 4/44-port 8-Gbps Host-Optimized switching modules. The minimum BB_credits required to bring up a port is two buffers.

Figure 22-6 BB_Credit Buffer Allocation in 4/44-port 8-Gbps Switching Modules

Figure 22-7 shows the default BB_credit buffer allocation model for 24-port 4-Gbps switching modules. The minimum BB_credits required to bring up a port is two buffers.

Figure 22-7 BB_Credit Buffer Allocation in 24-port 4-Gbps Switching Modules


Note The default BB_credit buffer allocation is the same for all port speeds.


BB_Credit Buffers for Switching Modules

This section describes how buffer credits are allocated to Cisco MDS 9000 switching modules, and includes the following topics:

48-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

24-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

4/44-Port 8-Gbps Host-Optimized Fibre Channel Module BB_Credit Buffers

48-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

24-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

18-Port Fibre Channel/4-Port Gigabit Ethernet Multiservice Module BB_Credit Buffers

4-Port 10-Gbps Switching Module BB_Credit Buffers

48-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

Table 22-8 lists the BB_credit buffer allocation for the 48-port 8-Gbps Fibre Channel switching module.

Table 22-8 48-Port 8-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
8-Gbps Speed
Shared Rate Mode
8-Gbps Speed
ISL
Fx Port
Fx Port

Default BB_credit buffers

250

32

32

Maximum BB_credit buffers

500

500

32

Total Number of BB_Credit Buffers per Module

Ports 1 through 24

6000

Ports 25 through 48

6000


The following guidelines apply to BB_credit buffers on 48-port 8-Gbps Fibre Channel switching modules:

BB_credit buffers allocated for ports 1 through 24 and 25 through 48 can be a maximum of 6000 each so that the load is distributed.

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 500 buffers for dedicated rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 500 buffers for dedicated rate mode or 32 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 48-port 8-Gbps Fibre Channel switching module consists of six ports. The ports in shared rate mode in a port group can have a maximum bandwidth oversubscription of 10:1 considering that each port group has 12.8-Gbps bandwidth.

The following example configurations are supported by the 48-port 8-Gbps Fibre Channel switching modules:

Six ports with shared rate mode and 8-Gbps speed (4:1 oversubscription) (default)

One port with dedicated rate mode and 8-Gbps speed plus
five ports with shared rate mode and 8-Gbps speed (10:1 oversubscription)

Two ports with dedicated rate mode and 4-Gbps speed plus
four ports with shared rate mode and 4-Gbps speed (4:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
three ports with dedicated rate mode and 2-Gbps speed plus
two ports with shared rate mode and 4-Gbps speed (4:1 oversubscription) 

Six ports with dedicated rate mode and 2-Gbps speed

24-Port 8-Gbps Fibre Channel Module BB_Credit Buffers

Table 22-9 lists the BB_credit buffer allocation for the 24-port 8-Gbps Fibre Channel switching module.

Table 22-9 24-Port 8-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
8-Gbps Speed
Shared Rate Mode
8-Gbps Speed
ISL
Fx Port
Fx Port

Default BB_credit buffers

500

32

32

Maximum BB_credit buffers

5001

5001

32

Total Number of BB_Credit Buffers per Module

Ports 1 through 12

6000

Ports 13 through 24

6000

1 When connected to Generation 1 modules, reduce the maximum BB_credit allocation to 250.


The following guidelines apply to BB_credit buffers on 24-port 8-Gbps Fibre Channel switching modules:

BB_credit buffers allocated for ports 1 through 12 and 13 through 24 can be a maximum of 6000 each so that the load is distributed.

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 500 buffers for dedicated rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 500 buffers for dedicated rate mode or 32 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 24-port 8-Gbps Fibre Channel switching module consists of three ports. The ports in shared rate mode in a port group can have a maximum bandwidth oversubscription of 10:1 considering that each port group has 12.8-Gbps bandwidth.

The following example configurations are supported by the 24-port 8-Gbps Fibre Channel switching modules:

Three ports with shared rate mode and 8-Gbps speed (2:1 oversubscription) (default)

One port with dedicated rate mode and 8-Gbps speed plus
two ports with shared rate mode and 8-Gbps speed (4:1 oversubscription)

One port with dedicated rate mode and 8-Gbps speed plus
one port with dedicated rate mode and 4-Gbps speed plus
one port with shared rate mode and 8-Gbps speed (10:1 oversubscription)

Two ports with dedicated rate mode and 4-Gbps speed plus
one port with shared rate mode and 8-Gbps speed (2:1 oversubscription) 

Three ports with dedicated rate mode and 4-Gbps speed

4/44-Port 8-Gbps Host-Optimized Fibre Channel Module BB_Credit Buffers

Table 22-10 lists the BB_credit buffer allocation for the 4/44-port 8-Gbps Fibre Channel switching module.

Table 22-10 4/44-Port 8-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
8-Gbps Speed
Shared Rate Mode
8-Gbps Speed
ISL
Fx Port
Fx Port

Default BB_credit buffers

125

32

32

Maximum BB_credit buffers

250

250

32

Total number of BB_credit buffers per module

6000


The following guidelines apply to BB_credit buffers on 4/44-port 8-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 500 buffers for dedicated rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 32 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 24-port 8-Gbps Fibre Channel switching module consists of 12 ports. The ports in shared rate mode in a port group can have a maximum bandwidth oversubscription of 10:1 considering that each port group has 12.8-Gbps bandwidth.

The following example configurations are supported by the 4/44-port 8-Gbps Fibre Channel switching modules:

Twelve ports with shared rate mode and 4-Gbps speed (5:1 oversubscription) (default)

One port with dedicated rate mode and 8-Gbps speed plus
eleven ports with shared rate mode and 4-Gbps speed (10:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
three ports with dedicated rate mode and 3-Gbps speed plus
eight ports with shared rate mode and 4-Gbps speed (2:1 oversubscription)

Twelve ports with dedicated rate mode and 1-Gbps speed

48-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

Table 22-11 lists the BB_credit buffer allocation for 48-port 4-Gbps Fibre Channel switching modules.

Table 22-11 48-Port 4-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL 1
Fx Port
Fx Port

Default BB_credit buffers

125

16

16

Maximum BB_credit buffers

250

250

16

Total number of BB_credit buffers per module

6000

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 48-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 48-port 4-Gbps Fibre Channel switching module consists of 12 ports. The ports in shared rate mode have bandwidth oversubscription of 2:1 by default. However, some configurations of the shared ports in a port group can have maximum bandwidth oversubscription of 4:1 (considering that each port group has 12.8-Gbps bandwidth).

The following example configurations are supported by the 48-port 4-Gbps Fibre Channel switching modules:

Twelve ports with shared rate mode and 4-Gbps speed (4:1 oversubscription) (default)

One port with dedicated rate mode and 4-Gbps speed plus
11 ports with shared rate mode and 4-Gbps speed (5:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
11 ports with shared rate mode and 2-Gbps speed (2.5:1 oversubscription)

Two ports with dedicated rate mode and 2-Gbps speed plus
10 ports with shared rate mode and 4-Gbps speed (5:1 oversubscription) 

Two ports with dedicated rate mode and 2-Gbps speed plus
10 ports with shared rate mode and 2-Gbps speed (2.5:1 oversubscription) 

Twelve ports with dedicated rate mode and 1-Gbps speed

Three ports with dedicated rate mode and 4-Gbps speed plus
four ports with shared rate mode and 1-Gbps speed plus
five ports put out-of-service (see Figure 22-8)

Figure 22-8 Example Speed and Rate Configuration on a 48-Port 4-Gbps Switching Module

Six ports with dedicated rate mode and 2-Gbps speed plus
four ports with shared rate mode and 1-Gbps speed plus
two ports put out-of-service (see Figure 22-9)

Figure 22-9 Example Speed and Rate Configuration on a 48-Port 4-Gbps Switching Module

24-Port 4-Gbps Fibre Channel Module BB_Credit Buffers

Table 22-12 lists the BB_credit buffer allocation for 24-port 4-Gbps Fibre Channel switching modules.

Table 22-12 24 Port 4-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL 1
Fx Port
Fx Port

Default BB_credit buffers

250

16

16

Maximum BB_credit buffers

250

250

16

Total number of BB_credits buffers per module

6000

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 24-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 24-port 4-Gbps Fibre Channel switching module consists of six ports. The ports in shared rate mode have a bandwidth oversubscription of 2:1 by default. However, some configurations of the shared ports in a port group can have a maximum bandwidth oversubscription of 4:1 (considering that each port group has 12.8-Gbps bandwidth).

The following example configurations are supported by the 24-port 4-Gbps Fibre Channel switching modules:

Six ports with shared rate mode and 4-Gbps speed (2:1 oversubscription) (default)

Two ports with dedicated rate mode and 4-Gbps speed plus
four ports with shared rate mode and 4-Gbps speed (with 4:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
three ports with dedicated rate mode and 2-Gbps speed plus
two ports with shared rate mode and 4-Gbps speed (4:1 oversubscription)

Six ports with dedicated rate mode and 2-Gbps speed

Three ports with dedicated rate mode and 4-Gbps speed plus
three ports with shared rate mode and 1-Gbps speed (see Figure 22-10)

Figure 22-10 Example Speed and Rate Configuration on a 24-Port 4-Gbps Switching Module

18-Port Fibre Channel/4-Port Gigabit Ethernet Multiservice Module BB_Credit Buffers

Table 22-12 lists the BB_credit buffer allocation for 18-port 4-Gbps multiservice modules.

Table 22-13 18-Port 4-Gbps Multiservice Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation
BB_Credit Buffers Per Port
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL 1
Fx Port
ISL 1
Fx Port

Defualt BB_credit buffers

250

16

16

16

Maximum BB_credit buffers

250

250

16

16

Total number of BB_credit buffers per module

4509

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on18-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

12-Port 4-Gbps Switching Module BB_Credit Buffers

Table 22-14 lists the BB_credit buffer allocation for 12-port 4-Gbps switching modules.

Table 22-14 12-Port 4-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port
Dedicated Rate Mode
4-Gbps Speed
ISL 1
Fx Port

Default BB_credit buffers

250

16

Maximum BB_credit buffers

250

16

Default Performance buffers

145

12

Total number of BB_credit buffers per module

5488

Total number of performance buffers per module

512 (shared)

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 12-port 4-Gbps switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

By default, 512 performance buffers are preallocated and are shared by all the ports. These buffers are configurable and the buffers are assigned to the port based on the availability of the buffers in the shared pool.

There are 2488 extra buffers available as extended BB_credit buffers after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 - (250 * 12)).


Note Extended BB_credits are allocated across all ports on the switch. That is, they are not allocated by port group.



Note By default, the ports in the 12-port 4-Gbps switching modules come up in 4-Gbps dedicated rate mode but can be configured as 1-Gbps and 2-Gbps dedicated rate mode. Shared mode is not supported.


4-Port 10-Gbps Switching Module BB_Credit Buffers

Table 22-15 lists the BB_credit buffer allocation for 4-port 10-Gbps switching modules.

Table 22-15 4-Port 10-Gbps Switching Module BB_Credit Buffer Allocation

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port
Dedicated Rate Mode
10-Gbps Speed
ISL 1
F port 2

Default BB_credit buffers

250

16

Maximum BB_credit buffers

750

16

Maximum BB_credit buffers on one of the ports with Enterprise license

4095

 

Total number of BB_credit buffers per module

5488

Default Performance buffers

145

12

Total number of performance buffers per module

512 (shared)

1 ISL = E port or TE port.

2 Ports on the 4-port 10-Gbps cannot operate in FL port mode.



Note The ports in the 4-port 10-Gbps switching module only support 10-Gbps dedicated rate mode. FL port mode and shared rate mode are not supported.


The following considerations apply to BB_credit buffers on 4-port 10-Gbps switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 750 buffers.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 750 buffers.

By default, 512 performance buffers are preallocated and are shared by all the ports. These buffers are configurable and the buffers are assigned to the port based on the availability of the buffers in the shared pool.

There are 2488 extra buffers available as extended BB_credits after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 - (750 * 4)).


Note Extended BB_credits are allocated across all ports on the switch. That is, they are not allocated by port group.


BB_Credit Buffers for Fabric Switches

This section describes how buffer credits are allocated to Cisco MDS 9000 Fabric switches, and includes the following topics:

Cisco MDS 9134 Fabric Switch BB_Credit Buffers

Cisco MDS 9124 Fabric Switch BB_Credit Buffers

Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers

Cisco MDS 9134 Fabric Switch BB_Credit Buffers

Table 22-16 lists the BB_credit buffer allocation for 32-port 4-Gbps Fibre Channel switches.

Table 22-16 32-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port
 
ISL 1
Fx Port

User-configurable BB_credit buffers on 4-Gbps mode

64

64

64

Default BB_credit buffers

64

16

16

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 32-port 4-Gbps switches:

BB_credit buffers can be configured from a minimum of 1 buffer to a maximum of 61 buffers per port when the ports are in F mode and in 4-Gbps speed mode.

BB_credit buffers can be configured from a minimum of 2 buffers to a maximum of 64 buffers per port when the ports are in auto or E mode and in 4-Gbps speed mode.

BB_credit buffers can be configured from a minimum of 64 buffers to a maximum of 64 buffers per port when a port is in 10-Gbps speed mode. There can be only one port per port group configured in 10-Gbps mode. The rest of the three ports must be in down state.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 64 buffers.

Cisco MDS 9124 Fabric Switch BB_Credit Buffers

Table 22-17 lists the BB_credit buffer allocation for 24-port 4-Gbps Fibre Channel switches.

Table 22-17 24-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port Defaults
 
ISL 1
Fx Port

User-configurable BB_credit buffers

64

16

16

1 ISL = E port or TE port.


Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers

Table 22-18 lists the BB_credit buffer allocation for 18-port 4-Gbps Multiservice Modular switches.

Table 22-18 18-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port Defaults
 
ISL 1
Fx Port

User-configurable BB_credit buffers

4509

250

16

1 ISL = E port or TE port.


Extended BB_Credits


Note Extended BB_credits are not supported on the Cisco MDS 9124 Fabric Switch, Cisco MDS 9134 Fabric Switch, Cisco MDS 9222i Fabric Switch, the Cisco Fabric Switch for HP c-Class BladeSystem, and the Cisco Fabric Switch for IBM BladeCenter.


To facilitate BB_credits for long haul links, the extended BB_credits feature allows the user to configure the receive buffers above the maximum value on all Generation 2 switching modules (see the "Buffer Credit Allocation" section). When necessary, you can reduce the buffers on one port and assign them to another port, exceeding the default maximum. The minimum extended BB_credits per port is 256 and the maximum is 4095.

In general, the user can configure any port in a port group to dedicated rate mode. To do this, you must first release the buffers from the other ports before configuring larger extended BB_credits for a port.


Note The ENTERPRISE_PKG license is required to use extended BB_credits on Generation 2 switching modules. Also, extended BB_credits are not supported by ports in shared rate mode.

All ports on the Generation 2 switching modules support extended BB_credits. There are no limitations for how many extended BB_credits you can assign to a port (except for the maximum and minimum limits). If necessary, you can take interfaces out of service to make more extended BB_credits available to other ports.


Combining Generation 1, Generation 2, and Generation 3 Modules

Cisco MDS NX-OS Release 4.1(1) and later supports combining Generation 1, Generation 2, and Generation 3 modules and switches with the following considerations:

MDS NX-OS Release 4.1(1) and later features are not supported on Generation 1 switches and modules.

Generation 3 modules do not support the following Generation 1 hardware:

Supervisor 1 module

4-Port IP Storage Services module

8-Port IP Storage Services module

MDS 9216 Switch

MDS 9216A switch

MDS 9020 switch

MDS 9120 switch

MDS 9140 swtich

Supervisor-1 modules must be upgraded to Supervisor-2 modules on the MDS 9506 and MDS 9509 Directors.

IPS-4 and IPS-8 modules must be upgraded to the MSM-18/4 Multiservice modules.

Fabric 1 modules must be upgraded to Fabric 2 modules on the MDS 9513 Director to use the 48-port or the 24-port 8-Gbps module, and the Fabric 2 modules must be configured in active-active mode.

MDS Fabric Manager Release 4.x supports MDS SAN-OS Release 3.x and NX-OS 4.x in mixed mode through Interswitch Link (ISL) connectivity.


Note When a Cisco or a other vendor switch port is connected to a Generation 1 module port (ISL connection), the receive buffer-to-buffer credits of the port connected to the Generation 1 module port should not exceed 255.


Port Indexes

Cisco MDS 9000 switches allocate index identifiers for the ports on the modules. These port indexes cannot be configured. You can combine Generation 1, Generation 2, and Generation 3 switching modules, with either Supervisor-1 modules or Supervisor-2 modules. However, combining switching modules and supervisor modules has the following port index limitations:

Supervisor-1 modules only support a maximum of 252 port indexes, regardless of the type of switching modules.

Supervisor-2 modules support a maximum of 1020 port indexes when all switching modules in the chassis are Generation 2 or Generation 3.

Supervisor-2 modules only support a maximum of 252 port indexes when only Generation 1 switching modules, or a combination of Generation 1, Generation 2, or Generation 3 switching modules, are installed in the chassis.


Note On a switch with the maximum limit of 252 port index maximum limit, any new module that exceeds the limit when installed does not power up.


Generation 1 switching modules have specific numbering requirements. If these requirements are not met, the module does not power up. The port index numbering requirements include the following:

If port indexes in the range of 256 to 1020 are assigned to operational ports, Generation 1 switching modules do not power up.

A block of contiguous port indexes is available. If this block of port indexes is not available, Generation 1 modules do not power up. Table 22-19 shows the port index requirements for the Generation 1 modules.


Note If the switch has Supervisor-1 modules, the block of 32 contiguous port indexes must begin on the slot boundary. The slot boundary for slot 1 is 0, for slot 2 is 32, and so on. For Supervisor-2 modules, the contiguous block can start anywhere.


Table 22-19 Port Index Requirements for Generation 1 Modules 

Generation 1 Module
Number of Port Indexes Required
Supervisor-1 Module
Supervisor-2 Module

16-port 2-Gbps Fibre Channel module

16

16

32-port 2-Gbps Fibre Channel module

32

32

8-port Gigabit Ethernet IP Storage Services module

32

32

4-port Gigabit Ethernet IP Storage Services module

32

16

32-port 2-Gbps Fibre Channel Storage Services Module (SSM).

32

32

14-port Fibre Channel/2-port Gigabit Ethernet Multiprotocol Services (MPS-14/2) module

32

22


The allowed mix of Generation 1 and Generation 2 switching modules in a chassis is determined at run-time, either when booting up the switch or when installing the modules. In some cases, the sequence in which switching modules are inserted into the chassis determines if one or more modules is powered up. When a module does not power up because of a resource limitation, you can see the reason by viewing the module information in the Information pane.

For information on recovering a module powered-down because port indexes are not available, refer to the Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x.

PortChannels

PortChannels have the following restrictions:

The maximum number of PortChannels allowed is 256 if all switching modules are Generation 2 or Generation 3, or both.

The maximum number of PortChannels allowed is 128 whenever there is a Generation 1 switching module in use with a Generation 2 or Generation 3 switching module.

Ports need to be configured in dedicated rate mode on the Generation 2 and Generation 3 switching module interfaces to be used in the PortChannel.


Note The number of PortChannels allowed does not depend on the type of supervisor module. However, Generation 3 modules require the Supervisor 2 module on the MDS 9506 and 9509 switches.


The Generation1, Generation 2, and Generation 3 modules have the following restrictions for PortChannel configuration:

Generation 1 switching module interfaces do not support auto speed with a maximum of 2 Gbps.

Generation 1 and Generation 2 module interfaces do not support auto speed with maximum of 4 Gbps.

Generation 2 and Generation 3 switching module interfaces cannot be forcefully added to a PortChannel if sufficient resources are not available.

When configuring PortChannels on switches with Generation 1, Generation 2, and Generation 3 switching modules, follow one of these procedures:

Configure the PortChannel, and then configure the Generation 2 and Generation 3 interfaces to auto with a maximum of 2 Gbps.

Configure the Generation 1 switching modules followed by the Generation 2 switching modules, and then the Generation 3 switching modules, and then configure the PortChannel.

When configuring PortChannels on switches with only Generation 2 and Generation 3 switching modules, follow one of these procedures:

Configure the PortChannel, and then configure the Generation 3 interfaces to auto with a maximum of 4 Gbps.

Configure the Generation 2 switching modules, followed by the Generation 3 switching modules, and then configure the PortChannel.

Table 22-20 describes the results of adding a member to a PortChannel for various configurations.

Table 22-20 PortChannel Configuration and Addition Results 

PortChannel Members
Configured Speed
New Member Type
Addition Type
Result
PortChannel
New Member

No members

Any

Any

Generation 1 or Generation 2 or Generation 3

Force

Pass

Auto

Auto

Generation 1 or Generation 2 or Generation 3

Normal or force

Pass

Auto

Auto max 2000

Generation 2 or Generation 3

Normal

Fail

Force

Pass or fail1

Auto

Auto max 4000

Generation 3

   

Auto max 2000

Auto

Generation 2 or Generation 3

Normal

Fail

Force

Pass

Auto max 2000

Auto max 4000

Generation 3

   

Auto max 4000

Auto

Generation 2 or Generation 3

   
   

Auto max 4000

Auto max 2000

Generation 2 or Generation 3

   

Generation 1 interfaces

Auto

Auto

Generation 2 or Generation 3

Normal

Fail

Force

Pass

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2 or Generation 3

Normal

Fail

Force

Pass or fail1

Auto max 4000

Auto

Generation 1 or Generation 2

   
   

Auto max 4000

Auto

Generation 3

   
   

Generation 2 interfaces

Auto

Auto

Generation 1

Normal or force

Fail

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2 or Generation 3

Normal

Fail

Force

Pass

Auto

Auto max 2000

Generation 2 or Generation 3

Normal

Fail

Force

Pass

Generation 3 interfaces

Auto

Auto

Generation 1

Normal or force

Fail

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2

Normal

Fail

Force

Pass

Auto

Auto max 2000

Generation 2

Normal

Fail

Force

Pass

Auto max 2000

Auto

Generation 3

Normal

Fail

Force

Pass

Auto

Auto max 2000

Generation 3

Normal

Fail

Force

Pass

1 If resources are not available.


Configuring Module Interface Shared Resources

This section describes how to configure Generation 2 and Generation 3 module interface shared resources and contains the following sections:

Configuration Guidelines for 48-Port, 24-Port, and 4/44-Port 8-Gbps Fibre Channel Switching Modules

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

Configuring Port Speed

Configuring Rate Mode

Configuring Oversubscription Ratio Restrictions

Configuring Bandwidth Fairness

Taking Interfaces Out of Service

Releasing Shared Resources in a Port Group

 
   

Configuration Guidelines for 48-Port, 24-Port, and 4/44-Port 8-Gbps Fibre Channel Switching Modules

The 48-Port, 24-Port, and 4/44-Port 8-Gbps Fibre Channel switching modules support the following features:

1-Gbps, 2-Gbps, 4-Gbps, and 8-Gbps speed traffic

Shared and dedicated rate mode

ISL and Fx port modes

Extended BB_credits

Migrating from Shared Mode to Dedicated Mode

To configure 48-port, 24-port, 4/44-port 8-Gbps Fibre Channel switching modules when starting with the default configuration or when migrating from shared rate mode to dedicated rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the traffic speed to use (1 Gbps, 2 Gbps, 4 Gbps, 8 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps).

See the "Configuring Port Speed" section.

3. Configure the rate mode (dedicated or shared).

See the "Configuring Rate Mode" section.

4. Configure the port mode.

See the "About Interface Modes" section on page 20-3.


Note ISL ports cannot operate in shared rate mode.


5. Configure the BB_credits and extended BB_credits, as necessary.

See the "Extended BB_Credits" section.

Migrating from Dedicated Mode to Shared Mode

To configure 48-port, 24-port, 4/44-port 8-Gbps Fibre Channel switching modules migrating from dedicated rate mode to shared rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the BB_credits and extended BB_credits, as necessary.

See the "BB_Credit Buffers for Switching Modules" section, "BB_Credit Buffers for Fabric Switches" section, and the "Extended BB_Credits" section.

3. Configure the port mode.

See the "About Interface Modes" section on page 20-3.


Note ISL ports cannot operate in shared rate mode.


4. Configure the rate mode (dedicated or shared) to use.

See the "Configuring Rate Mode" section.

5. Configure the traffic speed (1 Gbps, 2 Gbps, 4 Gbps, 8 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps) to use.

See the "Configuring Port Speed" section.

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

The 48-port and 24-port 4-Gbps Fibre Channel switching modules support the following features:

1-Gbps, 2-Gbps, and 4-Gbps speed traffic

Shared and dedicated rate mode

ISL (E or TE) and Fx (F or FL) port modes

Extended BB_credits

Migrating from Shared Mode to Dedicated Mode

To configure 48-port and 24-port 4-Gbps Fibre Channel switching modules when starting with the default configuration or when migrating from shared rate mode to dedicated rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the traffic speed to use (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps).

See the "Configuring Port Speed" section.

3. Configure the rate mode (dedicated or shared) to use.

See the "Configuring Rate Mode" section.

4. Configure the port mode.

See the "About Interface Modes" section on page 20-3.


Note ISL ports cannot operate in shared rate mode.


5. Configure the BB_credits and extended BB_credits, as necessary.

See the "Extended BB_Credits" section.

Migrating from Dedicated Mode to Shared Mode

To configure 48-port and 24-port 4-Gbps Fibre Channel switching modules migrating from dedicated rate mode to shared rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the BB_credits and extended BB_credits, as necessary.

See the "BB_Credit Buffers for Switching Modules" section, "BB_Credit Buffers for Fabric Switches" section, and the "Extended BB_Credits" section.

3. Configure the port mode.

See the "About Interface Modes" section on page 20-3.


Note ISL ports cannot operate in shared rate mode.


4. Configure the rate mode (dedicated or shared) to use.

See the "Configuring Rate Mode" section.

5. Configure the traffic speed (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps) to use.

See the "Configuring Port Speed" section.

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

The 12-port 4-Gbps switching modules support the following features:

1-Gbps, 2-Gbps, and 4-Gbps speed traffic

Only dedicated rate mode

ISL (E or TE) and Fx (F or FL) port modes

Extended BB_credits

Performance buffers

To configure 4-port 10-Gbps switching modules when starting with the default configuration, follow these guidelines:

1. Configure the traffic speed (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps) to use.

See the "Configuring Port Speed" section.

2. Configure the port mode.

See the "About Interface Modes" section on page 20-3.

3. Configure the BB_credits, performance buffers, and extended BB_credits, as necessary.

See the "BB_Credit Buffers for Switching Modules" section, "BB_Credit Buffers for Fabric Switches" section, and the "Extended BB_Credits" section.


Note If you change the port bandwidth reservation parameters on a 48-port or 24-port module, the change affects only the changed port. No other ports in the port group are affected.


Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

The 4-port 10-Gbps switching modules support the following features:

Only 10-Gbps speed traffic

Only dedicated rate mode

ISL (E or TE) and F port modes

Extended BB_credits

Performance buffers

Use the following guidelines to configure 4-port 10-Gbps switching modules when starting with the default configuration:

1. Configure the port mode.

See the "About Interface Modes" section on page 20-3.

2. Configure the BB_credits, performance buffers, and extended BB_credits, as necessary.

See the "BB_Credit Buffers for Switching Modules" section, "BB_Credit Buffers for Fabric Switches" section, and the "Extended BB_Credits" section.

Configuring Port Speed

The port speed on an interface, combined with the rate mode, determines the amount of shared resources available to the ports in the port group on a 48-port, 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching module. Especially in the case of dedicated rate mode, the port group resources are reserved even though the bandwidth is not used. For example, on Generation 2 modules, if an interface is configured for autosensing (auto) and dedicated rate mode, then 4 Gbps of bandwidth is reserved even though the maximum operating speed is 2 Gbps. For the same interface, if autosensing with a maximum speed of 2 Gbps (auto max 2000) is configured, then only 2 Gbps of bandwidth is reserved and the unused 2 Gbps is shared with the other interface in the port group.


NoteThe Generation 2, 4-port 10-Gbps switching module supports 10-Gbps traffic only.

On Generation 3, 8-Gbps modules, setting the port speed to auto enables autosensing, which negotiates to a maximum speed of 8 Gbps.

On Generation 2, 4-Gbps modules, setting the port speed to auto enables autosensing, which negotiates to a maximum speed of 4 Gbps.



Caution Changing port speed and rate mode disrupts traffic on the port. Traffic on other ports in the port group is not affected.

To configure dedicated bandwidth on an interface using Fabric Manager, follow these steps:


Step 1 Select a switch from the Fabric pane, or select a group of switches (SAN, fabric, VSAN) from the Logical Domains pane.

Step 2 Expand Switches, expand Interfaces and select FC Physical from the Physical Attributes pane.

You see the FC Physical > General tab in the Interfaces pane.

Step 3 Scroll until you see the row containing the switch and port you want to configure.

Step 4 Select auto, 1Gb, 4Gb, or autoMax2G from the Speed Admin column (see Figure 22-11).


Note The Generation 3, 8-Gbps Fibre Channel swtiching modules support the following speed configurations: 1G, 2G, 4G, 8G, autoMax2G, autoMax4G and the auto speed configuration configures autosensing for the interface with 8 Gbps of bandwidth reserved.


Figure 22-11 Speed Admin Column in Port Configuration

The auto parameter enables autosensing on the interface. The autoMax2G parameter enables autosensing on the interface with a maximum speed of 2 Gbps.


Note If you change the port bandwidth reservation parameters on a 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching module, the change affects only the changed port. No other ports in the port group are affected.


Step 5 Click the Apply Changes icon.


Configuring Rate Mode

To configure the rate mode (dedicated or shared) on an interface on a 4-Gbps or 8-Gbps Fibre Channel switching module using Fabric Manager, follow these steps:


Step 1 Select a switch from the Fabric pane, or select a group of switches (SAN, fabric, VSAN) from the Logical Domains pane.

Step 2 Expand Switches > Interfaces and then select FC Physical from the Physical Attributes pane.

You see the FC Physical > General tab in the Interfaces pane.

Step 3 Scroll until you see the row containing the switch and port you want to configure.

Step 4 Select dedicated or shared from the Rate Mode column (see Figure 22-12).

Figure 22-12 Rate Mode Port Configuration

Step 5 Click the Apply Changes icon.



Caution Changing port speed and rate mode disrupts traffic on the port.

Configuring Oversubscription Ratio Restrictions

The 48-port and 24-port 4-Gbps, and all 8-Gbps Fibre Channel switching modules support oversubscription on switches with shared rate mode configurations. By default, all 48-port and 24-port 4-Gbps, and 8-Gbps Fibre Channel switching modules have restrictions on oversubscription ratios enabled. As of Cisco SAN-OS Release 3.1(1) and NX-OS Release 4.1(1), you can disable restrictions on oversubscription ratios.

Table 22-21 describes the bandwidth allocation for oversubscribed interfaces configured in shared mode on the 4-Gbps modules.

Table 22-21 Bandwidth Allocation for Oversubscribed Interfaces 

Switching Module
Configured Speed
Reserved Bandwidth (Gbps)
Maximum Bandwidth (Gbps)
Ratios enabled
Ratios disabled

48-Port 8-Gbps Fibre Channel Module

Auto 8 Gbps

0.36

0.2

8

Auto Max 4 Gbps

0.24

0.1

4

Auto Max 2 Gbps

0.12

0.05

2

24-Port 8-Gbps Fibre Channel Module

Auto 8 Gbps

0.8

0.8

8

Auto Max 4 Gbps

0.4

0.4

4

Auto Max 2 Gbps

0.2

0.2

2

4/44-Port 8-Gbps Host-Optimized Fibre Channel Module

8 Gbps

0.87

0.16

8

Auto Max 4 Gbps

0.436

0.08

4

Auto Max 2 Gbps

0.218

0.04

2

1 Gbps

0.109

0.02

1

48-port 4-Gbps Fibre Channel switching module

Auto 4 Gbps

0.8

0.09

4

Auto Max 2 Gbps

0.4

0.045

2

1 Gbps

0.2

0.0225

1

24-port 4-Gbps Fibre Channel switching module

Auto 4 Gbps

1

0.27

4

Auto Max 2 Gbps

0.5

0.135

2

1 Gbps

0.25

0.067

1


All ports in the 48-port and 24-port 4-Gbps modules can be configured to operate at 4 Gbps in shared mode even if other ports in the port group are configured in dedicated mode, regardless of available bandwidth. However, when oversubscription ratio restrictions are enabled, you may not have all shared 4-Gbps module ports operating at 4 Gbps.

All ports in the 48-port and 24-port 8-Gbps modules can be configured to operate at 8 Gbps in shared mode even if other ports in the port group are configured in dedicated mode, regardless of available bandwidth. However, when oversubscription ratio restrictions are enabled you may not have all shared 8-Gbps module ports operating at 8 Gbps.

On the 48-port and 24-port 8-Gbps modules, if you have configured one 8-Gbps dedicated port in one port group, no other ports in the same port group can be configured to operate at 8-Gbps dedicated mode. You can have any number of 8-Gbps shared and 4-Gbps dedicated or shared ports. On the 4/44-port 8-Gbps module, only one port per port group can be configured in 8-Gbps dedicated or shared mode.

In the following example, a 24-port 4-Gbps module has oversubscription ratios enabled and three dedicated ports in one port group operating at 4-Gbps. No other ports in the same port group can be configured to operate at 4 Gbps.

For dedicated ports, oversubscription ratio restrictions do not apply to the shared pool in port groups. So if oversubscription ratio restrictions are disabled, and you have configured three 4-Gbps dedicated ports in one port group, then you can configure all other ports in the same port group to operate at a shared rate of 4 Gbps.

When disabling restrictions on oversubscription ratios, all ports in shared mode on 48-port and 24-port 4-Gbps or any 8-Gbps Fibre Channel switching modules must be shut down. When applying restrictions on oversubscription ratios, you must take shared ports out of service.


Note When restrictions on oversubscription ratios are disabled, the bandwidth allocation among the shared ports is proportionate to the configured speed. If the configured speed is auto on Generation 2 modules, then bandwidth is allocated assuming a speed of 4 Gbps. For example, if you have three shared ports configured at 1, 2, and 4 Gbps, then the allocated bandwidth ratio is 1:2:4.
As of Cisco SAN-OS Release 3.0 and NX-OS Release 4.1(1) or when restrictions on oversubscription ratios are enabled, the port bandwidths are allocated in equal proportions, regardless of port speed, so, the bandwidth allocation for the same three ports mentioned in the example would be 1:1:1.


Disabling Restrictions on Oversubscription Ratios

Before disabling restrictions on oversubscription ratios, ensure that you have explicitly shut down shared ports. To disable restrictions on oversubscription ratios on multiple 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching modules using Device Manager, follow these steps:


Step 1 Choose Physical > Modules.

You see the Module dialog box as shown in Figure 22-13.

Figure 22-13 Module Dialog Box


Caution This feature is only supported on 48-port and 24-port 4-Gbps, and 8-Gbps Fibre Channel switching modules.

Step 2 Select disabled from the RateModeOversubscriptionLimit drop-down list for each module for which you want to disable restrictions on oversubscription ratios.

Step 3 Click Apply to save the changes.


To disable restrictions on oversubscription ratios on a single 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching module using Device Manager, follow these steps:


Step 1 Right-click a module and select Configure.

You see the Module dialog box as shown in Figure 22-14.

Figure 22-14 Module Dialog Box

Step 2 Click the disabled radio button to disable restrictions on oversubscription ratios.

Step 3 Click Apply to save the changes.


 
   
 
   

Enabling Restrictions on Oversubscription Ratios


Caution You must enable restrictions on oversubscription ratios before you can downgrade modules to a previous release.

Before enabling restrictions on oversubscription ratios, ensure that you have explicitly configured shared ports to out-of-service mode. To enable restrictions on oversubscription ratios on multiple 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching modules using Device Manager, follow these steps:


Step 1 Choose Physical > Modules.

You see the Module dialog box as shown in Figure 22-13.

Step 2 Select enabled from the RateModeOversubscriptionLimit drop-down list for each module for which you want to enable restrictions on oversubscription ratios.

Step 3 Click Apply to save the changes.


To enable restrictions on oversubscription ratios on a single 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching module using Device Manager, follow these steps:


Step 1 Right-click a module and select Configure.

You see the Module dialog box as shown in Figure 22-14.

Step 2 Click the enabled radio button to enable restrictions on oversubscription ratios.

Step 3 Click Apply to save the changes.


Configuring Bandwidth Fairness

This feature improves fairness of bandwidth allocation among all ports and provides better throughput average to individual data streams. Bandwidth fairness can be configured per module.

As of Cisco SAN-OS Release 3.1(2), all 48-port and 24-port 4-Gbps Fibre Channel switching modules, as well as 18-port Fibre Channel/4-port Gigabit Ethernet Multiservice modules, have bandwidth fairness enabled by default. As of Cisco NX-OS Release 4.1(1), all the 8-Gbps Fibre Channel switching modules have bandwidth fairness enabled by default.


Caution When you disable or enable bandwidth fairness, the change does not take effect until you reload the module.


Note This feature is supported only on the 48-port and 24-port 4-Gbps modules, the 8-Gbps modules, and the 18/4-port Multiservice Module (MSM).


Enabling Bandwidth Fairness

To enable bandwidth fairness on multiple 48-port or 24-port 4-Gbps, or any 8-Gbps Fibre Channel switching modules using Device Manager, follow these steps:


Step 1 Choose Physical > Modules.

You see the Module dialog box as shown in Figure 22-15.

Figure 22-15 Module Dialog Box

Step 2 Select enable from the BandwidthFairnessConfig drop-down list for each module for which you want to enable bandwidth fairness.

Step 3 Click Apply to save the changes.


To enable bandwidth fairness on a single 48-port or 24-port 4-Gbps Fibre Channel switching module using Device Manager, follow these steps:


Step 1 Right-click a module and select Configure.

You see the Module dialog box as shown in Figure 22-16.

Figure 22-16 Module Dialog Box

Step 2 Click the enable radio button to enable bandwidth fairness.

Step 3 Click Apply to save the changes.


Disabling Bandwidth Fairness


Note If you disable bandwidth fairness, up to a 20 percent increase in internal bandwidth allocation is possible for each port group; however, bandwidth fairness is not guaranteed when there is a mix of shared and full-rate ports in the same port group.


To disable bandwidth fairness on multiple 48-port or 24-port 4-Gbps, or 8-Gbps Fibre Channnel switching modules using Device Manager, follow these steps:


Step 1 Choose Physical > Modules.

You see the Module dialog box as shown in Figure 22-15.

Step 2 Select disable from the BandwidthFairnessConfig drop-down list for each module for which you want to disable bandwidth fairness.

Step 3 Click Apply to save the changes.


To disable bandwidth fairness on a single 48-port or 24-port 4-Gbps, or 8-Gbps Fibre Channel switching module using Device Manager, follow these steps:


Step 1 Right-click a module and select Configure.

You see the Module dialog box as shown in Figure 22-16.

Step 2 Click the disable radio button to disable bandwidth fairness.

Step 3 Click Apply to save the changes.


Upgrade or Downgrade Scenario

When you are upgrading from a release earlier than Cisco SAN-OS Release 3.1(2), all modules operate with bandwidth fairness disabled until the next module reload. After the upgrade, any new module that is inserted has bandwidth fairness enabled.

When you are downgrading to a release earlier than Cisco SAN-OS Release 3.1(2), all modules keep operating in the same bandwidth fairness configuration prior to the downgrade. After the downgrade, any new module that is inserted has bandwidth fairness disabled.

Taking Interfaces Out of Service

You can take interfaces out of service on Generation 2 and Generation 3 switching modules. When an interface is out of service, all the shared resources for the interface are released as well as the configuration associated with those resources.


Note The interface must be disabled before it can be taken out of service.



Caution Taking interfaces out of service releases all the shared resources to ensure that they are available to other interfaces. This causes the configuration in the shared resources to revert to default when the interface is brought back into service. Also, an interface cannot come back into service unless the default shared resources for the port are available. The operation to free up shared resources from another port is disruptive.


Note The interface cannot be a member of a PortChannel.


To take an interface out of service using Fabric Manager, follow these steps:


Step 1 Select a switch from the Fabric pane, or select a group of switches (SAN, fabric, VSAN) from the Logical Domains pane.

Step 2 Expand Switches, expand Interfaces and select FC Physical in the Physical Attributes pane.

You see the FC Physical > General tab in the Information pane.

Step 3 Scroll down until you see the row containing the switch and port you want to configure.

Step 4 Scroll right (if necessary) until you see the Status Service column.

Step 5 Select in or out from the Status Service column.

Step 6 Click the Apply Changes icon.


Releasing Shared Resources in a Port Group

When you want to reconfigure the interfaces in a port group on a Generation 2 or Generation 3 module, you can return the port group to the default configuration to avoid problems with allocating shared resources.


Note The interface cannot be a member of a PortChannel.



Caution Releasing shared resources disrupts traffic on the port. Traffic on other ports in the port group is not affected.

To release the shared resources for a port group using Fabric Manager, follow these steps:


Step 1 Select a switch from the Fabric pane, or select a group of switches (SAN, fabric, VSAN) from the Logical Domains pane.

Step 2 Expand Switches > Interfaces and then select FC Physical from the Physical Attributes pane.

You see the FC Physical > General tab in the Information pane.

Step 3 Scroll down until you see the row containing the switch and port you want to configure.

Step 4 Scroll right (if necessary) until you see the Status Service column (see Figure 22-17).

Figure 22-17 Status Service Column for FC Physical

Step 5 Select the out status from the Status Service column.

Step 6 Click the Apply Changes icon.

Step 7 Select the in status from the Status Service column.

Step 8 Click the Apply Changes icon.


Displaying SFP Diagnostic Information

To view diagnostic information for multiple ports using Device Manager, follow these steps:


Step 1 Choose Interface > FC All and click the Diagnostics tab or hold down the Control key, and then click each port for which you want to view diagnostic information.

Step 2 Right-click the selected ports, select Configure, and click the Diagnostics tab.

You see the FC Interfaces dialog box shown in Figure 22-18.

Figure 22-18 FC Interfaces Dialog Box

Step 3 Click Refresh to view the latest diagnostic information.


To view diagnostic information for a single port using Device Manager, follow these steps:


Step 1 Right-click a port, select Configure, and click the Diagnostics tab.

You see the port licensing options for the selected port shown in Figure 22-19.

Figure 22-19 Diagnostics Tab for Selected Port

Step 2 Click Refresh to view the latest diagnostic information.


Default Settings

Table 22-22 lists the default settings for Generation 2 interface parameters.

Table 22-22 Default Generation 2 Interface Parameters 

Parameter
Default
48-Port 4-Gbps Switching Module
24-Port 4-Gbps Switching Module
12-Port 4-Gbps Switching Module
4-Port 10-Gbps Switching Module

Speed mode

auto1

auto1

auto1

auto2

Rate mode

shared

shared

dedicated

dedicated

Port mode

Fx

Fx

auto3

auto4

BB_credit buffers

16

16

250

250

Performance buffers

-

-

1455

1455

1 Auto speed mode on the 4-Gbps switching modules enables autosensing and negotiates to a maximum speed of 4 Gbps.

2 The 4-port 10-Gbps switching module only supports 10-Gbps traffic.

3 Auto port mode on the 12-port 4-Gbps switching module interfaces can operate in E port mode, TE port mode, and Fx port mode.

4 Auto port mode on the 4-port 10-Gbps switching module interfaces can operate in E port mode, TE port mode, and F port mode.

5 Performance buffers are shared among all ports on the module.


Table 22-23 lists the default settings for Generation 3 interface parameters.

Table 22-23 Default Generation 3 Interface Parameters 

Parameter
Default
48-Port 8-Gbps Switching Module
24-Port 8-Gbps Switching Module
4/44-Port 8-Gbps Host-Optimized Switching Module

Speed mode

auto1

auto1

auto_max_4G2

Rate mode

shared

shared

shared

Port mode

Fx

Fx

Fx

BB_credit buffers

32

32

32

1 Auto speed mode on the 8-Gbps switching modules enables autosensing and negotiates to a maximum speed of 8 Gbps.

2 Auto_max_4G speed mode on the 4/44-port 8-Gbps switching module negotiates to a maximum speed of
4 Gbps.