Configuring Generation 2 Switching Modules

Table Of Contents

Configuring Generation 2 Switches and Modules

About Generation 2 Modules and Switches

Port Groups

Port Rate Modes

Dynamic Bandwidth Management

Out-of-Service Interfaces

Buffer Credit Allocation

Buffer Pools

BB_Credit Buffers for Switching Modules

BB_Credit Buffers for Fabric Switches

Extended BB_Credits

About Combining Generation 1 and Generation 2 Switching Modules

Port Indexes

PortChannels

Configuring Generation 2 Module Interface Shared Resources

Displaying Interface Capabilities

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

Configuring Port Speed

Configuring Rate Mode

Configuring Oversubscription Ratio Restrictions

Configuring Bandwidth Fairness

Taking Interfaces Out of Service

Releasing Shared Resources in a Port Group

Enabling the Buffer-to-Buffer State Change Number

Disabling ACL Adjacency Sharing for System Image Downgrade

Displaying SFP Diagnostic Information

Example Configurations

Configuring a 24-port 4-Gbps Fibre Channel Switching Module Example

Configuring a 48-port 4-Gbps Fibre Channel Switching Module Example

Default Settings


Configuring Generation 2 Switches and Modules


The Cisco MDS 9500 Series switches and Cisco MDS 9216A and Cisco MDS 9216i switches support a set of modules called Generation 2 modules. This chapter describes how to configure these modules, as well as Generation 2 Multilayer Fabric Switches.

This chapter includes the following sections:

About Generation 2 Modules and Switches

Buffer Credit Allocation

About Combining Generation 1 and Generation 2 Switching Modules

Configuring Generation 2 Module Interface Shared Resources

Disabling ACL Adjacency Sharing for System Image Downgrade

Displaying SFP Diagnostic Information

Example Configurations

Default Settings

About Generation 2 Modules and Switches

Table 14-1 identifies the modules supported by the Cisco MDS 9500 Series switches and Cisco MDS 9216A and Cisco MDS 9216i switches, as well as the Fabric switches:

Table 14-1 Generation 2 Fibre Channel Modules and Fabric Switches 

Part Number
Product Name/Description
Module

DS-X9148

48-port 4-Gbps Fibre Channel switching module

DS-X9134

34-port 4-Gbps Fibre Channel switching module

DS-X9124

24-port 4-Gbps Fibre Channel switching module

DS-X9304-18K9

18-port 4-Gbps Fibre Channel switching module with 4 GigabitEthernet ports

DS-X9112

12-port 4-Gbps Fibre Channel switching module

DS-X9704

4-port 10-Gbps Fibre Channel switching module

DS-X9530-SF2-K9

Supervisor-2 module (Cisco MDS 9500 Series switches only)

Switch

DS-C9134-K9

Cisco MDS 9134 Fabric switch

32-port 4-Gbps Fabric switch with 2 additional 10-Gbps ports

DS-C9124

Cisco MDS 9124 Fabric switch

24-port 4-Gbps Fabric switch

DS-C9222i-K9

Cisco MDS 9222i Multiservice Modular switch

18-port 4-Gbps switch with 4 GigabitEthernet IP storage services ports, and a modular expansion slot to host Cisco MDS 9000 Family Switching and Services Modules



Note Generation 2 Fibre Channel switching modules are not supported on the Cisco MDS 9216 switch; however, they are supported by both the Supervisor-1 module and the Supervisor-2 module.


For detailed information about the installation and specifications for these modules and switches, refer to the hardware installation guide for your switch.

This section includes the following topics:

Port Groups

Port Rate Modes

Dynamic Bandwidth Management

Out-of-Service Interfaces

Buffer Pools

Extended BB_Credits

Port Groups

Each module or switch can have one or more ports in port groups that share common resources (such as bandwidth and buffer credits). Table 14-2 shows the port groups for the Generation 2 Fibre Channel switches and modules.

Table 14-2 Bandwidth and Port Groups for Generation 2 FC Modules and Fabric Switches 

Part Number
Product Name/ Description
Number of Ports Per Port Group
Bandwidth Per Port Group
Maximum Bandwidth Per Port
Module

DS-X9148

Cisco 48-port 4-Gbps Fibre Channel module

48-port 4-Gbps Fibre Channel switching module1

12

12.8

4-Gbps

DS-X9124

Cisco 24-port 4-Gbps Fibre Channel module

24-port 4-Gbps Fibre Channel switching module

6

12.8

4

DS-X9304-18K9

Cisco 18-port Fibre Channel /4-port GigabitEthernet Multiservice (MSM-18/4) module

18-port 4-Gbps Fibre Channel switching module with 4 GigabitEthernet ports

6

12.8

4-Gbps

DS-X9112

Cisco 12-port 4-Gbps Fibre Channel module

12-port 4-Gbps Fibre Channel switching module

3

12.8

4-Gbps

DS-X9704

Cisco 4-port 10-Gbps Fibre Channel module

4-port 10-Gbps Fibre Channel switching module

1

10

10-Gbps

Switches

DS-C9134-K9

Cisco MDS 9134 Fabric switch

32-port 4-Gbps Fabric switch

4

16

4-Gbps

2-port 10-Gbps Fabric switch

1

10

10-Gbps

DS-C9124-K9

Cisco MDS 9124 Fabric switch

24-port 4-Gbps

4

16

4-Gbps

DS-C9222i-K9

Cisco MDS 9222i Multiservice Modular switch

18-port 4-Gbps

6

12.8

4-Gbps

1 By default, all ports in a 48-port 4-Gbps switching module operate in shared mode with administrative operating speed set to auto. All ports in a 48-port 4-Gbps switching module can operate in dedicated mode with a 1-Gbps operating speed. However, if you configure one or more ports to operate in 2-Gbps or 4-Gbps dedicated mode, some of the other ports in the module would have to operate in shared mode.



Note Port groups are defined by the hardware and consist of sequential ports. For example, ports 1 through 12, ports 13 through 24, ports 25 through 36, and ports 37 through 48 are the port groups on the 48-port 4-Gbps Fibre Channel switching modules.


Port Rate Modes

The Port rate mode configuration is used to determine the bandwidth allocation for ports in a port group. Two port rate modes are supported: Dedicated Mode and Shared Mode. In Generation 1 modules, port rate mode is not configurable by users; rather, it is determined implicitly based on the port mode and linecard type. In Generation 2 modules, port rate mode is user-configured.

Table 14-3 Port Rate Mode Support on Generation 2 Modules and Switches 

Part Number
Product Name/
Description
Supports Dedicated Rate Mode
Supports Shared Rate Mode
Modules

DS-X9148

Cisco 48-port 4-Gbps Fibre Channel module

48-port 4-Gbps Fibre Channel switching module1

Yes

Yes

DS-X9124

Cisco 24-port 4-Gbps Fibre Channel module

24-port 4-Gbps Fibre Channel switching module

Yes

Yes

DS-X9304-18K9

Cisco 18-port Fibre Channel /4-port GigabitEthernet Multiservice (MSM-18/4) module

18-port 4-Gbps Fibre Channel switching module with 4 GigabitEthernet ports

Yes

Yes

DS-X9112

12-port 4-Gbps Fibre Channel module

12-port 4-Gbps Fibre Channel switching module

Yes

No

DS-X9704

4-port 10-Gbps Fibre Channel module

4-port 10-Gbps Fibre Channel switching module

Yes

No

Switches

DS-C9134-K9

Cisco MDS 9134 Fabric switch

32-port 4-Gbps Fabric switch

Yes

Yes

2-port 10-Gbps Fabric switch

Yes

No

DS-C9124

Cisco MDS 9124 Fabric switch

24-port 4-Gbps Fabric switch2

Yes

No

DS-C9222i-K9

Cisco MDS 9222i Multiservice Modular switch

18-port 4-Gbps Fibre Channel switch with 4 GigabitEthernet IP storage services ports, and a modular expansion slot to host Cisco MDS 9000 Family Switching and Services Modules

Yes

Yes

1 By default, all ports in a 48-port 4-Gbps switching module operate in shared mode with administrative operating speed set to auto. All ports in a 48-port 4-Gbps switching module can operate in dedicated mode with a 1-Gbps operating speed. However, if you configure one or more ports to operate in 2-Gbps or 4-Gbps dedicated mode, some of the other ports in the module would have to operate in shared mode.

2 By default, all ports in a 24-port 4-Gbps switching module operate in shared mode with administrative operating speed set to auto. All ports in a 24-port 4-Gbps switching module can operate in dedicated mode with a 2-Gbps operating speed. However, if you configure one or more ports to operate in 4-Gbps dedicated mode, some of the other ports in the module would have to operate in shared mode



Note Port rate modes are supported on all Generation 2 modules and fabric switches. Port rate modes are not supported on the Cisco Fabric Switch for HP c-Class BladeSystem, and the Cisco Fabric Switch for IBM BladeCenter.


Dedicated Mode

When port rate mode is configured as dedicated, a port is allocated required fabric bandwidth and related resources to sustain line rate traffic at the maximum operating speed configured for the port. In this mode, ports do not use local buffering and all receive buffers are allocated from a global buffer pool (see the "Buffer Pools" section).

Table 14-4 show the amount of bandwidth reserved for a configured port speed on 4-Gbps switching modules.

Table 14-4 Bandwidth Reserved for the Port Speeds on 4-Gbps Switching Modules 

Configured Speed
Reserved Bandwidth

Auto

4 Gbps

4-Gbps

Auto with 2-Gbps maximum

2 Gbps

2-Gbps

1-Gbps

1 Gbps



Note 10-Gbps ports in auto mode only support auto speed mode at 10 Gbps.


Shared Mode

When port rate mode is configured as shared, multiple ports within a port group share data paths to the switch fabric so that fabric bandwidth and related resources are shared. Often, the available bandwidth to the switch fabric may be less than the negotiated operating speed of a port. Ports in this mode use local buffering for the BB_credit buffers.

All ports in switching modules where bandwidth is shared support 1-Gbps, 2-Gbps, or 4-Gbps traffic. However, it is possible to configure one or more ports in a port group to operate in dedicated mode with 1-Gbps, 2-Gbps or 4-Gbps operating speed.

Dynamic Bandwidth Management

On port switching modules where bandwidth is shared, the bandwidth available to each port within a port group can be configured based on the port rate mode and speed configurations. Within a port group, some ports can be configured in dedicated mode while others operate in shared mode.

Ports configured in dedicated mode are allocated the required bandwidth to sustain a line rate of traffic at the maximum configured operating speed, and ports configured in shared mode share the available remaining bandwidth within the port group. Fair allocation of bandwidth among a group of ports is determined, in part, by the rate mode and speed configurations. For example, if the set ports in a module are configured with the same rate mode and speed (such as 4 Gbps of shared bandwidth), then all the ports should have fair allocation of bandwidth and eventually, similar throughput. When you enable bandwidth fairness, you should notice a reduction in any disparity that may otherwise exist in similar configurations.

Bandwidth allocation among the shared mode ports is based on the operational speed of the ports. For example, if four ports operating at speeds 1 Gbps, 1 Gbps, 2 Gbps, and 4 Gbps share bandwidth of 8 Gbps, the ratio of allocation would be 1:1:2:4.


Note If dedicated ports are not using all of their allocated bandwidth, the unused bandwidth is made available for use by all ports configured for shared bandwidth mode.



Tip When migrating a host that supports up to 2-Gbps traffic (that is, not 4-Gbps with autosensing capabilities) to the 4-Gbps switching modules, use autosensing with a maximum bandwidth of 2-Gbps.



Note If you configure an interface for autosensing speed with a maximum bandwidth of 2 Gbps and want to change to the default of 4 Gbps, ensure that there are enough shared resources available to support the configuration on the module.


Out-of-Service Interfaces

On supported modules and fabric switches, you might need to allocate all the shared resources for one or more interfaces to another interface in the port group or module. You can take interfaces out of service to release shared resources that are needed for dedicated bandwidth. When an interface is taken out of service, all shared resources are released and made available to the other interface in the port group or module. These shared resources include bandwidth, rate mode, BB_credits, and extended BB_credits. All shared resource configurations are returned to their default values when the interface is brought back into service. Corresponding resources must be made available in order for the port to be successfully returned to service.


Caution If you need to bring an interface back into service, you might disrupt traffic if you need to release shared resources from other interfaces in the same port group.

Buffer Credit Allocation

This sections describe how buffer credits are allocated to switches and modules, and includes the following topics:

Buffer Pools

BB_Credit Buffers for Switching Modules

BB_Credit Buffers for Fabric Switches

Buffer Pools

In the architecture of Generation 2 modules, receive buffers shared by a set of ports are called buffer groups. The receive buffer groups are organized into global and local buffer pools.

The receive buffers allocated from the global buffer pool to be shared by a port group are called a global buffer pool. Global receive buffer pools include the following buffer groups:

Reserved internal buffers

Allocated BB_credit buffers for each Fibre Channel interface (user configured or assigned by default)

Common unallocated buffer pool for BB_credits, if any, to be used for additional BB_credits as needed

Performance buffers (only used on 12-port 4-Gbps and 4-port 10-Gbps switching modules)

Figure 14-1 shows the allocation of BB_credit buffers on linecards (24-port and 48-port line cards).


Note In some modules, performance buffers are not supported.


Figure 14-1 Receive Buffers for Fibre Channel Ports in a Global Buffer Pool

Figure 14-2 shows the default BB_credit buffer allocation model for 24-port 4-Gbps switching modules. The minimum BB_credits required to bring up a port is two buffers.

Figure 14-2 BB_Credit Buffer Allocation in 24-port 4-Gbps Switching Modules


Note The default BB_credit buffer allocation is the same for all port speeds.


BB_Credit Buffers for Switching Modules

This section describes how buffer credits are allocated to Cisco MDS 9000 switching modules, and includes the following topics:

48-port 4-Gbps Fibre Channel Module BB_Credit Buffers

24-port 4-Gbps Fibre Channel Module BB_Credit Buffers

18-Port Fibre Channel/4-Port GigabitEthernet Multiservice Module BB_Credit Buffers

Example Speed and Rate Configuration on a 24-Port 4-Gbps Switching Module

4-Port 10-Gbps Switching Module BB_Credit Buffers

48-port 4-Gbps Fibre Channel Module BB_Credit Buffers

Table 14-5 lists the BB_credit buffer allocation for 48-port 4-Gbps Fibre Channel switching modules.

Table 14-5 48-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Module
BB_Credit Buffers Per Port Defaults
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL1
Fx Port
ISL1
Fx Port

User configurable BB_credit buffers

6000

125

16

16

16

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 48-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 48-port 4-Gbps Fibre Channel switching module consists of 12 ports. The ports in shared rate mode have bandwidth oversubscription of 4:1 by default. However, some configurations of the shared ports in a port group can have maximum bandwidth oversubscription of 5:1 (considering that each port group has 12.8-Gbps bandwidth).

The following example configurations are supported by the 48-port 4-Gbps Fibre Channel switching modules:

Twelve ports with shared rate mode and 4-Gbps speed (4:1 oversubscription) (default)

One port with dedicated rate mode and 4-Gbps speed plus
11 ports with shared rate mode and 4-Gbps speed (5:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
11 ports with shared rate mode and 2-Gbps speed (2.5:1 oversubscription)

Two ports with dedicated rate mode and 2-Gbps speed plus
10 ports with shared rate mode and 4-Gbps speed (5:1 oversubscription) 

Two ports with dedicated rate mode and 2-Gbps speed plus
10 ports with shared rate mode and 2-Gbps speed (2.5:1 oversubscription) 

Twelve ports with dedicated rate mode and 1-Gbps speed

Three ports with dedicated rate mode and 4-Gbps speed plus
four ports with shared rate mode and 1-Gbps speed plus
five ports put out-of-service (see Figure 14-3)

Figure 14-3 Example Speed and Rate Configuration on a 48-Port 4-Gbps Switching Module

Six ports with dedicated rate mode and 2-Gbps speed plus
four ports with shared rate mode and 1-Gbps speed plus
two ports put out-of-service (see Figure 14-4)


Note For an example of the configuration of the this example, see "Configuring a 48-port 4-Gbps Fibre Channel Switching Module Example" section.


Figure 14-4 Example Speed and Rate Configuration on a 48-Port 4-Gbps Switching Module

24-port 4-Gbps Fibre Channel Module BB_Credit Buffers

Table 14-7 lists the BB_credit buffer allocation for 24-port 4-Gbps Fibre Channel switching modules.

Table 14-6 24 Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Module
BB_Credit Buffers Per Port Defaults
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL1
Fx Port
ISL1
Fx Port

User configurable BB_credit buffers

6000

250

16

16

16

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 24-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

Each port group on the 24-port 4-Gbps Fibre Channel switching module consists of six ports. The ports in shared rate mode have bandwidth oversubscription of 2:1 by default. However, some configurations of the shared ports in a port group can have maximum bandwidth oversubscription of 4:1 (considering that each port group has 12.8-Gbps bandwidth). The following example configurations are supported by the 24-port 4-Gbps Fibre Channel switching modules:

Six ports with shared rate mode and 4-Gbps speed (2:1 oversubscription) (default)

Two ports with dedicated rate mode and 4-Gbps speed plus
four ports with shared rate mode and 4-Gbps speed (with 4:1 oversubscription)

One port with dedicated rate mode and 4-Gbps speed plus
three ports with dedicated rate mode and 2-Gbps speed plus
two ports with shared rate mode and 4-Gbps speed (4:1 oversubscription)

Six ports with dedicated rate mode and 2-Gbps speed

Three ports with dedicated rate mode and 4-Gbps speed plus
three ports with shared rate mode and 1-Gbps speed (see Figure 14-5)


Note For an example of the configuration of the this example, see the "Configuring a 24-port 4-Gbps Fibre Channel Switching Module Example" section.


Figure 14-5 Example Speed and Rate Configuration on a 24-Port 4-Gbps Switching Module

18-Port Fibre Channel/4-Port GigabitEthernet Multiservice Module BB_Credit Buffers

Table 14-7 lists the BB_credit buffer allocation for 18-port 4-Gbps multiservice modules.

Table 14-7 18-Port 4-Gbps Multiservice Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Module
BB_Credit Buffers Per Port Defaults
Dedicated Rate Mode
4-Gbps Speed
Shared Rate Mode
4-Gbps Speed
ISL1
Fx Port
ISL1
Fx Port

User configurable BB_credit buffers

4509

250

16

16

16

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on18-port 4-Gbps Fibre Channel switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

BB_credit buffers for Fx port mode connections can be configured. The minimum is 2 buffers and the maximum of 250 buffers for dedicated rate mode or 16 buffers for shared rate mode.

Performance buffers are not supported on this module.

12-Port 4-Gbps Switching Module BB_Credit Buffers

Table 14-8 lists the BB_credit buffer allocation for 12-port 4-Gbps switching modules.

Table 14-8 12-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Module
BB_Credit Buffers Per Port Defaults
Dedicated Rate Mode
4-Gbps Speed
ISL 1
Fx Port

User configurable BB_credit buffers

5488

250

16

Performance buffers

512 (shared)

145

12

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 12-port 4-Gbps switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

By default, 512 performance buffers are preallocated and are shared by all the ports. These buffers are configurable and the buffers are assigned to the port based on the availability of the buffers in the shared pool.

There are 2488 extra buffers available as extended BB_credit buffers after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 - (250 * 12)).


Note Extended BB_credits are allocated across all ports on the switch. That is, they are not allocated by port group.



Note By default, the ports in the 12-port 4-Gbps switching modules come up in 4-Gbps dedicated rate mode but can be configured as 1-Gbps and 2-Gbps dedicated rate mode. Shared mode is not supported.


4-Port 10-Gbps Switching Module BB_Credit Buffers

Table 14-9 lists the BB_credit buffer allocation for 4-port 10-Gbps switching modules.

Table 14-9 4-port 10-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Module
BB_Credit Buffers Per Port Defaults
Dedicated Rate Mode
4-Gbps Speed
ISL1
F port2

User configurable BB_credit buffers

5488

250

16

Performance buffers

512 (shared)

145

12

1 ISL = E port or TE port.

2 Ports on the 4-port 10-Gbps cannot operate in FL port mode.



Note The ports in the 4-port 10-Gbps switching module only support 10-Gbps dedicated rate mode. FL port mode and shared rate mode are not supported.


The following considerations apply to BB_credit buffers on 4-port 10-Gbps switching modules:

BB_credit buffers for ISL connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

By default, 512 performance buffers are preallocated and are shared by all the ports. These buffers are configurable and the buffers are assigned to the port based on the availability of the buffers in the shared pool.

There are 4488 extra buffers available as extended BB_credits after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 - (250 * 4)).


Note Extended BB_credits are allocated across all ports on the switch. That is, they are not allocated by port group.


BB_Credit Buffers for Fabric Switches

This section describes how buffer credits are allocated to Cisco MDS 9000 Fabric switches, and includes the following topics:

Cisco MDS 9134 Fabric Switch BB_Credit Buffers

Cisco MDS 9124 Fabric Switch BB_Credit Buffers

Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers

Cisco MDS 9134 Fabric Switch BB_Credit Buffers

Table 14-10 lists the BB_credit buffer allocation for 32-port 4-Gbps Fibre Channel switches.

Table 14-10 32-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port Defaults
 
ISL1
Fx Port

User configurable BB_credit buffers

64

64

64

1 ISL = E port or TE port.


The following considerations apply to BB_credit buffers on 32-port 4-Gbps switches:

BB_credit buffers for connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers per port.

BB_credit buffers for Fx port mode connections can be configured from a minimum of 2 buffers to a maximum of 250 buffers.

Cisco MDS 9124 Fabric Switch BB_Credit Buffers

Table 14-11 lists the BB_credit buffer allocation for 24-port 4-Gbps Fibre Channel switches.

Table 14-11 24-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port Defaults
 
ISL1
Fx Port

User configurable BB_credit buffers

64

16

16

1 ISL = E port or TE port.


Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers

Table 14-12 lists the BB_credit buffer allocation for 18-port 4-Gbps Multiservice Modular switches.

Table 14-12 18-Port 4-Gbps Switching Module BB_Credit Buffer Allocation Defaults 

BB_Credit Buffer Allocation Type
BB_Credit Buffers Per Port Group
BB_Credit Buffers Per Port Defaults
 
ISL1
Fx Port

User configurable BB_credit buffers

4509

250

16

1 ISL = E port or TE port.


Extended BB_Credits


Note Extended BB_credits are not supported on the Cisco MDS 9124 Fabric Switch, Cisco MDS 9134 Fabric Switch, Cisco MDS 9222i Fabric Switch, the Cisco Fabric Switch for HP c-Class BladeSystem, and the Cisco Fabric Switch for IBM BladeCenter.


To facilitate BB_credits for long haul links, the extended BB_credits feature allows the user to configure the receive buffers above the maximum value on all Generation 2 switching modules (see the "Buffer Credit Allocation" section). When necessary, you can reduce the buffers on one port and assign them to another port, exceeding the default maximum. The minimum extended BB_credits per port is 256 and the maximum is 4095.

In general, the user can configure any port in a port group to dedicated mode. To do this, you must first release the buffers from the other ports before configuring larger extended BB_credits for a port.


Note The ENTERPRISE_PKG license is required to use extended BB_credits on Generation 2 switching modules. Also, extended BB_credits are not supported by ports in shared rate mode.

All ports on the Generation 2 switching modules support extended BB_credits. There are no limitations for how many extended BB_credits you can assign to a port (except for the maximum and minimum limits). If necessary, you can take interfaces out of service to make more extended BB_credits available to other ports.


About Combining Generation 1 and Generation 2 Switching Modules

You can combine Generation 1 and Generation 2 switching modules, with either Supervisor-1 modules or Supervisor-2 modules. However, combining switching modules and supervisor modules has the following port index limitations:

Supervisor-1 modules only support a maximum of 252 port indexes, regardless of the type of switching modules.

Supervisor-2 modules support a maximum of 1020 port indexes when all switching modules in the chassis are Generation 2.

Supervisor-2 modules only support a maximum of 252 port indexes when only Generation 1 switching modules, or a combination of Generation 1 and Generation 2 switching modules, are installed in the chassis.

Port Indexes

Cisco MDS 9000 switches allocate index identifiers for the ports on the modules. These port indexes cannot be configured. You can combine Generation 1 and Generation 2 switching modules, with either Supervisor-1 modules or Supervisor-2 modules. However, combining switching modules and supervisor modules has the following port index limitations:

Supervisor-1 modules only support a maximum of 252 port indexes, regardless of the type of switching modules.

Supervisor-2 modules support a maximum of 1020 port indexes when all switching modules in the chassis are Generation 2.

Supervisor-2 modules only support a maximum of 252 port indexes when only Generation 1 switching modules, or a combination of Generation 1 and Generation 2 switching modules, are installed in the chassis.


Note On a switch with the maximum limit of 252 port index maximum limit, any new module that exceeds the limit when installed does not power up.


You can use the show port index-allocation command to display the allocation of port indexes on the switch.

switch# show port index-allocation

Module index distribution:
------------------------------------------------------+
Slot | Allowed |      Alloted indices info            |
     |  range  | Total |          Index values        |
-----|---------|-------|------------------------------|
 1   |   0- 255|   16  | 32-47                        |
 2   |   0- 255|   12  | 0-11                         |
 3   |   0- 255|   -   | (None)                       |
 4   |   0- 255|   -   | (None)                       |
 7   |   0- 255|   -   | (None)                       |
 8   |   0- 255|   -   | (None)                       |
 9   |   0- 255|   -   | (None)                       |
SUP  |  -----  |    3  | 253-255                      |

Generation 1 switching modules have specific numbering requirements. If these requirements are not met, the module does not power up. The port index numbering requirements include the following:

If port indexes in the range of 256 to 1020 are assign to operational ports, Generation 1 switching modules do not power up.

A block of contiguous port indexes is available. If such a block of port indexes is not available, Generation 1 modules do not power up. Table 14-13 shows the port index requirements for the Generation 1 modules.


Note If the switch has Supervisor-1 modules, the block of 32 contiguous port indexes must begin on the slot boundary. The slot boundary for slot 1 is 0, for slot 2 is 32, and so on. For Supervisor-2 modules, the contiguous block can start anywhere.


Table 14-13 Port Index Requirements for Generation 1 Modules 

Generation 1 Module
Number of Port Indexes Required
Supervisor-1 Module
Supervisor-2 Module

16-port 2-Gbps Fibre Channel module

16

16

32-port 2-Gbps Fibre Channel module

32

32

8-port Gigabit Ethernet IP Storage Services module

32

32

4-port Gigabit Ethernet IP Storage Services module

32

16

32-port 2-Gbps Fibre Channel Storage Services Module (SSM).

32

32

14-port Fibre Channel/2-port Gigabit Ethernet Multiprotocol Services (MPS-14/2) module.

32

22


The allowed mix of Generation 1 and Generation 2 switching modules in a chassis is determined at run-time, either when booting up the switch or when installing the modules. In some cases, the sequence in which switching modules are inserted into the chassis determines if one or more modules is powered up. When a module does not power up because of a resource limitation, you can display the reason using show module command.

switch# show module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    16     1/2 Gbps FC Module               DS-X9016           ok
2    12     1/2/4 Gbps FC Module                                powered-dn
5    0      Supervisor/Fabric-2              DS-X9530-SF2-K9    active *

Mod  Power-Status  Power Down Reason
---  ------------  ---------------------------
2    powered-dn    Insufficient resources (dest Index)
* this terminal session

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-0b-be-f7-4c-24 to 00-0b-be-f7-4c-28  JAB07030723
2    00-05-30-01-a8-b2 to 00-05-30-01-a8-b6  JAB090401AA
5    00-05-30-01-aa-7e to 00-05-30-01-aa-82  JAB091100TF

* this terminal session

The running configuration is updated when modules are installed. If you save the running configuration to the startup configuration (using the copy running-config startup-config command), during reboot the switch powers up the same set of modules as before the reboot regardless of the sequence in which the modules initialize. You can use the show port index-allocation startup command to display the index allocation the switch uses at startup.

switch# show port index-allocation startup

Startup module index distribution:
------------------------------------------------------+
Slot | Allowed |      Alloted indices info            |
     |  range  | Total |          Index values        |
-----|---------|-------|------------------------------|
 1   |  -----  |   34  | 0-31,80-81                   |
 2   |  -----  |   32  | 32-63                        |
 3   |  -----  |   16  | 64-79                        |(Slot 1 shares 80-81)  
 4   |  -----  |   48  | 96-127,224-239               |
SUP  | 253-255 |   3   | 253-255                      |


Note The output of the show port index-allocation startup command does not display anything in the "Allowed range" column because the command extracts the indices from the persistent storage service (PSS) and displaying an allowed range for startup indices is meaningless.


If a module fails to power up, you can use the show module slot recovery-steps command to display the reason. For information on recovering a module powered-down because port indexes are not available, refer to the Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x.


Tip Whenever using mixed Generation 1 and Generation 2 modules, power up the Generation 1 modules first. During a reboot of the entire switch, the Generation 1 modules power up first (default behavior).


PortChannels

PortChannels have the following restrictions:

The maximum number of PortChannels allowed is 256 if all switching modules are Generation 2.

The maximum number of PortChannels allowed is 128 if the switching modules are Generation 1 or both Generation 1 and Generation 2.

You must reserve the resources on the Generation 2 switching module interfaces to be used in the PortChannel.


Note The number of PortChannels allowed does not depend on the type of supervisor module.


When configuring PortChannels on switches with both Generation 1 and Generation 2 switching modules, configure the PortChannel and Generation 2 switching modules interfaces to auto with a maximum of 2 Gbps or configure the Generation 1 switching modules followed by the Generation 2 switching modules.


Note Generation 1 switching module interfaces do not support auto speed with max 2Gbps. Also, Generation 2 switching module interfaces cannot be forcefully added to a PortChannel if sufficient resources are not available.



Note Before adding a Generation 2 interface to a PortChannel, use the show port-resources module command to check for resource availability.


Table 14-14 describes the results of adding a member to a PortChannel for various configurations.

Table 14-14 PortChannel Configuration and Addition Results 

PortChannel Members
Configured Speed
New Member Type
Addition Type
Result
PortChannel
New Member

No members

Any

Any

Generation 1 or Generation 2

Force

Pass

Auto

Auto

Generation 1 or Generation 2

Normal or force

Pass

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2

Normal

Fail

Force

Pass

Auto

Auto max 2000

Generation 2

Normal

Fail

Force

Pass or fail1

Generation 1 interfaces

Auto

Auto

Generation 2

Normal

Fail

Force

Pass

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2

Normal

Fail

Force

Pass or fail1

Generation 2 interfaces

Auto

Auto

Generation 1

Normal or force

Fail

Auto max 2000

Auto

Generation 1

Normal or force

Pass

Auto max 2000

Auto

Generation 2

Normal

Fail

Force

Pass

Auto

Auto max 2000

Generation 2

Normal

Fail

Force

Pass

1 Is resources not available.


Use the show port-channel compatibility parameters command to obtain information about PortChannel addition errors.

Configuring Generation 2 Module Interface Shared Resources

This section describes how to configure Generation 2 module interface shared resources and contains the following sections:

Displaying Interface Capabilities

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

Configuring Port Speed

Configuring Rate Mode

Configuring Oversubscription Ratio Restrictions

Configuring Bandwidth Fairness

Taking Interfaces Out of Service

Releasing Shared Resources in a Port Group

Enabling the Buffer-to-Buffer State Change Number

Displaying Interface Capabilities

Before configuring a Generation 2 interface, you can use the show interface capabilities command to display detailed information about the capabilities of the interface.

switch# show interface fc 9/1 capabilities
 Min Speed is 1 Gbps
 Max Speed is 4 Gbps
 FC-PH Version (high, low)                         (0,6)
 Receive data field size (max/min)                 (2112/256)  bytes
 Transmit data field size (max/min)                (2112/128)  bytes
 Classes of Service supported are                  Class 2, Class 3, Class F
 Class 2 sequential delivery                       supported
 Class 3 sequential delivery                       supported
 Hold time (max/min)                               (100/1) micro sec
 BB state change notification                      supported
 Maximum BB state change notifications             14
 Rate Mode change                                  supported

 Rate Mode Capabilities                            Shared        Dedicated
  Receive BB Credit modification supported            yes              yes
  FX mode Receive BB Credit (min/max/default)     (1/16/16)      (1/250/16)
  ISL mode Receive BB Credit (min/max/default)         --        (2/250/250)
  Performace buffer modification supported             no               no

 Out of Service capable                             yes
 Beacon mode configurable                           yes

Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules

The 48-port and 24-port 4-Gbps Fibre Channel switching modules support the following features:

1-Gbps, 2-Gbps, and 4-Gbps speed traffic

Shared and dedicated rate mode

ISL (E or TE) and Fx (F or FL) port modes

Extended BB_credits

Migrating from Shared Mode to Dedicated Mode

To configure 48-port and 24-port 4-Gbps Fibre Channel switching modules when starting with the default configuration or when migrating from shared rate mode to dedicated rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the traffic speed to use (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps).

See the "Configuring Port Speed" section.

3. Configure the rate mode (dedicated or shared) to use.

See the "Configuring Rate Mode" section.

4. Configure the port mode.

See the "About Interface Modes" section on page 12-3.


Note ISL ports cannot operate in shared rate mode.


5. Configure the BB_credits and extended BB_credits, as necessary.

See the "About Buffer-to-Buffer Credits" section on page 12-33 and the "About Extended BB_credits" section on page 12-35.

Migrating from Dedicated Mode to Shared Mode

To configure 48-port and 24-port 4-Gbps Fibre Channel switching modules migrating from dedicated rate mode to shared rate mode, follow these guidelines:

1. Take unused interfaces out of service to release resources for other interfaces, if necessary.

See the "Taking Interfaces Out of Service" section.

2. Configure the BB_credits and extended BB_credits, as necessary.

See the "About Buffer-to-Buffer Credits" section on page 12-33 and the "About Extended BB_credits" section on page 12-35.

3. Configure the port mode.

See the "About Interface Modes" section on page 12-3.


Note ISL ports cannot operate in shared rate mode.


4. Configure the rate mode (dedicated or shared) to use.

See the "Configuring Rate Mode" section.

5. Configure the traffic speed (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps) to use.

See the "Configuring Port Speed" section.

Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces

The 12-port 4-Gbps switching modules support the following features:

1-Gbps, 2-Gbps, and 4-Gbps speed traffic

Only dedicated rate mode

ISL (E or TE) and Fx (F or FL) port modes

Extended BB_credits

Performance buffers

To configure 4-port 10-Gbps switching modules when starting with the default configuration, follow these guidelines:

1. Configure the traffic speed (1 Gbps, 2 Gbps, 4 Gbps, or autosensing with a maximum of 2 Gbps or 4 Gbps) to use.

See the "Configuring Port Speed" section.

2. Configure the port mode.

See the "About Interface Modes" section on page 12-3.

3. Configure the BB_credits, performance buffers, and extended BB_credits, as necessary.

See the "About Buffer-to-Buffer Credits" section on page 12-33 and the "About Extended BB_credits" section on page 12-35.


Note If you change the port bandwidth reservation parameters on a 48-port or 24-port module, the change affects only the changed port. No other ports in the port group are affected.


Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces

The 4-port 10-Gbps switching modules support the following features:

Only 10-Gbps speed traffic

Only dedicated rate mode

ISL (E or TE) and F port modes

Extended BB_credits

Performance buffers

Use the following guidelines to configure 4-port 10-Gbps switching modules when starting with the default configuration:

1. Configure the port mode.

See the "About Interface Modes" section on page 12-3.

2. Configure the BB_credits, performance buffers, and extended BB_credits, as necessary.

See the "About Buffer-to-Buffer Credits" section on page 12-33 and the "About Extended BB_credits" section on page 12-35.

Configuring Port Speed

The port speed on an interface, combined with the rate mode, determines the amount of shared resources available to the ports in the port group on a 48-port or 24-port 4-Gbps Fibre Channel switching module. Especially in the case of dedicated rate mode, the port group resources are reserved even though the bandwidth is not used. For example, if an interface is configured for autosensing (auto) and dedicated rate mode, then 4 Gbps of bandwidth is reserved even though the maximum operating speed is 2 Gbps. For the same interface, if autosensing with a maximum speed of 2 Gbps (auto max 2000) is configured, then only 2 Gbps of bandwidth is reserved and the unused 2 Gbps is shared with the other interface in the port group.


Caution Changing port speed and rate mode disrupts traffic on the port. Traffic on other ports in the port group is not affected.


Note The 4-port 10-Gbps switching module supports 10-Gbps traffic only.


To configure the port speed on an interface on a 4-Gbps switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc 1/1

switch(config-if)#

Selects the interface and enters interface configuration submode.

Step 3 

switch(config-if)# switchport speed 4000

Configures the port speed in megabits per second. Valid values are 1000, 2000, 4000 and auto. The auto parameter enables autosensing on the interface.

switch(config-if)# switchport speed auto

Configures autosensing for the interface with
4 Gbps of bandwidth reserved.

switch(config-if)# switchport speed auto max 2000

Configures autosensing with a maximum of
2 Gbps of bandwidth reserved.

switch(config-if)# no switchport speed

Reverts to the default speed for the interface (auto).

Use the show interface command to verify the port speed configuration for an interface on a 24-port or 48-port 4-Gbps Fibre Channel switching module.

switch# show interface fc 9/1
fc9/1 is up
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 22:01:00:05:30:01:9f:02
    Admin port mode is F
    snmp traps are enabled
    Port mode is F, FCID is 0xeb0002
    Port vsan is 1
    Speed is 2 Gbps
    Rate mode is shared
    Transmit B2B Credit is 64
    Receive B2B Credit is 16
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      226 frames input, 18276 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      326 frames output, 21364 bytes
        0 discards, 0 errors
      0 input OLS, 0 LRR, 1 NOS, 0 loop inits
      3 output OLS, 2 LRR, 0 NOS, 0 loop inits
      16 receive B2B credit remaining
      64 transmit B2B credit remaining

Configuring Rate Mode

To configure the rate mode (dedicated or shared) on an interface on a 48-port or 24-port 4-Gbps Fibre Channel switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc 1/1

switch(config-if)#

Selects the interface and enters interface configuration submode.

Step 3 

switch(config-if)# switchport rate-mode dedicated

Reserves dedicated bandwidth for the interface.

Note If you cannot reserve dedicated bandwidth on an interface, you might have exceeded the port group maximum bandwidth. Use the show port-resources command to determine what resources are already allocated.

switch(config-if)# switchport rate-mode shared

Reserves shared (default) bandwidth for the interface.

switch(config-if)# no switchport rate-mode

Reverts to the default state (shared).


Caution Changing port speed and rate mode disrupts traffic on the port.

Use show port-resources module command to verify the rate mode configuration for interfaces on a 48-port or 24-port 4-Gbps Fibre Channel switching module.

switch# show port-resources module 9
Module 9
Available dedicated buffers are 5400

 Port-Group 1
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc9/1                                      16        4.0  shared
  fc9/2                                      16        4.0  shared
  fc9/3                                      16        4.0  shared
  fc9/4                                      16        4.0  shared
  fc9/5                                      16        4.0  shared
  fc9/6                                      16        4.0  shared

 Port-Group 2
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc9/7                                      16        4.0  shared
  fc9/8                                      16        4.0  shared
  fc9/9                                      16        4.0  shared
  fc9/10                                     16        4.0  shared
  fc9/11                                     16        4.0  shared
  fc9/12                                     16        4.0  shared

 Port-Group 3
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc9/13                                     16        4.0  shared
  fc9/14                                     16        4.0  shared
  fc9/15                                     16        4.0  shared
  fc9/16                                     16        4.0  shared
  fc9/17                                     16        4.0  shared
  fc9/18                                     16        4.0  shared

 Port-Group 4
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc9/19                                     16        4.0  shared
  fc9/20                                     16        4.0  shared
  fc9/21                                     16        4.0  shared
  fc9/22                                     16        4.0  shared
  fc9/23                                     16        4.0  shared
  fc9/24                                     16        4.0  shared

Configuring Oversubscription Ratio Restrictions

The 48-port and 24-port 4-Gbps Fibre Channel switching modules support oversubscription on switches with shared rate mode configurations. Table 14-15 describes the bandwidth allocation for oversubscribed interfaces configured in shared mode.

Table 14-15 Bandwidth Allocation for Oversubscribed Interfaces 

Switching Module Type
Configured Speed
Reserved Bandwidth (Gbps)
Maximum Bandwidth (Gbps)
Ratios enabled
Ratios disabled

48 ports

Auto 4 Gbps

0.8

0.09

4

Auto (maximum is 2 Gbps) 2 Gbps

0.4

0.045

2

1 Gbps

0.2

0.0225

1

24 ports

Auto 4 Gbps

1

0.27

4

Auto (maximum is 2 Gbps) 2 Gbps

0.5

0.135

2

1 Gbps

0.25

0.067

1


By default, all 48-port and 24-port 4-Gbps Fibre Channel switching modules have restrictions on oversubscription ratios enabled.

As of Cisco SAN-OS Release 3.1(1) and later, you can disable restrictions on oversubscription ratios. All ports in 48-port and 24-port modules can be configured to operate at 4 Gbps in shared mode-even if other ports in the port group are configured in dedicated mode-regardless of available bandwidth. However, when oversubscription ratio restrictions are enabled you may not have all shared ports operating at 4 Gbps. For example, oversubscription ratios are enabled, and you have configured three 4 Gbps dedicated ports in one port group, no other ports in the same port group can be configured to operate at 4 Gbps.

switch# show port-resources module 8
Module 8
  Available dedicated buffers are 5478
 Port-Group 1
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 0.8 Gbps
  Allocated dedicated bandwidth is 12.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc8/1                                      16        4.0  dedicated
  fc8/2                                      16        4.0  dedicated
  fc8/3                                      16        4.0  dedicated
  fc8/4 (out-of-service)
  fc8/5 (out-of-service)
  fc8/6 (out-of-service)

For dedicated ports, oversubscription ratio restrictions do not apply to the shared pool in port groups. So if oversubscription ratio restrictions are disabled, and you've configured three 4 Gbps dedicated ports in one port group, then you can configure all other ports in the same port group to operate at a shared rate of 4 Gbps. In the following example, a 24-port module has a group of 6 ports-3 dedicated ports are operating at 4 Gbps, and 3 shared ports operating at 4 Gbps:

switch# show port-resources module 8
Module 8
  Available dedicated buffers are 5382

Port-Group 1
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 0.8 Gbps
  Allocated dedicated bandwidth is 12.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode 
                                        Buffers     (Gbps)            
  --------------------------------------------------------------------
  fc8/1                                      16        4.0  dedicated                    
  fc8/2                                      16        4.0  dedicated 
  fc8/3                                      16        4.0  dedicated 
  fc8/4                                      16        4.0  shared    
  fc8/5                                      16        4.0  shared    
  fc8/6                                      16        4.0  shared    
 Port-Group 2
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 0.8 Gbps
  Allocated dedicated bandwidth is 12.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode 
                                        Buffers     (Gbps)            
  --------------------------------------------------------------------
  fc8/7                                      16        4.0  dedicated 
  fc8/8                                      16        4.0  dedicated 
  fc8/9                                      16        4.0  dedicated 
  fc8/10                                     16        4.0  shared    
  fc8/11                                     16        4.0  shared    
  fc8/12                                     16        4.0  shared 
...

When disabling restrictions on oversubscription ratios, all ports in shared mode on 48-port and 24-port 4-Gbps Fibre Channel switching modules must be shut down. When applying restrictions on oversubscription ratios, you must take shared ports out of service.


Note When restrictions on oversubscription ratios are disabled, the bandwidth allocation among the shared ports is proportionate to the configured speed. (If the configured speed is auto, then bandwidth is allocated assuming a speed of 4 Gbps.) For example, if you have three shared ports configured at 1, 2, and 4 Gbps, then the allocated bandwidth ratio is 1:2:4. In Cisco SAN-OS Release 3.0 and later (or when restrictions on oversubscription ratios are enabled), port bandwidths are allocated in equal proportions, regardless of port speed, so, the bandwidth allocation for the same three ports mentioned in the example would be 1:1:1.


Disabling Restrictions on Oversubscription Ratios

Before disabling restrictions on oversubscription ratios, ensure that you have explicitly shut down shared ports. To disable restrictions on oversubscription ratios on a 48-port or 24-port 4-Gbps Fibre Channel switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# no rate-mode oversubscription-limit module 1

Disables restrictions on oversubscription ratios for a module.

Note You must enter this command separately for each module for which you want to remove the restrictions.

Step 3 

switch(config)# exit

Exits configuration mode.

Step 4 

switch# copy running-config startup-config

Saves the new oversubscription ratio configuration to the startup configuration, and then the new configuration is enforced upon subsequent reboots of the module.

Use the show running-config command to view oversubscription ratios for a module. If oversubscription ratios are enabled, then no restriction appears in the output.

Example 14-1 Module with Restrictions on Oversubscription Ratios Disabled

switch# show running-config
version 3.1(1)
...
no rate-mode oversubscription-limit module 2 
interface fc2/1
  switchport speed 2000
interface fc2/1
...

Oversubscription Ratio Restrictions Example

To disable restrictions on oversubscription ratios for ports on a 48-port Gen2 switch that is configured with both shared and dedicated ports, follow these steps:


Step 1 To disable restrictions on oversubscription ratios, you must shut down any shared ports. Use the show port-resources command to view the configuration on a module and to identify shared ports.

switch# show port-resources module 2
Module 2
Available dedicated buffers are 4656

 Port-Group 1
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc2/1                                      16        4.0  shared
  fc2/2                                      16        4.0  shared
  fc2/3                                      16        4.0  dedicated
  fc2/4                                      16        4.0  shared
  fc2/5                                      16        4.0  shared
  fc2/6                                      16        4.0  dedicated
  fc2/7                                      16        4.0  dedicated
  fc2/8                                      16        4.0  shared
  fc2/9                                      16        4.0  shared
  fc2/10                                     16        4.0  shared
  fc2/11                                     16        4.0  shared
  fc2/12                                     16        4.0  shared
  ...

 Port-Group 4
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc2/37                                     16        4.0  shared
  fc2/38                                     16        4.0  shared
  fc2/39                                     16        4.0  dedicated
  fc2/40                                     16        4.0  dedicated
  fc2/41                                     16        4.0  dedicated
  fc2/42                                     16        4.0  shared
  fc2/43                                     16        4.0  shared
  fc2/44                                     16        4.0  shared
  fc2/45                                     16        4.0  shared
  fc2/46                                     16        4.0  shared
  fc2/47                                     16        4.0  shared
  fc2/48                                     16        4.0  shared

Step 2 Shut down all shared ports for which you want to remove restrictions on oversubscription ratios.

switch (config)# interface fc2/1-2, fc2/4-5, fc2/8-38, fc2/43-48
switch (config-if)# shutdown

Step 3 Display the interface status to confirm the shutdown of all shared ports.

switch(config-if)# end
switch# show interface brief

-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port
                  Mode   Trunk                          Mode  Speed  Channel
                         Mode                                 (Gbps)
-------------------------------------------------------------------------------
fc2/1      1      FX     --      down             swl    --           --
fc2/2      1      FX     --      down             swl    --           --
fc2/3      1      T      --      up               swl    --           --
fc2/4      1      FX     --      down             swl    --           --
fc2/5      1      FX     --      down             swl    --           --
fc2/6      1      TE     --      up               swl    --           --
fc2/7      1      TE     --      up               swl    --           --
fc2/8      1      FX     --      down             swl    --           --
...
fc2/48     1      FX     --      down             sw1    --           --

Step 4 Disable restrictions on oversubscription ratios for the ports.

switch# config t
Enter configuration commands, one per line.  End with CNTL/Z.
switch(config)# no rate-mode oversubscription-limit module 2

Step 5 Bring up the ports that you shut down in step 2, and display their status to confirm that they are no longer shut down.

switch(config)# interface fc2/1-2, fc2/4-5, fc2/8-38, fc2/43-48
switch(config-if)# no shutdown
switch(config-if)# end
switch# show interface brief

-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port
                  Mode   Trunk                          Mode  Speed  Channel
                         Mode                                 (Gbps)
-------------------------------------------------------------------------------
fc2/1      1      FX     --      up               swl    --           --
fc2/2      1      FX     --      up               swl    --           --
fc2/3      1      T      --      up               swl    --           --
fc2/4      1      FX     --      up               swl    --           --
fc2/5      1      FX     --      up               swl    --           --
fc2/6      1      TE     --      up               swl    --           --
fc2/7      1      TE     --      up               swl    --           --
fc2/8      1      FX     --      up               swl    --           --
...
fc2/48     1      FX     --      up               sw1    --           --

Step 6 Confirm that the ports are now operating with no restrictions on oversubscription ratios.

switch# show running-config | include oversubscription-limit
no rate-mode oversubscription-limit module 2 <---indicates no restrictions on 
                                                 oversubscrption ratios

Step 7 Save the new oversubscription ratio configuration to the startup configuration.

switch# copy running-config startup-config

Enabling Restrictions on Oversubscription Ratios


Caution You must enable restrictions on oversubscription ratios before you can downgrade modules to a previous release.

Before enabling restrictions on oversubscription ratios, ensure that you have explicitly configured shared ports to out-of-service mode. To enable restrictions on oversubscription ratios on a 48-port or 24-port 4-Gbps Fibre Channel switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc2/1-2, fc2/4-5, fc2/8-38, fc2/43-48

Specifies the port interfaces for which you want to enable restrictions on oversubscription ratios.

Step 3 

switch(config-if)# shutdown

Shuts down shared ports.

Step 4 

switch(config-if)# out-of-service

Takes shared ports out of service.

Step 5 

switch# rate-mode oversubscription-limit module 1

Enables restrictions on oversubscription ratios for the module.

Note You must enter this command separately for each module for which you want to add the restriction.

Step 6 

switch# config t

switch(config)# interface fc2/1-2, fc2/4-5, fc2/8-38, fc2/43-48

switch(config-if)# no out-of-service

switch(config-if)# no shutdown

Returns all shared ports to service.

Step 7 

switch(config)# exit

Exits configuration mode.

Step 8 

switch# copy running-config startup-config

Saves the new oversubscription ratio configuration to the startup configuration, and then the new configuration is enforced upon subsequent reboots of the module.

Configuring Bandwidth Fairness

As of Cisco SAN-OS Release 3.1(2) and later, all 48-port and 24-port 4-Gbps Fibre Channel switching modules, as well as 18-port Fibre Channel/4-port GigabitEthernet Multiservice modules, have bandwidth fairness enabled by default, which improves fairness of bandwidth allocation among all ports and provides better throughput average to individual data streams. Bandwidth fairness can be configured per module.


Caution When you disable or enable bandwidth fairness, the change does not take effect until you reload the module.

Use the show module bandwidth-fairness command to check whether ports in a module are operating with bandwidth fairness enabled or disabled.

switch# show module 2 bandwidth-fairness
Module 2 bandwidth-fairness is enabled

Note This feature is only supported onthe 48-port and 24-port 4-Gbps Fibre Channel switching modules, as well as the 18-port Fibre Channel/4-port GigabitEthernet Multiservice module.


Enabling Bandwidth Fairness

To enable bandwidth fairness on switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# rate-mode bandwidth-fairness module 1

Enables bandwidth fairness for a module.

Note You must enter this command separately for each module for which you want to enable bandwidth fairness. You must reload the module for the command to take effect.

Step 3 

switch(config)# exit

Exits configuration mode.

Disabling Bandwidth Fairness


Note If you disable bandwidth fairness, up to a 20 percent increase in internal bandwidth allocation is possible for each port group; however, bandwidth fairness is not guaranteed when there is a mix of shared and full-rate ports in the same port group.


To disable bandwidth fairness on a switching module, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# no rate-mode bandwidth-fairness module 1

Disables bandwidth fairness for a module.

Note You must enter this command separately for each module for which you want to disable bandwidth fairness. You must reload the module for the command to take effect.

Step 3 

switch(config)# exit

Exits configuration mode.

Upgrade or Downgrade Scenario

When you are upgrading from a release earlier than Cisco SAN-OS Release 3.1(2), all modules operate with bandwidth fairness disabled until the next module reload. After the upgrade, any new module that is inserted has bandwidth fairness enabled.

When you are downgrading to a release earlier than Cisco SAN-OS Release 3.1(2), all modules keep operating in the same bandwidth fairness configuration prior to the downgrade. After the downgrade, any new module that is inserted has bandwidth fairness disabled.


Note After the downgrade, any insertion of a module or module reload will have bandwidth fairness disabled.


Taking Interfaces Out of Service

You can take interfaces out of service on Generation 2 switching modules. When an interface is out of service, all the shared resources for the interface are released as well as the configuration associated with those resources.


Note The interface must be disabled using a shutdown command before it can be taken out of service.



Caution Taking interfaces out of service releases all the shared resources to ensure that they are available to other interfaces. This causes the configuration in the shared resources to revert to default when the interface is brought back into service. Also, an interface cannot come back into service unless the default shared resources for the port are available. The operation to free up shared resources from another port is disruptive.


Note The interface cannot be a member of a PortChannel.


To take an interface out of service, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc 1/1

switch(config-if)#

Selects the interface and enters interface configuration submode.

Step 3 

switch(config-if)# no channel-group

Removes the interface from a PortChannel.

Step 4 

switch(config-if)# shutdown

Disables the interface.

Step 5 

switch(config-if)# out-of-service

Putting an interface into out-of-service will cause its shared resource configuration to revert to default

Do you wish to continue(y/n)? [n] y

Takes the interface out of service.

Use the show port-resources module command to verify the out-of-service configuration for interfaces on a Generation 2 switching module.

switch# show port-resources module 9
Module 9
Available dedicated buffers are 5429

 Port-Group 1
  Total bandwidth is 12.8 Gbps
  Total shared bandwidth is 12.8 Gbps
  Allocated dedicated bandwidth is 0.0 Gbps
  --------------------------------------------------------------------
  Interfaces in the Port-Group       B2B Credit  Bandwidth  Rate Mode
                                        Buffers     (Gbps)
  --------------------------------------------------------------------
  fc9/1                                      16        4.0  shared
  fc9/2 (out-of-service)
  fc9/3                                      16        4.0  shared
  fc9/4                                      16        4.0  shared
  fc9/5                                      16        4.0  shared
  fc9/6                                      16        4.0  shared
...

Releasing Shared Resources in a Port Group

When you want to reconfigure the interfaces in a port group on a Generation 2 module, you can return the port group to the default configuration to avoid problems with allocating shared resources.


Note The interface cannot be a member of a PortChannel.



Caution Releasing shared resources disrupts traffic on the port. Traffic on other ports in the port group is not affected.

To release the shared resources for a port group, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc 1/1

switch(config-if)#

Selects the interface and enters interface configuration submode.

Tip You can use an interface range to release the resources for all interfaces in a port group.

Step 3 

switch(config-if)# no channel-group

Removes the interface from a PortChannel.

Step 4 

switch(config-if)# shutdown

Disables the interface.

Step 5 

switch(config-if)# out-of-service

Putting an interface into out-of-service will cause its shared resource configuration to revert to default

Do you wish to continue(y/n)? [n] y

Takes the interface out of service.

Step 6 

switch(config-if)# no out-of-service

Makes the interface available for service. Repeat Step 2 through Step 6 for all the interfaces in the port group.

Enabling the Buffer-to-Buffer State Change Number

The BB_SC_N field (word 1, bits 15-12) specifies the buffer-to-buffer state change (BB_SC) number. The BB_SC_N field indicates that the sender of the port login (PLOGI) or fabric login (FLOGI) frame is requesting twice the number of frames specified by BB_SC_N to be sent between two consecutive BB_SC send primitives, and twice the number of R_RDY primitives to be sent between two consecutive BB_SC receive primitives.

To use the BB_SC_N field during PLOGI or FLOGI, follow these steps:

 
Command
Purpose

Step 1 

switch# config t

switch(config)#

Enters configuration mode.

Step 2 

switch(config)# interface fc 1/1

switch(config-if)#

Selects the interface and enters interface configuration submode.

Step 3 

switch(config-if)# switchport fcbbscn

Enables the use of buffer-to-buffer state change number for PLOGIs and FLOGIs on the interface.

switch(config-if)# no switchport fcbbscn

Disables (default) the use of buffer-to-buffer state change number for PLOGIs and FLOGIs on the interface.

Disabling ACL Adjacency Sharing for System Image Downgrade

As of Cisco MDS SAN-OS Release 3.0(3), Fibre Channel ACL adjacency sharing is enabled by default on the switches with an active Generation 2 switching module. Fibre Channel ACL adjacency sharing improves the performance for zoning and inter-VSAN routing (IVR) network address translation (NAT). To prevent disruptions when downgrading the system image on your switch to a release prior to Cisco SAN-OS Release 3.0(3), issue the following command in EXEC mode:

switch# system no acl-adjacency-sharing

To reenable Fibre Channel ACL adjacency sharing on your switch, issue the following command in EXEC mode:

switch# system acl-adjacency-sharing

Displaying SFP Diagnostic Information

You can use the show interface transceiver command to display small form-factor pluggable (SFP) diagnostic information for Generation 2 switching modules.

switch# show interface transceiver
...
fc12/12 sfp is present
    name is CISCO-FINISAR
    part number is FTRJ-8519-7D2CS1
    revision is A
    serial number is H11TVQB
    fc-transmitter type is short wave laser w/o OFC (SN)
    fc-transmitter supports intermediate distance link length
    media type is multi-mode, 62.5m (M6)
    Supported speed is 200 MBytes/sec
    Nominal bit rate is 2100 MBits/sec
    Link length supported for 50/125mm fiber is 300 m(s)
    Link length supported for 62.5/125mm fiber is 150 m(s)
    cisco extended id is unknown (0x0)

    no tx fault, rx loss, no sync exists, Diag mon type 104
    SFP Diagnostics Information
        Temperature      :  24.33 Celsius
        Voltage          :   3.33 Volt
        Current          :   0.04 mA       --
        Optical Tx Power :  N/A   dBm      --
        Optical Rx Power :  N/A   dBm      -
    Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
...

Example Configurations

This section describes example configurations and includes the following sections:

Configuring a 24-port 4-Gbps Fibre Channel Switching Module Example

Configuring a 48-port 4-Gbps Fibre Channel Switching Module Example

Configuring a 24-port 4-Gbps Fibre Channel Switching Module Example

This section describes how to configure the example shown in Figure 14-5.


Step 1 Select interfaces fc 3/1 through fc 3/3.

switch# config t
switch(config)# interface fc 3/1 - 3

Step 2 Configure the port speed, rate mode, and port mode on the interfaces.

switch(config-if)# switchport speed 4000
switch(config-if)# switchport rate-mode dedicated
switch(config-if)# switchport mode e

Step 3 Enable the interfaces and return to configuration mode.

switch(config-if)# no shutdown
switch(config-if)# exit
switch#

Step 4 Select the interfaces fc 3/4 through fc 3/6.

switch# config t
switch(config)# interface fc 3/4 - 6

Step 5 Configure the port speed, rate mode, and port mode on the interfaces.

switch(config-if)# switchport speed 1000
switch(config-if)# switchport rate-mode shared
switch(config-if)# switchport mode f

Step 6 Enable the interfaces and return to configuration mode.

switch(config-if)# no shutdown
switch(config-if)# exit
switch#


Configuring a 48-port 4-Gbps Fibre Channel Switching Module Example

This section describes how to configure the example shown in Figure 14-4.


Step 1 Select interfaces fc 4/11 through fc 4/12.

switch# config t
switch(config)# interface fc 4/11 - 12

Step 2 Disable the interfaces and take them out of service.

switch(config-if)# shutdown
switch(config-if)# out-of-service

Step 3 Return to configuration mode.

switch(config-if)# exit
switch#

Step 4 Select the interfaces fc 4/1 through fc 4/6.

switch# config t
switch(config)# interface fc 4/1 - 6

Step 5 Configure the port speed, rate mode, and port mode on the interfaces.

switch(config-if)# switchport speed auto max 2000
switch(config-if)# switchport rate-mode dedicated
switch(config-if)# switchport mode e

Step 6 Enable the interfaces and return to configuration mode.

switch(config-if)# no shutdown
switch(config-if)# exit
switch#

Step 7 Select the interfaces fc 4/7 through fc 4/10.

switch# config t
switch(config)# interface fc 4/7 - 10

Step 8 Configure the port speed, rate mode, and port mode on the interfaces.

switch(config-if)# switchport speed 1000
switch(config-if)# switchport rate-mode shared
switch(config-if)# switchport mode f

Step 9 Enable the interfaces and return to configuration mode.

switch(config-if)# no shutdown
switch(config-if)# exit
switch#

Default Settings

Table 14-16 lists the default settings for Generation 2 interface parameters.

Table 14-16 Default Generation 2 Interface Parameters 

Parameter
Default
48-Port 4-Gbps Switching Module
24-Port 4-Gbps Switching Module
12-Port 4-Gbps Switching Module
4-Port 10-Gbps Switching Module

Speed mode

auto1

auto1

auto1

auto2

Rate mode

shared

shared

dedicated

dedicated

Port mode

Fx

Fx

auto3

auto4

BB_credit buffers

16

16

250

250

Performance buffers

-

-

1455

1455

1

2 The 4-port 10-Gbps switching module only supports 10-Gbps traffic.

3 Auto port mode on the 12-port 4-Gbps switching module interfaces can operate in E port mode, TE port mode, and Fx port mode.

4 Auto port mode on the 4-port 10-Gbps switching module interfaces can operate in E port mode, TE port mode, and F port mode.

5 Performance buffers are shared among all ports on the module.