Cisco MDS 9000 Family Configuration Guide, Release 1.2(1a) - Configuring PortChannels [Cisco MDS 9000 NX-OS and SAN-OS Software]

Table Of Contents

Configuring PortChannels

PortChannel Examples

Configuring 32-port Switching Modules and Oversubscribed Ports

About PortChanneling and Trunking

About Load Balancing

Creating a PortChannel

Deleting a PortChannel

Adding Interfaces to a PortChannel

Forcing an Interface Addition

Compatibility Check

Suspended State

Deleting Interfaces from a PortChannel

Considerations for PortChannel Configurations

Error Detection

Viewing PortChannel Information

Default Settings

Configuring PortChannels

PortChannels refer to the aggregation of multiple physical interfaces into one logical interface to provide higher aggregated bandwidth, load balancing, and link redundancy. PortChannels can connect to interfaces across switching modules, so a failure of a switching module cannot bring down the PortChannel link. Specifically, a PortChannel has the following functionality:

•Provides a point-to-point connection over an ISL (E ports) or EISL (TE ports). Multiple links can be combined into a PortChannel.

•Increases the aggregate bandwidth on an ISL by distributing traffic among all functional links in the channel.

•Load balances across multiple links and maintains optimum bandwidth utilization. Load balancing is based on the source ID, destination ID, and exchange ID (OX ID).

•Provides high availability on an ISL. If one link fails, traffic previously carried on this link is switched to the remaining links. If a link goes down in a PortChannel, the upper protocol is not aware of it. To the upper protocol, the link is still there, although the bandwidth is diminished. The routing tables are not affected by link failure. PortChannels may contain up to 16 physical links and may span multiple modules for added high availability.

Note See the "Fail-over Scenarios for PortChannels and FSPF Links" section for fail-over scenarios.

Cisco MDS 9000 Family of switches support 128 PortChannels with 16 interfaces per PortChannel.

This chapter discusses the PortChannel feature provided in the switch. This chapter includes the following sections:

•PortChannel Examples

•About PortChanneling and Trunking

•About Load Balancing

•Creating a PortChannel

•Deleting a PortChannel

•Adding Interfaces to a PortChannel

•Deleting Interfaces from a PortChannel

•Considerations for PortChannel Configurations

•Viewing PortChannel Information

•Default Settings

PortChannel Examples

PortChannels on Cisco MDS 9000 Family switches allow flexibility in configuration. Figure 11-1 illustrates three possible PortChannel configurations:

•PortChannel A aggregates two links on two interfaces on the same switching module at each end of a connection.

•PortChannel B also aggregates two links, but each link is connected to a different switching module. If the switching module goes down, traffic is not affected.

•PortChannel C aggregates three links. Two links are on the same switching module at each end, while one is connected to a different switching module on switch 2.

Figure 11-1 PortChannel Flexibility

Configuring 32-port Switching Modules and Oversubscribed Ports

The 32-port 1/2-Gbps switching module contains 8 port groups of 4 ports each. When configuring these modules or the oversubscribed ports in the Cisco 9100 Series, the following guidelines apply:

•Any (or all) full line rate port(s) in the Cisco MDS 9100 Series can be included in a PortChannel.

•The oversubscribed ports in the Cisco MDS 9100 Series are subject to same PortChannel rules as 32-port switching modules—only the first port of each group of 4 ports is included in a PortChannel.

–You can configure only the first port in each 4-port group (for example, the first port in ports 1-4, the fifth port in ports 5-8 and so on) as an E port. If the first port in the group is configured as a PortChannel, the other three ports in each group (ports 2-4, 6-8 and so on) are not usable and remain in the shutdown state.

–If any of the other three ports are configured in a no shutdown state, you cannot configure the first port as a PortChannel. The other three ports continue to remain in a no shutdown state.

Note In the Cisco MDS 9100 Series, the left most groups of ports outlined in white (4 ports in the 9120 switch and 8 ports in the 9140 switch) are full line rate like the 16-port switching module. The other ports (16 ports in the 9120 switch and 32 ports in the 9140 switch) are oversubscribed like the 32-port switching module. Each group of 4 oversubscribed ports have the same rules as for the 32-port switching module.

About PortChanneling and Trunking

PortChanneling enables several links to be combined into one aggregated link.

Trunking enables an ISL to carry (trunk) multiple VSANs. Trunking can only be configured on a TE port. A TE port is specific to switches in the Cisco MDS 9000 Family. An industry standard E port can link to other vendor switches and is referred to as a nontrunking interface (see Figure 11-2).

See "Configuring Trunking" for information on trunked interfaces.

Figure 11-2 PortChanneling and Trunking

PortChanneling and trunking are used separately across an ISL:

•PortChanneling—Interfaces can be channeled between E ports over multiple ISLs or between TE ports over multiple EISLs.

•Trunking—Trunking, which permits carrying VSAN IDs between switches, can be done only between TE ports over EISLs.

See "Configuring and Managing VSANs."

Both PortChanneling and trunking can be used between TE ports over EISLs.

About Load Balancing

Two mechanisms support the load balancing functionality:

•Flow based—All frames between source and destination follow the same links for a given flow. That is, whichever link is selected for the first exchange of the flow is used for all subsequent exchanges.

•Exchange based—The first frame in an exchange picks a link and subsequent frames in the exchange follow the same link. However, subsequent exchanges can use a different link. This provides more granular load balancing while preserving the order of frames for each exchange.

Figure 11-3 illustrates how source ID 1 (SID1) and destination ID1-based(DID1) load balancing works. When the first frame in a flow is received on an interface for forwarding, link 1 is selected. Each subsequent frame in that flow is sent over the same link. No frame in SID1 and DID1 utilizes link 2.

Figure 11-3 SID1 and DID1Based Load Balancing

Figure 11-4 illustrates how exchange based load balancing works. When the first frame in an exchange is received for forwarding on an interface, link 1 is chosen by a hash algorithm. All remaining frames in that particular exchange are sent on the same link. For exchange 1, no frame uses link 2. For the next exchange, link 2 is chosen by the hash algorithm. Now all frames in exchange 2 use link 2.

Figure 11-4 SID1, DID1, and Exchange Based Load Balancing

For more information on configuring load balancing and in-order delivery features, see the "VSAN Attributes" section.

Creating a PortChannel

You can create PortChannels using the interface port-channel command. PortChannels are created with default values. You can change the default configuration just like any other physical interface.

To create a PortChannel, follow these steps:
Command

Purpose
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# interface port-channel 1
switch(config-if)# 
Configures the specified PortChannel (1).
Note All interfaces added to PortChannels are administratively shut down, and the PortChannel remains administratively up.

Deleting a PortChannel

To delete the PortChannel, you must explicitly issue the no interface port-channel command. When you delete the PortChannel, the corresponding channel membership is also deleted. All interfaces in the deleted PortChannel convert to individual physical links. To avoid inconsistent states across switches, and to maintain consistency across switches, the ports shut down. They continue to use the configured values of the physical port.

To delete a PortChannel, follow these steps:
Command

Purpose
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# no interface port-channel 1
port-channel 1 deleted and all its members 
disabled
please do the same operation on the switch at the 
other end of the port-channel
switch(config)# 
Deletes the specified PortChannel (1), its associated interface mappings, and the hardware associations for this PortChannel.
Adding Interfaces to a PortChannel

You can add a physical interface (or a range of interfaces) to a nonexistent or an existing PortChannel and the PortChannel is automatically created. If the PortChannel does not exist, it is created. The compatible parameters on the configuration are mapped to the PortChannel.

To add a port (or a range of ports) to a PortChannel, follow these steps:
Command

Purpose
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# interface fc1/15
switch(config-if)# 
Configures the specified port interface (fc1/15).
switch(config)# interface fc1/1 - 5
switch(config-if)# 
Configures the specified range of interfaces. In this example, interfaces from 1/1 to 1/5
Step 3
switch(config-if)# channel-group 15
fc1/15 added to port-channel 15 and disabled
please do the same operation on the switch at 
the other end of the port-channel, then do 
"no shutdown" at both ends to bring them up
switch(config-if)# 
Adds physical Fibre Channel port 1/15 to channel group 15. If channel group 15 does not exist, it is created. The port is shut down.
switch(config-if)# channel-group 2
fc1/1 fc1/2 fc1/3 fc1/4 fc1/5 added to 
port-channel 2 and disabled
please do the same operation on the switch at 
the other end of the port-channel, then do 
"no shutdown" at both ends to bring them up
switch(config-if)# 
Adds physical interfaces 1/1, 1/2, 1/3, 1/4, and 1/5 to channel group 2. If channel group 2 does not exist, it is created.

If the compatibility check is successful, the interfaces are operational and the corresponding states apply to these interfaces.
Forcing an Interface Addition

You can specify a force option to force the port configuration to be overwritten by the PortChannel. In this case, the interface is added to a PortChannel and the port is shut down.

Note When PortChannels are created automatically, the force option cannot be used.

To force the addition of a port to a PortChannel, follow these steps:
Command

Purpose
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# interface fc1/1
switch(config-if)# 
Configures the specified port interface (fc1/1).
Step 3
switch(config-if)# channel-group 1 force
fc1/1 added to port-channel 1 and disabled
please do the same operation on the switch at 
the other end of the port-channel, then do 
"no shutdown" at both ends to bring them up
switch(config-if)# 
Forces a physical Fibre Channel port 1/1 addition to channel group 1. The E port is shut down.
Compatibility Check

A compatibility check ensures that the same configuration values are used in all physical ports in the channel. For example, to enable trunk mode, all operational ports in the configuration must be configured in the trunk mode or in the nontrunking mode. Otherwise, they cannot become part of a PortChannel. A port cannot be operational if it is incompatible with the PortChannel. If the compatibility check is successful, the interfaces are operational and the corresponding states apply to these interfaces.

Suspended State

An interface enters the suspended state if its operational values are incompatible with the PortChannel. A compatibility check on operational parameters is done when one of the following events occurs:

•A port becomes operational in a PortChannel.

•An operational parameter changes for a port in a PortChannel.

The software performs a compatibility check on the operational parameters and places the interface in an operational or suspended state based on the result (see the "Reason Codes" section).

Deleting Interfaces from a PortChannel

To delete a physical interface (or a range of physical interfaces), you must explicitly issue the no channel-group command at the physical interface level. When a physical interface is deleted from the PortChannel, the channel membership is automatically updated. If the deleted interface is the last operational interface, then the PortChannel status is changed to a down state. Deleting an interface from a PortChannel decreases the channel size and bandwidth of the PortChannel.

Note When an interface is deleted, it is shut down but the physical configuration is retained. The inherited PortChannel configuration information is not deleted.

To delete a physical interface (or a range of physical interfaces), follow these steps:
Command

Purpose
Step 1
switch(config)# interface fc1/1
switch(config-if)#
Enters the selected physical interface level.
switch(config)# interface fc1/1 - 5
switch(config-if)#
Enters the selected range of physical interfaces.
Step 2
switch(config-if)# no channel-group 2
fc1/1 fc1/2 fc1/3 fc1/4 fc1/5 removed 
from port-channel 2 and disabled. Please 
do the same operation on the switch at 
the other end of the port-channel
switch(config-if)#
Deletes the physical Fibre Channel interfaces in channel group 2.
Considerations for PortChannel Configurations

Before configuring a PortChannel, consider the following guidelines

•Configure the PortChannel across switching modules to prevent redundancy on switching module reboots or upgrades.

•Ensure that one PortChannel is not connected to two switches. PortChannels require point-to-point connections.

Error Detection

If you misconfigure PortChannels, you may receive the Error disabled - Possible port channel misconfiguration message. If you receive this message, the PortChannel's physical links are disabled since an error has been detected.

A PortChannel error is detected if the following requirements are not met:

•Each switch on either side of a PortChannel must be connected to the same number of interfaces.

•Each interface must be connected to a corresponding interface on the other side.

•Links in a PortChannel must not be changed after the PortChannel is configured.

If you change the links after the PortChannel is configured, be sure to reconnect the links to interfaces within the PortChannel and re-enable the links. Issue the show interface command for that interface to verify that the PortChannel is functioning as required.

If all three conditions are not met, the faulty link is disabled.

Viewing PortChannel Information

You can view specific information about existing PortChannels at any time from EXEC mode. The following show commands provide further details on existing PortChannels. You can force all screen output to go to a printer or save it to a file.

The show port-channel summary command displays a summary of PortChannels within the switch. A one-line summary of each PortChannel provides the administrative state, the operational state, the number of attached and active interfaces (up), and the first operational interface (FOP), which is the primary operational interface selected in the PortChannel. See Examples 11-1 to 11-6.

Example 11-1 PortChannel Summary
switch# show port-channel summary  
------------------------------------------------------------------------------
Interface                 Total Ports        Oper Ports        First Oper Port
------------------------------------------------------------------------------
port-channel 1                 2                 2                  fc2/3
port-channel 2                 2                 2                  fc2/5
port-channel 3                 2                 2                  fc2/10
. 
. 
.
Example 11-2 PortChannel Compatibility
switch# show port-channel compatibility-parameters
physical port layer                 fibre channel or ethernet
    port mode                           E/AUTO only
    trunk mode
    speed
    port VSAN
    port allowed VSAN list
Example 11-3 PortChannel Database
switch# show port-channel database  
port-channel 1
    Administrative channel mode is on
    Operational channel mode is on
    Last membership update succeeded
    First operational port is fc2/3
    2 ports in total, 2 ports up
    Ports:   fc2/3    [up]
             fc2/4    [up]
port-channel 2
    Administrative channel mode is on
    Operational channel mode is on
    Last membership update succeeded
    First operational port is fc2/5
    2 ports in total, 2 ports up
    Ports:   fc2/5    [up]
             fc2/6    [up]
.
.
.
The show port-channel consistency command has two options—without detail and detail.

Example 11-4 Command Without Details
switch# show port-channel consistency  
sup database:
================================================
totally 7 port-channels
port-channel 1:
    2 ports, first operational port is fc2/3
    fc2/3    [up]
    fc2/4    [up]
port-channel 2:
    2 ports, first operational port is fc2/5
    fc2/5    [up]
    fc2/6    [up]
.
.
.
Example 11-5 Command With Details
switch# show port-channel consistency detail
Authoritative port-channel database:
================================================
totally 7 port-channels
port-channel 1:
    2 ports, first operational port is fc2/3
    fc2/3    [up]
    fc2/4    [up]
port-channel 2:
    2 ports, first operational port is fc2/5
    fc2/5    [up]
    fc2/6    [up]
.
.
.
================================================
database 1: from module 5
================================================
totally 7 port-channels
port-channel 1:
    2 ports, first operational port is fc2/3
    fc2/3    [up]
    fc2/4    [up]
port-channel 2:
    2 ports, first operational port is fc2/5
    fc2/5    [up]
    fc2/6    [up]
.
.
.
================================================
database 3: from module 2
================================================
totally 7 port-channels
port-channel 1:
    2 ports, first operational port is fc2/3
    fc2/3    [up]
    fc2/4    [up]
port-channel 2:
    2 ports, first operational port is fc2/5
    fc2/5    [up]
    fc2/6    [up]
.
.
.
The show port-channel usage command displays details of the used and unused PortChannel numbers.

Example 11-6 PortChannel Usage
switch# show port-channel usage  
Totally 7 port-channel numbers used
===================================
Used  :   1-7
Unused:   8-128
Default Settings

Table 11-1 lists the default settings for PortChannels.

Table 11-1 Default PortChannel Parameters

Parameters

Default

PortChannels

FSPF is enabled by default.

Create PortChannel

Administratively up.

Default mode

Auto.