Port channels refer to the aggregation of multiple physical interfaces into one logical interface to provide higher aggregated bandwidth, load balancing, and link redundancy (as shown in the following image). Port channels connect to interfaces across switching modules, so failure of a switching module cannot bring down the port channel link.

Port channels on Cisco MDS 9000 Series Multilayer Switches allow flexibility in configuration. This illustrates three possible port channel configurations:

• Port channel A aggregates two links on two interfaces on the same switching module at each end of a connection.

• Port channel B aggregates two links, but each link is connected to a different switching module. If the switching module goes down, traffic is not affected.

• Port channel C aggregates three links. Two links are on the same switching module at each end, while one is connected to a different switching module on switch 2.

An E port channel is the aggregation of multiple E ports into one logical interface to provide higher aggregated bandwidth, load balancing, and link redundancy. Port channels connect interfaces across switching modules, so a failure of a switching module cannot bring down the port channel link.

Features and restrictions of a port channel:

• Provides a point-to-point connection over ISL (E ports) or EISL (TE ports). You can combine multiple links into a port channel.

• Increases the aggregate bandwidth on an ISL by distributing traffic among all functional links in the channel.

• Load balances across multiple links and maintains optimum bandwidth utilization. There are two types of load balancing

  • Flow-based load balancing which is based on the source ID and destination ID.

  • Exchange-based load balancing which is based on source ID, destination ID, and exchange ID (OX ID).

• Provides high availability on an ISL. If one link fails, traffic that is previously carried on this link changes over to the remaining links. If a link goes down in a port channel, the upper protocol is not aware of it. To the upper protocol, the link is still there, although the bandwidth is diminished. Link failure of a port channel member does not affect the routing tables and fspf cost of the port channel. Port channels may contain up to 16 physical links and may span multiple modules for added high availability.

Note	See Cisco MDS 9000 Series NX-OS Fabric Configuration Guide for information about Failover scenarios for port channels and FSPF links.

Note	It is not recommended that you use interface, fWWN, or domain-ID based zoning for devices that are connected to the edge Cisco N-Port Virtualization (NPV) switches.

F port channels provide fault tolerance and performance benefits on connections to N-Port Virtualization (NPV) switches, including Cisco UCS Fabric Interconnects (FIs). F port channels present unique challenges to ACL TCAM programming. When F ports are aggregated into a port channel, ACL TCAM programming is repeated on each member interface. Consequently, these types of port channels multiply the amount of TCAM entries needed. Because of this, it is imperative that the member interfaces are allocated as optimally as possible, and that zoning best practices are also followed. Given that F port channels can contain 100+ host logins, TCAM can easily be exceeded, especially for fabric switches if best practices are not followed.

The following is a sample topology:

This example assumes that the port channel (PC) contains 8 interfaces, fc1/1-fc1/8.

In addition, the following two zones are active:

zone1
member host (host 0x010001)
member target1 (target1 0x010002)
zone2
member host (host 0x010001)
member target2 (target2 0x010003)
 

In such a scenario, the following ACL programming will be present on each member of the PC:

fc1/1(through fc1/8) (port-channel)
Entry#    Source ID     Mask         Destination ID         Mask              Action
1         010001       ffffff        010002(target1)       ffffff             Permit
2         010001       ffffff        010003(target2)       ffffff             Permit
3         000000       000000        000000                000000             Drop

The above example shows the ACL TCAM programming that will be duplicated on each member of the F port-channel.

The following are the best practices for efficient use of TCAM with respect to F ports and F port-channels to optimize TCAM usage on a forwarding engine:

• Distribute port-channel member interfaces into different forwarding engines, especially on fabric switches.

• If TCAM usage is still too high in the case of port-channel with a large number of interfaces, then split the port-channel into two separate port-channels each with half the interfaces. This provides redundancy but reduces the number of FLOGIs per individual port-channel and thus reduces TCAM usage.

• Distribute member interfaces into separate linecards on director-class switches.

• Distribute member interfaces into forwarding engines with lower TCAM zoning region usage.

• Use single-initiator zones, single-target zones, or Smart Zoning.

Trunking is a commonly used storage industry term. However, the Cisco NX-OS software and switches in the Cisco MDS 9000 Series Multilayer Switches implement trunking and port channels as follows:

• Port channels enable several physical links to be combined into one aggregated logical link.

• Trunking enables a link transmitting frames in the EISL format to carry (trunk) multiple VSAN traffic. For example, when trunking is operational on an E port, that E port becomes a TE port. A TE port is specific to switches in the Cisco MDS 9000 Series Multilayer Switches. An industry standard E port can link to other vendor switches and is referred to as a non-trunking interface (see Trunking Only and Port Channeling and Trunking). See Configuring Trunking for information on trunked interfaces.

Port channels and trunking are used separately across an ISL.

• Port channels—Interfaces can be channeled between the following sets of ports:

  • E ports and TE ports

  • F ports and NP ports

  • TF ports and TNP ports

• Trunking—Trunking permits carrying traffic on multiple VSANs between switches.

  See Cisco MDS 9000 Series NX-OS Fabric Configuration Guide.

• Both port channeling and trunking can be used between TE ports over EISLs.

Note

After changing the port-channel mode, each member interface must be brought down and brought back up via the shutdown and no shutdown commands for the port-channel mode to be changed. This can be done on an individual member-by-member basis such that the port-channel remains up and fully functional.

You can configure each port channel with a channel group mode parameter. Such configuration determines the port channel protocol behavior for all member ports in this channel group. The possible values for a channel group mode are as follows:

• On—When the port channel created in ON mode, user has to run no shutdown command for each port channel member. . Port channels that are configured in the On mode require you to explicitly enable and disable the port channel member ports at either end if you add or remove ports from the port channel configuration. Physically verify that the local and remote ports are connected to each other.

  Beginning with Cisco MDS Release 8.4(1), the default mode is changed from On to Active.

• Active—When the port channel created in ACTIVE mode, members will be operational automatically provided the ISL(s) is operationally up. The Active port channel mode allows automatic recovery without explicitly enabling and disabling the port channel member ports at either end.

Note

Before Cisco MDS NX-OS Release 8.3(1), the CLI and the Device Manager creates port channels in On mode on the NPIV core switches and Active mode on the NPV switches. NDFC SAN Controller creates all port channels in Active mode. We recommend that you create port channels in Active mode. From Cisco MDS NX-OS Release 8.4(1), the CLI and the Device Manager create the port channel in Active mode on the NPIV core switches.

The following table provides a comparison between On and Active modes.

When you delete port channels, their corresponding channel membership is also deleted. All interfaces in the deleted port channel convert to individual physical links. The ports going down gracefully indicates that no frames were lost when the interface went down Graceful Shutdown).

The individual ports within the deleted port channel retain compatibility parameter settings (speed, mode, port VSAN, allowed VSAN, port security and so on). You can modify these settings as required. When you delete port channel in On mode, run no shutdown command on each member to make it operational.

You can add or remove a physical interface (or range of interfaces) to an existing port channel. For information about port channel support on Generation 2 switching modules, see Port Channel Limitations.

Cisco NX-OS software provides robust error detection and synchronization capabilities. You can manually configure channel groups or they can be automatically created. In both cases, the channel groups have the same capability and configuration parameters. Any change in configuration applied to the associated port channel interface is propagated to all members of the channel group.

A protocol to exchange port channel configurations is available in all Cisco MDS switches. This simplifies port channel management with incompatible ISLs.

The port channel protocol is enabled by default.

It uses exchange peer parameters (EPP) services to communicate across peer ports in an ISL. Each port uses the information that is received from the peer ports along with its local configuration and operational values to decide if it has to be part of a port channel. The protocol ensures that a set of ports is eligible to be part of the same port channel. They are only eligible to be part of the same port channel if all the ports have a compatible partner.

The port channel protocol uses Bring-up protocol which helps in Automatically detecting the misconfigurations. This protocol synchronizes the port channel at both ends so that all frames for a given flow are carried over the physical link in both directions. This allows applications such as write acceleration, work for port channels over FCIP links.

Cisco MDS 9000 Series Multilayer switches support the following number of port channels per switch:

• A port channel number refers to the unique identifier for each channel group. This number ranges from of 1–256.

  Note	You cannot change the port channel number, but the member ports operate according to the properties of the port channel.

The following table describes the results of adding a member to a port channel for various configurations.

This section includes the restrictions on creation and addition of port channel members to a port channel on Generation 1 hardware:

• The 32-port 2 or 1 Gbps switching module.

• The MDS 9140 and 9120 switches.

When configuring the host-optimized ports on Generation 1 hardware, the following port channel guidelines apply:

• If you execute the write erase command on a 32-port switching module, and then copy a saved configuration to the switch from a text file that contains the no system default switchport shutdown command, you need to copy the text file to the switch again for the E ports to come up without manual configuration.

• Any (or all) full line rate ports in the Cisco MDS 9100 Series can be included in a port channel.

• The host-optimized ports in the Cisco MDS 9100 Series are subject to the same port channel rules as 32-port switching modules. Only the first port of each group of 4 ports is included in a port channel.

  • You can configure only the first port in each 4-port group as an E port (for example, the first port in ports 1–4, the fifth port in ports 5–8, and so on). If you configure the first port in the group as a port channel, the other three ports in each group (ports 2–4, 6–8, and so on) are not usable and remain in the shutdown state.

  • If you configure any of the other three ports in a no shutdown state, you cannot configure the first port to be a port channel. The other three ports continue to remain in a no shutdown state.

Port channels are created with default values. You can change the default configuration just like any other physical interface.

Valid Port Channel Configurations provides examples of valid port channel configurations.

Misconfigured Configurations provides examples of invalid configurations. Assuming that the links are brought up in the 1, 2, 3, 4 sequence, links 3 and 4 will be operationally down because the fabric is misconfigured.

The port channel interface must be in Active mode when you configure multiple FCIP interfaces with Write-Acceleration.

The following guidelines and restrictions are applicable for F, TF, and NP port channels:

• On the switch with feature npiv configured the ports must be in F mode.

• On the switch with feature npv configured the ports must be in NP mode.

• Automatic creation is not supported.

• On mode is not supported. Only Active-Active mode is supported. By default, the mode is Active on the NPV switches.

• Devices that are logged in through an F port channel on an MDS switch are not supported in IVR non-NAT configuration. The devices are supported only in IVR NAT configuration.

• Port security rules are enforced only on physical pWWNs at the single link level.

• FC-SP authenticates only the first physical FLOGI of every port channel member.

• Since the FLOGI payload carries only the VF bits to trigger the use of a protocol after the FLOGI exchange, those bits are overridden. If the Cisco NPV switches, the core has a Cisco WWN and tries to initiate the PCP protocol.

• The name server registration of the N ports logging in through an F port channel uses the fWWN of the port channel interface.

• DPVM configuration is not supported.

• You cannot configure the port channel port VSAN using DPVM.

• The Dynamic Port VSAN Management (DPVM) database is queried only for the first physical FLOGI of each member, so that the port VSAN can be configured automatically.

• DPVM does not bind FC_IDs to VSANs, but pWWNs to VSANs. It is queried only for the physical FLOGI.

Cisco MDS switches use Ternary Content Addressable Memory (TCAM) on its Fibre Channel modules. TCAM provides an Access Control List (ACL) type of function for Cisco MDS. The process that controls this functionality is called ACLTCAM. The E or TE ports (ISLs) and F (Fabric) ports have their own programming that is unique to their respective port types.

TCAM is allocated to individual forwarding engines and forwarding engines are assigned a group of ports. Director-class Fibre Channel modules have more TCAM space than fabric switches. The number of forwarding engines, the ports assigned to each forwarding engine, and the amount of TCAM allocated to each forwarding engine is hardware dependent.

The following example shows an output from Cisco MDS 9148S:

switch# show system internal acl tcam–soc tcam–usage
TCAM Entries:
=============
                  Region1   Region2    Region3     Region4   Region5   Region6
Mod Fwd   Dir     TOP SYS  SECURITY    ZONING      BOTTOM    FCC DIS   FCC ENA
    Eng          Use/Total Use/Total  Use/Total   Use/Total Use/Total Use/Total
––– –––  –––––– –––––––––– ––––––––– –––––––––––– ––––––––– ––––––––– –––––––––
1   1    INPUT     19/407     1/407      1/2852 *    4/407     0/0       0/0
1   1    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
1   2    INPUT     19/407     1/407      0/2852 *    4/407     0/0       0/0
1   2    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
1   3    INPUT     19/407     1/407      0/2852 *    4/407     0/0       0/0
1   3    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
–––––––––––––––––––––––––––––––––––––––––––––––––––
* 1024 entries are reserved for LUN Zoning purpose.

The above example indicates the following:

• There are three forwarding engines, 1 through 3.

• Since there are 48 ports on Cisco MDS 9148 switches, each forwarding engine handles 16 ports.

• Each forwarding engine has 2852 entries in region 3 (the zoning region) for input. This is the main region used, and consequently, has the largest amount of available entries.

• Forwarding engine 3 has only one entry that is currently in use out of the total 2852 in the zoning region.

The following example shows the output from Cisco MDS 9710 switch with a 2/4/8/10/16 Gbps Advanced Fibre Channel Module (DS–X9448–768K9):

F241–15–09–9710–2# show system internal acl tcam–usage
TCAM Entries:
=============
                  Region1   Region2    Region3     Region4   Region5   Region6
Mod Fwd   Dir     TOP SYS  SECURITY    ZONING      BOTTOM    FCC DIS   FCC ENA
    Eng          Use/Total Use/Total  Use/Total   Use/Total Use/Total Use/Total
––– –––  –––––– –––––––––– ––––––––– –––––––––––– ––––––––– ––––––––– –––––––––
1   0    INPUT     55/19664    0/9840     0/49136*   17/19664    0/0       0/0
1   0    OUTPUT    13/4075    0/1643     0/11467     0/4075    6/1649   21/1664
1   1    INPUT     52/19664    0/9840     2/49136*   14/19664    0/0       0/0
1   1    OUTPUT     7/4078    0/1646     0/11470     0/4078    6/1652    5/1651
1   2    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   2    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   3    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   3    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   4    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   4    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   5    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   5    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
...

The above example indicates the following:

• There are six forwarding engines, 0 through 5.

• Since there are 48 ports on a Cisco MDS DS–X9448–768K9 module, each forwarding engine handles eight ports.

• Each forwarding engine has 49136 entries in region 3 (the zoning region) for input. This is the main region that is used, and consequently, has the largest amount of available entries.

• Forwarding engine 2 has only two entries that are currently in use out of the total 49136 in the zoning region.

Note

The commands that are used to view TCAM usage on fabric switches are different from the ones used for director–class switches. For MDS 9148, MDS 9148S, and MDS 9250i fabric switches, use the show system internal acltcam-soc tcam-usage command. For director class switches, MDS 9396S, and 32 Gbps fabric switches, use the show system internal acl tcam-usage command.

The following table provides information about ports to forwarding engines mapping:

Port channels provide Inter Switch Links (ISLs) between switches. Typically, there is minimal TCAM programming on these types of interfaces. When the Inter VSAN Routing(IVR) feature is being deployed, extensive TCAM programming can exist on ISLs because the IVR topology transitions from one VSAN to another. Most of the considerations that apply on F/TF port channels will be applicable here too.

The following is an example of a topology:

In this topology:

• Both Cisco MDS 9148S-1 and MDS 9148S-2 are in the IVR VSAN topology:

  
  MDS9148S-1 vsan 1 and vsan 2
  MDS9148S-2 vsan 2 and vsan 3

• IVR NAT is configured.

• VSAN 2 is the transit VSAN.

  
  FCIDs per VSAN:
              VSAN 1  VSAN 2  VSAN 3
  Host     			010001  210001  550002
  Target1 			440002  360002  030001

  Note	Domains 0x44 in VSAN 1, 0x21 and 0x36 in VSAN 2, and 0x55 in VSAN 3 are virtual domains created by IVR NAT.

• The following is the IVR zoning topology:

  
  ivr zone zone1
  member host vsan 1
  member target1 vsan3

• The following is the ACL TCAM programming for the IVR zoning topology:

  
  MDS9148S-1  fc1/1(Host) - VSAN 1
  Entry# 			Source ID    				Mask      Destination ID  								Mask    Action
  1      			010001(host) 				ffffff    440002(target1) 								ffffff  Permit
             - Forward to fc1/2
         - Rewrite the following information:
           VSAN to 2
           Source ID to 210001
           Destination ID to 360002
  2      			000000     				000000    000000          								000000  Drop
  MDS9148S-1  fc1/2(ISL) - VSAN 2
  Entry# 			Source ID       					Mask      Destination ID  								Mask    Action
  1      			360002(Target1) 					ffffff    210001(host)    								ffffff  Permit
         - Forward to fc1/2
         - Rewrite the following information:
           VSAN to 1
           Source ID to 440002
           Destination ID to 010001
  MDS9148S-2 fc1/2(ISL) - VSAN 2
  Entry# 			Source ID    				Mask      Destination ID  								Mask    Action
  1      			210001(host) 				ffffff    360002(target1) 								ffffff  Permit
         - Forward to fc1/2
         - Rewrite the following information:
           VSAN to 3
           Source ID to 550002
           Destination ID to 030001
  MDS9148S-2  fc1/1(Target1) - VSAN 3
  Entry# 			Source ID       					Mask      Destination ID  								Mask    Action
  1      			030001(Target1) 					ffffff    550002(host)    								ffffff  Permit
         - Forward to fc1/2
         - Rewrite the following information:
           VSAN to 2
           Source ID to 360002
           Destination ID to 210001
  2      			000000     					000000    000000          									000000  Drop

  Note	Besides the entries in this example, there are other entries that IVR adds to capture important frames such as PLOGIs, PRILIs, and ABTS.

The programming on the host and target1 ports is similar to the way it is without IVR, except that the FCIDs and VSANs are explicitly forwarded to an egress port and are rewritten to values that are appropriate for the transit VSAN (VSAN 2). These forwarding and rewrite entries are separate and are not included in the TCAM-usage values.

However,now, on the ISLs in both the switches, programming that did not exist earlier is present. When frames from Host to Target1 are received by Cisco MDS 9148S-2 fc1/2, they are rewritten to the values in VSAN3 where the target resides. In the reverse direction, when frames from Target1 to the Host are received by Cisco MDS 9148S-1 fc1/2, they are rewritten to the values in VSAN 1 where the Host resides. Therefore, for each VSAN transition on an ISL (that typically occurs across a transit VSAN) there is TCAM programming for each device in the IVR zone set.

Consequently,most of the best practices followed for the F and TF port channels should be followed to ensure that TCAM is utilized as efficiently as possible for the following purposes:

Note

Unlike F and TF port-channels, the ACLTCAM programming on ISLs will be the same quantity regardless if the ISLs are part of a port-channel or not. If there are "n" ISLs between two MDS switches, then it doesn't matter if they are in one port-channel, two port-channels or just individual links. The ACLTCAM programming will be the same.

• Distribute port-channel member interfaces into different forwarding engines, especially on fabric switches.

• Distribute member interfaces into different linecards on director-class switches.

• Distribute member interfaces into forwarding engines with lower TCAM zoning region usage.

• Use single-initiator zones, single-target zones, or Smart Zoning.