Guest

EtherChannel

Configure EtherChannel Between Catalyst 4500/4000, 5500/5000, and 6500/6000 Switches That Run CatOS System Software

Cisco - Configure EtherChannel Between Catalyst 4500/4000, 5500/5000, and 6500/6000 Switches That Run CatOS System Software

Introduction

EtherChannel allows multiple physical Fast Ethernet or Gigabit Ethernet links to be combined into one logical channel. A logical channel allows load sharing of traffic among the links in the channel as well as redundancy in the event that one or more links in the channel fail. EtherChannel can be used to interconnect LAN switches, routers, servers, and clients with unshielded twisted pair (UTP) wiring or single-mode and multimode fiber.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

  • A console cable that is suitable for the Supervisor Engine in the switch

    For more information, refer to the Components Used section of the document Connecting a Terminal to the Console Port on Catalyst Switches.

  • Two Catalyst 5505 switches in a lab environment with cleared configurations

    The clear config all command was entered on the switch in order to ensure a default configuration.

  • A Fast Ethernet module that is capable of EtherChannel in each Catalyst 5505

  • Four RJ-45 Ethernet crossover cables to connect the EtherChannel

    For a pinout of an Ethernet crossover cable, see Appendix A: Ethernet Crossover Cables.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

EtherChannel is an easy way to aggregate bandwidth between critical networking devices. On the Catalyst 5500/5000, a channel can be created from two ports, which creates a 200-Mbps link (400-Mbps, full-duplex), or four ports, which creates a 400-Mbps link (800-Mbps, full-duplex). Some cards and platforms also support Gigabit EtherChannel and have the ability to use from two to eight ports in an EtherChannel. The concept is the same no matter the speeds or number of links that are involved. Normally, the Spanning Tree Protocol (STP) considers these redundant links between two devices to be loops and causes the redundant links to be in blocking mode. This effectively makes the links inactive (providing only backup capabilities if the main link fails). With use of Catalyst OS (CatOS) software version 3.1(1) or later, STP treats the channel as one large link, so all the ports in the channel can be active at the same time.

This document takes you through the steps to configure EtherChannel between two Catalyst 5500/5000 switches and shows you the results of the commands that you issue. You can use Catalyst 4500/4000 and 6500/6000 switches that run CatOS in the scenarios that this document presents in order to obtain the same results. For the Catalyst 2900XL and Catalyst 1900/2820, the command syntax differs, but the EtherChannel concepts are the same. For EtherChannel guidelines and configuration information for the Catalyst 6500/6000 series switches that run Cisco IOS® system software, refer to Sample Configuration: EtherChannel Between Catalyst Switches Running CatOS and Cisco IOS Software.

For an overview and comparison of the Catalyst 6500 CatOS and Cisco IOS Software platforms, refer to Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst 6500 Series Switch.

You can manually configure EtherChannel if you execute the appropriate commands. Or, for automatic configuration, have the switch negotiate the channel with the other side with use of Port Aggregation Protocol (PAgP). Whenever possible, use the PAgP desirable mode in order to configure EtherChannel because the manual configuration of EtherChannel sometimes creates complications. This document provides examples of the manual configuration of EtherChannel and examples of EtherChannel configuration with use of PAgP. The document also includes how to troubleshoot EtherChannel and how to use trunking with EtherChannel. In this document, the terms EtherChannel, Fast EtherChannel, Gigabit EtherChannel, or channel all refer to EtherChannel.

Network Diagram

The network setup in this section illustrates the test environment.

After the configuration of the switches was cleared with the clear config all command, the prompt was changed with the set system name command. An IP address and mask were assigned to the switch for management purposes with use of the set interface sc0 172.16.84.6 255.255.255.0 command for Switch A and the set interface sc0 172.16.84.17 255.255.255.0 command for Switch B. A default gateway was assigned to both switches with the set ip route default 172.16.84.1 command.

The switch configurations were cleared in order to start from the default conditions. The switches were given names for identification from the prompt on the command line. In order to ping between the switches for testing, the IP addresses were assigned. The default gateway was not used.

6a.gif

Many of the commands display more output than is needed for this discussion. Extraneous output is suppressed in this document.

Manually Configure EtherChannel

Step-by-Step

Complete these steps in order to manually configure EtherChannel:

  1. Issue the show version command and the show module command.

    The show version command displays the software version that the switch runs. The show module command lists which modules are installed in the switch.

    Switch-A> show version
    WS-C5505 Software, Version McpSW: 4.5(1) NmpSW: 4.5(1)     
    
    !--- This is the software version that runs on the switch.
    
    Copyright (c) 1995-1999 by Cisco Systems
    NMP S/W compiled on Mar 29 1999, 16:09:01
    MCP S/W compiled on Mar 29 1999, 16:06:50
    
    System Bootstrap Version: 3.1.2
    
    Hardware Version: 1.0  Model: WS-C5505  Serial #: 066507453
    
    Mod Port Model      Serial #  Versions
    --- ---- ---------- --------- ----------------------------------------
    1   0    WS-X5530   006841805 Hw : 1.3
                                  Fw : 3.1.2
                                  Fw1: 3.1(2)
                                  Sw : 4.5(1)
    2   24   WS-X5225R  012785227 Hw : 3.2
                                  Fw : 4.3(1)
                                  SW : 4.5(1)
    
           DRAM                    FLASH                   NVRAM
    Module Total   Used    Free    Total   Used    Free    Total Used  Free
    ------ ------- ------- ------- ------- ------- ------- ----- ----- -----
    1       32640K  13650K  18990K   8192K   4118K   4074K  512K  108K  404K
            
    Uptime is 0 day, 3 hours, 32 minutes
    
    Switch-A> show module
    Mod Module-Name         Ports Module-Type           Model    Serial-Num Status
    --- ------------------- ----- --------------------- --------- --------- -------
    1                       0     Supervisor III        WS-X5530  006841805 ok           
    
    !--- These are the modules that are installed on the switch.
    
    2                       24    10/100BaseTX Ethernet WS-X5225R 012785227 OK
    
    Mod MAC-Address(es)                        Hw     Fw         SW
    --- -------------------------------------- ------ ---------- -----------------
    1   00-90-92-b0-84-00 to 00-90-92-b0-87-ff 1.3    3.1.2      4.5(1)
    2   00-50-0f-b2-e2-60 to 00-50-0f-b2-e2-77 3.2    4.3(1)     4.5(1)
    
    Mod Sub-Type Sub-Model Sub-Serial Sub-Hw
    --- -------- --------- ---------- ------
    1   NFFC     WS-F5521  0008728786 1.0
    
    Switch-B> show version
    WS-C5505 Software, Version McpSW: 4.5(1) NmpSW: 4.5(1)        
    
    !--- This is the software version that runs on the switch.
    
    Copyright (c) 1995-1999 by Cisco Systems
    NMP S/W compiled on Mar 29 1999, 16:09:01
    MCP S/W compiled on Mar 29 1999, 16:06:50
    
    System Bootstrap Version: 5.1(2)
    
    Hardware Version: 1.0  Model: WS-C5505  Serial #: 066509957
    
    Mod Port Model      Serial #  Versions
    --- ---- ---------- --------- ----------------------------------------
    1   0    WS-X5530   008592453 Hw : 2.3
                                  Fw : 5.1(2)
                                  Fw1: 4.4(1)
                                  SW : 4.5(1)
    2   24   WS-X5234   015388641 Hw : 1.0
                                  Fw : 4.5(2)
                                  SW : 4.5(1)
    
           DRAM                    FLASH                   NVRAM
    Module Total   Used    Free    Total   Used    Free    Total Used  Free
    ------ ------- ------- ------- ------- ------- ------- ----- ----- -----
    1       32640K  13548K  19092K   8192K   7300K    892K  512K  119K  393K
            
    Uptime is 0 day, 3 hours, 36 minutes
    
    Switch-B> show module
    Mod Module-Name         Ports Module-Type           Model    Serial-Num Status
    --- ------------------- ----- --------------------- --------- --------- -------
    1                       0     Supervisor III        WS-X5530  008592453 OK
    
    !--- These are the modules that are installed on the switch.
    
    2                       24    10/100BaseTX Ethernet WS-X5234  015388641 OK
    
    Mod MAC-Address(Es)                        Hw     Fw         SW
    --- -------------------------------------- ------ ---------- -----------------
    1   00-10-0d-b2-8c-00 to 00-10-0d-b2-8f-ff 2.3    5.1(2)     4.5(1)
    2   00-d0-bc-03-58-98 to 00-d0-bc-03-58-af 1.0    4.5(2)     4.5(1)
    
    Mod Subtype Sub-Model Sub-Serial Sub-Hw
    --- -------- --------- ---------- ------
    1   EARL 1+  WS-F5520  0011591025 1.1
    
  2. Verify that EtherChannel is supported on the ports.

    Note: The show port capabilities command is available in CatOS software versions 4.x and later. If you have a software version that is earlier than 4.x, you must skip this step. Not every Fast Ethernet module supports EtherChannel. Some of the original EtherChannel modules have "Fast EtherChannel" printed on the bottom left corner of the module (as you face the module in the switch), which tells you that the feature is supported. But this convention was abandoned on later modules. The modules in this test do not have "Fast EtherChannel" printed on them, but they do support the feature.

    Switch-A> show port capabilities 2/1
    Model                    WS-X5225R
    Port                     2/1
    Type                     10/100BaseTX
    Speed                    auto,10,100
    Duplex                   half,full
    Trunk encap type         802.1Q,ISL
    Trunk mode               on,off,desirable,auto,nonegotiate
    Channel                  2/1-2,2/1-4                            
    
    !--- This indicates that EtherChannel can be configured on port 2/1 
    !--- with two or four contiguous ports. 
    
    Broadcast suppression    percentage(0-100)           
    Flow control             receive-(off,on),send-(off,on)
    Security                 yes
    Membership               static,dynamic
    Fast start               yes
    Rewrite                  yes
    
    Switch-B> show port capabilities 2/1
    Model                    WS-X5234
    Port                     2/1
    Type                     10/100BaseTX
    Speed                    auto,10,100
    Duplex                   half,full
    Trunk encap type         802.1Q,ISL
    Trunk mode               on,off,desirable,auto,nonegotiate
    
    !--- This indicates that EtherChannel can be configured on port 2/1 
    !--- with two or four contiguous ports. 
    
    Channel                  2/1-2,2/1-4                                   
    Broadcast suppression    percentage(0-100)                   
    Flow control             receive-(off,on),send-(off,on)
    Security                 yes
    Membership               static,dynamic
    Fast start               yes
    Rewrite                  no
    

    A port that does not support EtherChannel looks like this:

    Switch> show port capabilities 2/1
    Model                    WS-X5213A
    Port                     2/1
    Type                     10/100BaseTX
    Speed                    10,100,auto
    Duplex                   half,full
    Trunk encap type         ISL
    Trunk mode               on,off,desirable,auto,nonegotiate
    Channel                  no                         
    
    !--- This indicates that EtherChannel is not supported on this port 
    !--- or module.
    
    Broadcast suppression    pps(0-150000)
    Flow control             no
    Security                 yes
    Membership               static,dynamic
    Fast start               yes
  3. Verify that the ports are connected and operational.

    Before connection of the cables, the port status is:

    Switch-A> show port
    Port  Name               Status     Vlan       Level  Duplex Speed Type
    ----- ------------------ ---------- ---------- ------ ------ ----- ------------
     2/1                     notconnect 1          normal   auto  auto 10/100BaseTX
     2/2                     notconnect 1          normal   auto  auto 10/100BaseTX
     2/3                     notconnect 1          normal   auto  auto 10/100BaseTX
     2/4                     notconnect 1          normal   auto  auto 10/100BaseTX

    After connection of the cables between the two switches, the status is:

    1999 Dec 14 20:32:44 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1
    1999 DEC 14 20:32:44 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/2
    1999 DEC 14 20:32:44 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/3
    1999 DEC 14 20:32:44 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/4
    
    Switch-A> show port
    Port  Name               Status     Vlan       Level  Duplex Speed Type
    ----- ------------------ ---------- ---------- ------ ------ ----- ------------
     2/1                     connected  1          normal a-full a-100 10/100BaseTX
     2/2                     connected  1          normal a-full a-100 10/100BaseTX
     2/3                     connected  1          normal a-full a-100 10/100BaseTX
     2/4                     connected  1          normal a-full a-100 10/100BaseTX
    
    Switch-B> show port
    Port  Name               Status     Vlan       Level  Duplex Speed Type
    ----- ------------------ ---------- ---------- ------ ------ ----- ------------
     2/1                     connected  1          normal a-full a-100 10/100BaseTX
     2/2                     connected  1          normal a-full a-100 10/100BaseTX
     2/3                     connected  1          normal a-full a-100 10/100BaseTX
     2/4                     connected  1          normal a-full a-100 10/100BaseTX

    Because the switch configurations were cleared before the start of this test, the ports are in their default conditions. The ports are all in VLAN 1, and their speed and duplex are set to auto. After connection of the cables, the ports negotiate to a speed of 100 Mbps and full duplex. The status is connected. You are now able to ping the other switch.

    Switch-A> ping 172.16.84.17
    172.16.84.17 is alive

    In your network, you can set the speeds manually to 100 Mbps and full duplex if you want your ports to always run at the fastest speed. Then you do not need to rely on autonegotiation. For a discussion of autonegotiation, refer to Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation..

  4. Verify that the ports to be grouped have the same settings.

    This verification is an important step that the Troubleshoot EtherChannel section covers in more detail. If the command to set up EtherChannel does not work, the cause is typically that the ports that are involved in the channel have configurations that differ from each other. These ports include the ports on the other side of the link as well as the local ports. In this case, because the switch configurations were cleared before this test, the ports are in their default conditions. The ports are all in VLAN 1, their speed and duplex are set to auto, and all spanning tree parameters for each port are set to be the same. After connection of the cables in Step 3, you saw that the ports negotiate to a speed of 100 Mbps and full duplex. Because STP runs for each VLAN, a simple configuration of the channel and response to error messages is easier than an attempt to check every STP field for consistency for each port and VLAN in the channel.

  5. Identify valid port groups.

    On the Catalyst 5500/5000, you can only put certain ports together in a channel. These restrictive dependencies do not apply to all platforms. The ports in a channel on a Catalyst 5500/5000 must be contiguous. If you issue the show port capabilities command for port 2/1, the output shows the possible combinations:

    Switch-A> show port capabilities 2/1
    Model                    WS-X5225R
    Port                     2/1
    ...
    Channel                  2/1-2,2/1-4
    

    Notice that this port can be a part of a group of two (2/1-2) or part of a group of four (2/1-4). An Ethernet Bundling Controller (EBC) on the module causes these configuration limitations. Here is an example in which the show port capabilities command is issued for another port:

    Switch-A> show port capabilities 2/3
    Model                    WS-X5225R
    Port                     2/3
    ...
    Channel                  2/3-4,2/1-4
    

    This port can be part of a group of two ports (2/3-4) or a group of four ports (2/1-4).

    Note: There can be additional restrictions, which depends on the hardware. On certain modules (WS-X5201 and WS-X5203), you cannot form an EtherChannel with the last two ports in a port group unless the first two ports in the group already form an EtherChannel. A port group is a group of ports that are allowed to form an EtherChannel. In the example above, 2/1-4 is a port group.

    For example, if you want to create separate EtherChannels with only two ports in a channel, you cannot assign ports 2/3-4 to a channel until you have first configured ports 2/1-2 to a channel. This is true only for the modules that have this restriction. Similarly, before you configure ports 2/6-7, you must configure ports 2/5-6. This restriction does not occur on the modules that this document uses (WS-X5225R and WS-X5234).

    Because you are configuring a group of four ports (2/1-4), the group is within the approved grouping. You cannot assign a group of four to ports 2/3-6. This is a group of contiguous ports, but the ports do not start on the approved boundary, as the show port capabilities command shows. Valid groups are:

    • Ports 1-4

    • Ports 5-8

    • Ports 9-12

    • Ports 13-16

    • Ports 17-20

    • Ports 21-24

  6. Create the channel.

    In order to create the channel manually, use the set port channel mod/port on command for each switch. Turn the ports off on one side of the channel with use of the set port disable command before you turn EtherChannel on manually. This avoids possible problems with STP during the configuration process. STP can shut down some ports (with a port status of errdisable) if one side is configured as a channel before the other side can be configured as a channel. Because of this possibility, the creation of EtherChannels with use of PAgP is much easier. The Use PAgP to Configure EtherChannel (Recommended) section of this document covers the procedure. In order to avoid this situation when you configure EtherChannel manually, you disable the ports on Switch A, configure the channel on Switch A, configure the channel on Switch B, and then reenable the ports on Switch A.

    1. Verify that channelling is off.

      Switch-A> (enable) show port channel
      No ports channelling
      
      Switch-B> (enable) show port channel
      No ports channelling
    2. Disable the ports on Switch A until both switches have been configured for EtherChannel.

      Switch-A> (enable) set port disable 2/1-4
      Ports 2/1-4 disabled.
      
      [output from Switch A upon disabling ports]
      1999 DEC 15 00:06:40 %PAGP-5-PORTFROMSTP:Port 2/1 left bridg1
      1999 DEC 15 00:06:40 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
      1999 DEC 15 00:06:40 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
      1999 DEC 15 00:06:40 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4

      Now, STP does not generate errors and shut down the ports.

    3. Turn the channel mode to on for Switch A.

      Switch-A> (enable) set port channel 2/1-4 on
      Port(s) 2/1-4 channel mode set to on.

      Note: In this case, ports 2/1 to 2/4 are configured for EtherChannel with a single command. If you configure the EtherChannel for every port independently without use of the port range, remember to mention the same admin group for all the ports that need to be part of the same EtherChannel. If you do not specify the admin group, each port belongs to a different EtherChannel group and the desired EtherChannel bundle is never formed.

    4. Check the status of the channel.

      Switch-A> (enable) show port channel
      Port  Status     Channel   Channel     Neighbor                  Neighbor
                       mode      status      device                    port
      ----- ---------- --------- ----------- ------------------------- ---------- 
       2/1  disabled   on        channel    
       2/2  disabled   on        channel   
       2/3  disabled   on        channel    
       2/4  disabled   on        channel   
      ----- ---------- --------- ----------- ------------------------- ----------

      Notice that the channel mode has been set to on, but the status of the ports is disabled (because you disabled the ports earlier). The channel is not operational at this point, but the channel becomes operational when the ports are enabled.

      Because the Switch A ports were (temporarily) disabled, the Switch B ports no longer have a connection. This message is displayed on the Switch B console when Switch A ports are disabled:

      Switch-B> (enable)
      2000 Jan 13 22:30:03 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1
      2000 Jan 13 22:30:04 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
      2000 Jan 13 22:30:04 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
      2000 Jan 13 22:30:04 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
    5. Turn on the channel for Switch B.

      Switch-B> (enable) set port channel 2/1-4 on
      Port(s) 2/1-4 channel mode set to on.
    6. Verify that the channel mode is on for Switch B.

      Switch-B> (enable) show port channel
      Port  Status     Channel   Channel     Neighbor                  Neighbor
                       mode      status      device                    port
      ----- ---------- --------- ----------- ------------------------- ---------- 
       2/1  notconnect on        channel    
       2/2  notconnect on        channel   
       2/3  notconnect on        channel    
       2/4  notconnect on        channel   
      ----- ---------- --------- ----------- ------------------------- ----------

      Notice that the channel mode for Switch B is on, but the status of the ports is notconnect. This is the case because the Switch A ports are still disabled.

    7. Enable the ports on Switch A.

      Switch-A> (enable) set port enable 2/1-4
      Ports 2/1-4 enabled.
      1999 DEC 15 00:08:40 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
      1999 DEC 15 00:08:40 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
      1999 DEC 15 00:08:40 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
      1999 DEC 15 00:08:40 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Verify the EtherChannel Configuration

In order to verify that the channel is set up properly, issue the show port channel command.

Switch-A> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  on        channel     WS-C5505    066509957(SW  2/1       
 2/2  connected  on        channel     WS-C5505    066509957(SW  2/2       
 2/3  connected  on        channel     WS-C5505    066509957(SW  2/3       
 2/4  connected  on        channel     WS-C5505    066509957(SW  2/4       
----- ---------- --------- ----------- ------------------------- ----------

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  on        channel     WS-C5505    066507453(SW  2/1       
 2/2  connected  on        channel     WS-C5505    066507453(SW  2/2       
 2/3  connected  on        channel     WS-C5505    066507453(SW  2/3       
 2/4  connected  on        channel     WS-C5505    066507453(SW  2/4       
----- ---------- --------- ----------- ------------------------- ----------

If you have the output of a show port channel command from your Cisco device, you can use the Output Interpreter Tool (registered customers only) , which allows you to view an analysis of show command output.

The show spantree command shows that STP treats the ports as one logical port. This output lists the port as 2/1-4, which means that STP treats ports 2/1, 2/2, 2/3, and 2/4 as one port.

Switch-A> (enable) show spantree
VLAN 1
Spanning tree enabled
Spanning tree type          ieee

Designated Root             00-10-0d-b2-8c-00
Designated Root Priority    32768
Designated Root Cost        8
Designated Root Port        2/1-4
Root Max Age   20 sec    Hello Time 2  sec   Forward Delay 15 sec

Bridge ID MAC ADDR          00-90-92-b0-84-00
Bridge ID Priority          32768
Bridge Max Age 20 sec    Hello Time 2  sec   Forward Delay 15 sec

Port      Vlan  Port-State     Cost   Priority  Fast-Start  Group-Method
--------- ----  -------------  -----  --------  ----------  ------------
 2/1-4    1     forwarding         8        32   disabled       channel

If you have the output of a show spantree command from your Cisco device, you can use the Output Interpreter Tool (registered customers only) , which allows you to view an analysis of show command output.

EtherChannel can be implemented with different methods of traffic distribution across the ports in a channel. The EtherChannel specification does not dictate how the traffic distribution must occur across the links in a channel. The Catalyst 5500/5000 uses the last bit or the last two bits (which depends on how many links are in the channel) of the source and destination MAC addresses in the frame in order to determine which port in the channel to use. You should see a similar amount of traffic on each of the ports in the channel, if that traffic is generated by a normal distribution of MAC addresses on one side of the channel or the other. In order to verify that traffic goes over all the ports in the channel, you can use the show mac command. If your ports were active before the configuration of EtherChannel, you can reset the traffic counters to 0 with the clear counters command. The traffic values then represent how EtherChannel has distributed the traffic.

In this test environment, a real-world distribution is not achieved because there are no workstations, servers, or routers that generate traffic. The only devices that generate traffic are the switches themselves. Pings were issued from Switch A to Switch B. The unicast traffic uses the first port in the channel, as the output below shows. The receive information (Rcv-Unicast) in this case shows how Switch B distributed the traffic across the channel to Switch A. Also in the output, the transmit information (Xmit-Unicast) shows how Switch A distributed the traffic across the channel to Switch B. You also see that a small amount of switch-generated multicast traffic (Dynamic Inter-Switch Link Protocol [ISL], Cisco Discovery Protocol [CDP]) goes out all four ports. The broadcast packets are Address Resolution Protocol (ARP) queries (for the default gateway which does not exist in this lab). If you had workstations that send packets through the switch to a destination on the other side of the channel, you would expect to see that traffic goes over each of the four links in the channel. You can monitor the packet distribution in your network with use of the show mac command.

Switch-A> (enable) clear counters
This command will reset all MAC and port counters reported in CLI and SNMP.
Do you want to continue (y/n) [n]? y
MAC and Port counters cleared.
Switch-A> (enable) show mac
Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
-------- -------------------- -------------------- --------------------
 2/1                        9                  320                  183
 2/2                        0                   51                    0
 2/3                        0                   47                    0
 2/4                        0                   47                    0
(...)

Port     Xmit-Unicast         Xmit-Multicast       Xmit-Broadcast
-------- -------------------- -------------------- --------------------
 2/1                        8                   47                  184
 2/2                        0                   47                    0
 2/3                        0                   47                    0
 2/4                        0                   47                    0
(...)

Port     Rcv-Octet            Xmit-Octet
-------- -------------------- --------------------
 2/1                    35176                17443
 2/2                     5304                 4851
 2/3                     5048                 4851
 2/4                     5048                 4851
(...)

Last-Time-Cleared
--------------------------
Wed DEC 15 1999, 01:05:33

If you have the output of a show mac command from your Cisco device, you can use the Output Interpreter Tool (registered customers only) , which allows you to view an analysis of show command output.

Use PAgP to Configure EtherChannel (Recommended)

PAgP facilitates the automatic creation of EtherChannel links through the exchange of packets between channel-capable ports. The protocol learns the capabilities of port groups dynamically and informs the neighboring ports.

After PAgP identifies correctly paired channel-capable links, PAgP groups the ports into a channel. The channel is then added to the spanning tree as a single bridge port. A given outbound broadcast or multicast packet is transmitted out one port in the channel only, not out every port in the channel. In addition, outbound broadcast and multicast packets that are transmitted on one port in a channel are blocked so that the packets cannot return on any other port of the channel.

There are four user-configurable channel modes:

  • on

  • off

  • auto

  • desirable

PAgP packets are exchanged only between ports in auto and desirable mode. Ports that are configured in on or off mode do not exchange PAgP packets. For switches to which you want to form an EtherChannel, have both switches set to desirable mode. This setting gives the most robust behavior if one side or the other encounters error situations or is reset. The default mode of the channel is auto.

Both the auto and desirable modes allow ports to negotiate with connected ports in order to determine if the ports can form a channel. The determination is based on criteria such as port speed, trunking state, and native VLAN.

Ports can form an EtherChannel when they are in different channel modes as long as the modes are compatible. This list provides examples:

  • A port in desirable mode can successfully form an EtherChannel with another port that is in desirable or auto mode.

  • A port in auto mode can form an EtherChannel with another port in desirable mode.

  • A port in auto mode cannot form an EtherChannel with another port that is also in auto mode because neither port initiates negotiation.

  • A port in on mode can form a channel only with a port in on mode because ports in on mode do not exchange PAgP packets.

  • A port in off mode cannot form a channel with any port.

If this message (or a similar syslog message) is displayed when you use EtherChannel, the message indicates a mismatch of EtherChannel modes on the connected ports:

SPANTREE-2: Channel misconfig - x/x-x will be disabled

Issue the set port enable command in order to correct the configuration and reenable the ports. Valid EtherChannel configurations include:

Port Channel Mode Valid Neighbor Port Channel Modes
desirable desirable or auto
auto (default) desirable or auto1
on on
off off

1 If both the local and neighbor ports are in auto mode, an EtherChannel bundle does not form.

The next table provides a summary of all the possible channelling mode scenarios. Some of these combinations can cause STP to put the ports on the channelling side in errdisable state. In other words, some of the combinations shut down the ports on the channelling side.

Switch A Channel Mode Switch B Channel Mode Switch A Channel State Switch B Channel State
on on Channel (non-PAgP) Channel (non-PAgP)
on off Not channel (errdisable) Not channel
on auto Not channel (errdisable) Not channel
on desirable Not channel (errdisable) Not channel
off on Not channel Not channel (errdisable)
off off Not channel Not channel
off auto Not channel Not channel
off desirable Not channel Not channel
auto on Not channel Not channel (errdisable)
auto off Not channel Not channel
auto auto Not channel Not channel
auto desirable Channel (PAgP) Channel (PAgP)
desirable on Not channel Not channel (errdisable)
desirable off Not channel Not channel
desirable auto Channel (PAgP) Channel (PAgP)
desirable desirable Channel (PAgP) Channel (PAgP)

You turn off the channel from the example in Step 6b of the section Manually Configure EtherChannel if you issue this command on Switch A and Switch B:

Switch-A> (enable) set port channel 2/1-4 auto
Port(s) 2/1-4 channel mode set to auto.

The default channel mode for a port that is able to channel is auto. In order to verify this, issue this command:

Switch-A> (enable) show port channel 2/1
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  auto      not channel

The show port channel port command also shows that the ports currently are not channelling. This command provides another way to verify the channel state:

Switch-A> (enable) show port channel
No ports channelling

Switch-B> (enable) show port channel
No ports channelling

You can easily make the channel work with PAgP. At this point, both switches are set to auto mode, which means that they channel if a connected port sends a PAgP request to channel. If you set Switch A to desirable, Switch A sends PAgP packets to the other switch, asking it to channel.

Switch-A> (enable) set port channel 2/1-4 desirable
Port(s) 2/1-4 channel mode set to desirable.
1999 DEC 15 22:03:18 %PAGP-5-PORTFROMSTP:Port 2/1 left bridg1
1999 DEC 15 22:03:18 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
1999 DEC 15 22:03:18 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
1999 DEC 15 22:03:18 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
1999 DEC 15 22:03:19 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
1999 DEC 15 22:03:19 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
1999 DEC 15 22:03:20 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
1999 DEC 15 22:03:23 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
1999 DEC 15 22:03:23 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
1999 DEC 15 22:03:23 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
1999 DEC 15 22:03:24 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

In order to view the channel, issue this command:

Switch-A> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  desirable channel     WS-C5505    066509957(SW  2/1       
 2/2  connected  desirable channel     WS-C5505    066509957(SW  2/2       
 2/3  connected  desirable channel     WS-C5505    066509957(SW  2/3       
 2/4  connected  desirable channel     WS-C5505    066509957(SW  2/4       
----- ---------- --------- ----------- ------------------------- ---------- 
Switch-A> (enable)

Because Switch B is in auto mode, Switch B responds to the PAgP packets and creates a channel with Switch A.

Switch-B> (enable)
2000 Jan 14 20:26:41 %PAGP-5-PORTFROMSTP:Port 2/1 left bridg1
2000 Jan 14 20:26:41 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
2000 Jan 14 20:26:41 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
2000 Jan 14 20:26:41 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
2000 Jan 14 20:26:45 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
2000 Jan 14 20:26:45 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
2000 Jan 14 20:26:45 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
2000 Jan 14 20:26:47 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
2000 Jan 14 20:26:47 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
2000 Jan 14 20:26:47 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
2000 Jan 14 20:26:48 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  auto      channel     WS-C5505    066507453(SW  2/1       
 2/2  connected  auto      channel     WS-C5505    066507453(SW  2/2       
 2/3  connected  auto      channel     WS-C5505    066507453(SW  2/3       
 2/4  connected  auto      channel     WS-C5505    066507453(SW  2/4       
----- ---------- --------- ----------- ------------------------- ---------- 
Switch-B> (enable)

Note: It is best to set both sides of the channel to desirable so that both sides try to initiate the channel if one side drops out. If you set the EtherChannel ports on Switch B to desirable mode, even though the channel is currently active and in auto mode, it poses no problem. The command is:

Switch-B> (enable) set port channel 2/1-4 desirable
Port(s) 2/1-4 channel mode set to desirable.

Note: In this case, ports 2/1 to 2/4 are configured for EtherChannel with a single command. If you configure the EtherChannel for every port independently without use of the port range, remember to mention the same admin group for all the ports that need to be part of the same EtherChannel. If you do not specify the admin group, each port belongs to a different EtherChannel group and the desired EtherChannel bundle is never formed.

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  desirable channel     WS-C5505    066507453(SW  2/1       
 2/2  connected  desirable channel     WS-C5505    066507453(SW  2/2       
 2/3  connected  desirable channel     WS-C5505    066507453(SW  2/3       
 2/4  connected  desirable channel     WS-C5505    066507453(SW  2/4       
----- ---------- --------- ----------- ------------------------- ---------- 
Switch-B> (enable)

If Switch A drops out for some reason, or if new hardware replaces Switch A, Switch B tries to reestablish the channel. If the new equipment cannot channel, Switch B treats its ports 2/1-4 as normal nonchannelling ports. This is one of the benefits of use of the desirable mode. If you configure the channel with use of the PAgP on mode and one side of the connection has an error of some kind or a reset, an errdisable state (shutdown) results on the other side. With PAgP set in desirable mode on each side, the channel stabilizes and renegotiates the EtherChannel connection.

Silent/Non-Silent Mode

When you deal with fiber connections, there is a possibility that, even if a receive (Rx) transceiver dies, the transmit (Tx) transceiver on the other end is still up. In a similar scenario, packets can get black holed.

It is important for the switch that transmits to remove this port from the EtherChannel bundle. In order to do so on the Catalyst 5500/5000, you set PAgP in non-silent mode. Non-silent mode means that, if the Rx does not receive traffic, the port is not put into the channel. However, use of non-silent mode is not enough because this detection happens only when the channel is formed.

In order to prevent the black holing of traffic when the channel is already formed, this occurs:

  1. PAgP detects that the Rx port does not receive any traffic.

  2. PAgP resets the Tx transceiver of the port that does not receive traffic. PAgP resets it for 1.6 seconds so that the switch on the other end also resets the port.

  3. The faulty port does not join the channel anymore because no traffic is received on that port.

On the Catalyst 5500/5000, set non-silent mode on fiber strands and silent mode on copper strands. This is both the default and recommended setting because, on fiber connections on the Catalyst 5500/5000, the negotiation is usually not available, so there is no way to detect the problem at a physical layer.

Default PAgP Settings on the Catalyst 4500/4000 and 5500/5000

By default, PAgP is auto for a plug-and-play implementation. Disable PAgP manually from the ports where there is no need to have it.

By default, the silent mode is on. Non-silent is acceptable as well. However, because a port can be connected to a device that does not send traffic (for example, a sniffer), it is more general to have silent enabled.

Recommendations

  • Use the non-silent keyword when you connect to a device that transmits bridge protocol data units (BPDUs) or other traffic. Use this keyword with the auto or desirable mode. PAgP non-silent adds an extra level of link state detection because it listens for BPDUs or other traffic in order to determine if the link functions properly. This adds a form of UniDirectional Link Detection (UDLD) capability that is not available when you use the default silent PAgP mode.

  • Use the silent keyword when you connect to a silent partner (which is a device that does not generate BPDUs or other traffic). An example of a silent partner is a traffic generator that does not transmit packets. Use the silent keyword with auto or desirable mode. If you do not specify silent or non-silent, silent is assumed.

  • The silent mode does not disable the PAgP ability to detect unidirectional links. However, when you configure a channel, non-silent prevents a unidirectional port from even joining the link.

  • A PAgP configuration (the set port channel {desirable | auto} command) is safer than a non-PAgP configuration (the set port channel on command). A PAgP configuration provides protection for unidirectional links and also avoids misconfigurations that can arise when there are ports channeling on one side of the link and not on the other side.

  • Refer to Understanding and Configuring the Unidirectional Link Detection Protocol Feature for more information on UDLD.

Trunking and EtherChannel

EtherChannel is independent of trunking. You can turn trunking on or you can leave trunking off. Also, you can turn trunking on for all the ports before you create the channel, or you can turn trunking on after you create the channel (as in this example). In terms of EtherChannel, because trunking and EtherChannel are completely separate features, it does not matter when you turn trunking on. What does matter is that all the ports that are involved are in the same mode:

  • The ports are all trunking before you configure the channel

    or

  • The ports are all not trunking before you configure the channel

All the ports must be in the same trunking state before you create the channel.

After a channel is formed, whatever is changed on one port is also changed for the other ports in the channel. The modules that are used in this test bed can do ISL or IEEE 802.1Q trunking. By default, the modules are set to auto trunking and negotiate mode. This means that the ports trunk if the other side asks them to trunk, and they negotiate whether to use the ISL or 802.1Q method for trunking. If they are not asked to trunk, the ports work as normal nontrunking ports.

Switch-A> (enable) show trunk 2
Port      Mode         Encapsulation  Status        Native vlan
--------  -----------  -------------  ------------  -----------
 2/1      auto         negotiate      not-trunking  1
 2/2      auto         negotiate      not-trunking  1
 2/3      auto         negotiate      not-trunking  1
 2/4      auto         negotiate      not-trunking  1

There are a number of different ways to turn on trunking. For this example, Switch A is set to desirable. Switch A is already set to negotiate. The combination of desirable/negotiate causes Switch A to ask Switch B to trunk and to negotiate the type of trunking to perform (ISL or 802.1Q). Since Switch B defaults to autonegotiate, Switch B responds to the Switch A request. These are the results:

Switch-A> (enable) set trunk 2/1 desirable
Port(s) 2/1-4 trunk mode set to desirable.

Switch-A> (enable)
1999 DEC 18 20:46:25 %DTP-5-TRUNKPORTON:Port 2/1 has become isl trunk
1999 DEC 18 20:46:25 %DTP-5-TRUNKPORTON:Port 2/2 has become isl trunk
1999 DEC 18 20:46:25 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1-4
1999 DEC 18 20:46:25 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/1-4
1999 DEC 18 20:46:25 %DTP-5-TRUNKPORTON:Port 2/3 has become isl trunk
1999 DEC 18 20:46:26 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/1-4
1999 DEC 18 20:46:26 %DTP-5-TRUNKPORTON:Port 2/4 has become isl trunk
1999 DEC 18 20:46:26 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/1-4
1999 DEC 18 20:46:28 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
1999 DEC 18 20:46:29 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
1999 DEC 18 20:46:29 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
1999 DEC 18 20:46:29 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Switch-A> (enable) show trunk 2
Port      Mode         Encapsulation  Status        Native vlan
--------  -----------  -------------  ------------  -----------
 2/1      desirable    n-isl          trunking      1
 2/2      desirable    n-isl          trunking      1
 2/3      desirable    n-isl          trunking      1
 2/4      desirable    n-isl          trunking      1

The trunk mode was set to desirable. The result was that trunking mode was negotiated with the neighbor switch, and the switches decided on ISL (n-isl). The current status now is trunking. This output shows what happened on Switch B because of the command that was issued on Switch A:

Switch-B> (enable) 
2000 Jan 17 19:09:52 %DTP-5-TRUNKPORTON:Port 2/1 has become isl trunk
2000 Jan 17 19:09:52 %DTP-5-TRUNKPORTON:Port 2/2 has become isl trunk
2000 Jan 17 19:09:52 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1-4
2000 Jan 17 19:09:52 %DTP-5-TRUNKPORTON:Port 2/3 has become isl trunk
2000 Jan 17 19:09:52 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/1-4
2000 Jan 17 19:09:53 %DTP-5-TRUNKPORTON:Port 2/4 has become isl trunk
2000 Jan 17 19:09:53 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/1-4
2000 Jan 17 19:09:53 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/1-4
2000 Jan 17 19:09:55 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
2000 Jan 17 19:09:55 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
2000 Jan 17 19:09:55 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
2000 Jan 17 19:09:55 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Switch-B> (enable) show trunk 2
Port      Mode         Encapsulation  Status        Native vlan
--------  -----------  -------------  ------------  -----------
 2/1      auto         n-isl          trunking      1
 2/2      auto         n-isl          trunking      1
 2/3      auto         n-isl          trunking      1
 2/4      auto         n-isl          trunking      1

Notice that all four ports (2/1-4) became trunking, even though you only specifically changed one port (2/1) to desirable. This is an example of how a change of one port in the channel affects all the ports.

Note: You must undestand that EtherChannel combines or bundles multiple links into single logical link, so it is not possible to send data through a dedicated link.

Disable EtherChannel

If you want to disable an EtherChannel or you do not want ports to participate in the EtherChannel negotiation, you can turn off the EtherChannel. Here is an example:

Switch-A> (enable) set port channel 2/1-4 off
Port(s) 2/1-4 channel mode set to off.

If the ports of Switch B are configured in either the auto mode or desirable mode, the channel is not formed. If the ports of Switch B are configured as on, the ports go into the errdisable state after a few minutes. See the Waiting Too Long Before You Configure the Other Side section of this document in order to recover the ports from this state. For more information about the errdisable state, refer to Recovering From errDisable Port State on the CatOS Platforms.

The default port channel mode for the switch ports is auto. If you turn off the EtherChannel on any ports, you see the set port channel 2/1-4 off command in the switch configuration. Here is sample output that shows this command in the switch configuration:

Switch-A> (enable)show config

!--- Output suppressed.

#module 2 : 24-port 10/100BaseTX Ethernet
set port channel 2/1-4 off

If you want to reset the port channel configuration to the default settings, you can configure the port channel mode to auto. Here is an example:

Switch-A> (enable) set port channel 2/1-4 auto
Port(s) 2/1-4 channel mode set to auto.

Now the set port channel command does not appear in the switch configuration.

Troubleshoot EtherChannel

The challenges for EtherChannel can be divided into two main areas:

  • Troubleshooting during the configuration phase

  • Troubleshooting during the execution phase

Configuration errors usually occur because of mismatched parameters on the ports that are involved (for example, different speeds, different duplex, or different STP port values). However, you can also generate errors during the configuration if you set the channel on one side to on and wait too long before you configure the channel on the other side. This causes STP loops which generate an error and shut down the port.

When you encounter an error during the configuration of EtherChannel, be sure to check the status of the ports after you correct the EtherChannel error situation. If the port status is errdisable, this status indicates that the software has shut down the ports. The ports do not come on again until you issue the set port enable command.

Note: If the port status becomes errdisable, you must specifically enable the ports with use of the set port enable command in order for the ports to become active. Currently, you can correct all the EtherChannel issues, but the ports do not come up or form a channel until the ports are enabled again. Later versions of the operating system may periodically check in order to determine if errdisable ports should be enabled.

These tests are covered in this section. For the tests, trunking and EtherChannel are turned off:

Mismatched Parameters

Here is an example of mismatched parameters. Port 2/4 is set in VLAN 2 while the other ports are still in VLAN 1. In order to create a new VLAN, you must assign a VLAN Trunk Protocol (VTP) domain for the switch and then create the VLAN.

Switch-A> (enable) show port channel
No ports channelling

Switch-A> (enable) show port
Port  Name               Status     Vlan       Level  Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
 2/1                     connected  1          normal a-full a-100 10/100BaseTX
 2/2                     connected  1          normal a-full a-100 10/100BaseTX
 2/3                     connected  1          normal a-full a-100 10/100BaseTX
 2/4                     connected  1          normal a-full a-100 10/100BaseTX

Switch-A> (enable) set vlan 2
Cannot add/modify VLANs on a VTP server without a domain name.

Switch-A> (enable) set vtp domain testDomain
VTP domain testDomain modified

Switch-A> (enable) set vlan 2 name vlan2
Vlan 2 configuration successful

Switch-A> (enable) set vlan 2 2/4
VLAN 2 modified.
VLAN 1 modified.
VLAN  Mod/Ports
---- -----------------------
2     2/4
      
Switch-A> (enable)
1999 DEC 19 00:19:34 %PAGP-5-PORTFROMSTP:Port 2/4 left bridg4

Switch-A> (enable) show port
Port  Name               Status     Vlan       Level  Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
 2/1                     connected  1          normal a-full a-100 10/100BaseTX
 2/2                     connected  1          normal a-full a-100 10/100BaseTX
 2/3                     connected  1          normal a-full a-100 10/100BaseTX
 2/4                     connected  2          normal a-full a-100 10/100BaseTX

Switch-A> (enable) set port channel 2/1-4 desirable
Port(s) 2/1-4 channel mode set to desirable.

Switch-A> (enable)
1999 DEC 19 00:20:19 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1
1999 DEC 19 00:20:19 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
1999 DEC 19 00:20:19 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
1999 DEC 19 00:20:20 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
1999 DEC 19 00:20:20 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
1999 DEC 19 00:20:22 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
1999 DEC 19 00:20:22 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
1999 DEC 19 00:20:24 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-2
1999 DEC 19 00:20:25 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-2
1999 DEC 19 00:20:25 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/3
1999 DEC 19 00:20:25 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/4

Switch-A> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  desirable channel     WS-C5505    066509957(SW  2/1       
 2/2  connected  desirable channel     WS-C5505    066509957(SW  2/2       
----- ---------- --------- ----------- ------------------------- ---------- 
Switch-A> (enable)

Notice that the channel formed only between ports 2/1-2. Ports 2/3-4 were left out because port 2/4 is in a different VLAN. There was no error message; PAgP just did what it could do to make the channel work. Watch the results when you create the channel to be sure that the results are what you expected.

Now, set the channel manually to on with port 2/4 in a different VLAN and see what happens. First, set the channel mode back to auto. This tears down the existing channel. Then, manually set the channel to on.

Switch-A> (enable) set port channel 2/1-4 auto
Port(s) 2/1-4 channel mode set to auto.
Switch-A> (enable)
1999 DEC 19 00:26:08 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1-2
1999 DEC 19 00:26:08 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/1-2
1999 DEC 19 00:26:08 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
1999 DEC 19 00:26:08 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
1999 DEC 19 00:26:18 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1
1999 DEC 19 00:26:19 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/2
1999 DEC 19 00:26:19 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/3
1999 DEC 19 00:26:19 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/4

Switch-A> (enable) show port channel
No ports channelling

Switch-A> (enable) set port channel 2/1-4 on
Mismatch in vlan number.
Failed to set port(s) 2/1-4 channel mode to on.

Switch-A> (enable) show port channel
No ports channelling

On Switch B, when you turn the channel on, it indicates that the ports are channelling fine. You know, however, that Switch A is not configured correctly.

Switch-B> (enable) show port channel
No ports channelling

Switch-B> (enable) show port
Port  Name               Status     Vlan       Level  Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
 2/1                     connected  1          normal a-full a-100 10/100BaseTX
 2/2                     connected  1          normal a-full a-100 10/100BaseTX
 2/3                     connected  1          normal a-full a-100 10/100BaseTX
 2/4                     connected  1          normal a-full a-100 10/100BaseTX

Switch-B> (enable) set port channel 2/1-4 on
Port(s) 2/1-4 channel mode set to on.

Switch-B> (enable)
2000 Jan 17 22:54:59 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1
2000 Jan 17 22:54:59 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/2
2000 Jan 17 22:54:59 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/3
2000 Jan 17 22:54:59 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/4
2000 Jan 17 22:55:00 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
2000 Jan 17 22:55:00 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
2000 Jan 17 22:55:00 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
2000 Jan 17 22:55:00 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  on        channel     WS-C5505    066507453(SW  2/1       
 2/2  connected  on        channel     WS-C5505    066507453(SW  2/2       
 2/3  connected  on        channel     WS-C5505    066507453(SW  2/3       
 2/4  connected  on        channel     WS-C5505    066507453(SW  2/4       
----- ---------- --------- ----------- ------------------------- ----------

You must check both sides of the channel when you manually configure the channel in order to ensure that both sides, not just one side, are up. The above output shows that Switch B is set for a channel, but Switch A is not channelling because Switch A has one port that is in the wrong VLAN.

Waiting Too Long Before You Configure the Other Side

In this situation, Switch B has EtherChannel turned on, but Switch A does not have EtherChannel turned on because the switch has a VLAN configuration error. Ports 2/1-3 are in VLAN 1, and port 2/4 is in VLAN 2. When one side of an EtherChannel is set to on while the other side is still in auto mode, these events occur:

  1. After a few minutes, Switch B shuts down its ports because of a spanning loop detection. This occurs because Switch B ports 2/1-4 all act as one large port while Switch A ports 2/1-4 are all completely independent ports.

  2. A broadcast that is sent from Switch B to Switch A on port 2/1 is sent back to Switch B on ports 2/2, 2/3, and 2/4 because Switch A treats these ports as independent ports.

  3. Switch B interprets this as a spanning tree loop. Notice that the ports on Switch B are now disabled and have a status of errdisable:

    Switch-B> (enable)
    2000 Jan 17 22:55:48 %SPANTREE-2-CHNMISCFG: STP loop - channel 2/1-4 is disabled 
     in vlan 1.
    2000 Jan 17 22:55:49 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1-4
    2000 Jan 17 22:56:01 %PAGP-5-PORTFROMSTP:Port 2/2 left bridge port 2/1-4
    2000 Jan 17 22:56:13 %PAGP-5-PORTFROMSTP:Port 2/3 left bridge port 2/1-4
    2000 Jan 17 22:56:36 %PAGP-5-PORTFROMSTP:Port 2/4 left bridge port 2/1-4
    
    Switch-B> (enable) show port channel
    Port  Status     Channel   Channel     Neighbor                  Neighbor
                     mode      status      device                    port
    ----- ---------- --------- ----------- ------------------------- ---------- 
     2/1  errdisable on        channel    
     2/2  errdisable on        channel    
     2/3  errdisable on        channel    
     2/4  errdisable on        channel    
    ----- ---------- --------- ----------- ------------------------- ----------
    
    Switch-B> (enable) show port
    Port  Name               Status     Vlan       Level  Duplex Speed Type
    ----- ------------------ ---------- ---------- ------ ------ ----- ------------
     2/1                     errdisable 1          normal   auto  auto 10/100BaseTX
     2/2                     errdisable 1          normal   auto  auto 10/100BaseTX
     2/3                     errdisable 1          normal   auto  auto 10/100BaseTX
     2/4                     errdisable 1          normal   auto  auto 10/100BaseTX
    

Correct the errdisable State

Sometimes, when you try to configure EtherChannel but the ports are not configured in the same way, the ports on one side of the channel shut down. The link lights are yellow on the port. The console indicates this in the show port command output, in which the ports are listed as errdisable. In order to recover, fix the mismatched parameters on the ports that are involved, and then reenable the ports.

Note: The reenablement of the ports is a separate step that you must do if the ports are to become functional again.

In this example, you know that Switch A had a VLAN mismatch. On Switch A, put port 2/4 back into VLAN 1. Then turn on the channel for ports 2/1-4. Switch A does not show that it is connected until you reenable the Switch B ports. After you have fixed Switch A and put it in channelling mode, go back to Switch B and reenable the ports.

Switch-A> (enable) set vlan 1 2/4
VLAN 1 modified.
VLAN 2 modified.
VLAN  Mod/Ports
---- -----------------------
1     2/1-24
      
Switch-A> (enable) set port channel 2/1-4 on
Port(s) 2/1-4 channel mode set to on.
Switch-A> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  notconnect on        channel    
 2/2  notconnect on        channel    
 2/3  notconnect on        channel    
 2/4  notconnect on        channel    
----- ---------- --------- ----------- ------------------------- ----------

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  errdisable on        channel    
 2/2  errdisable on        channel    
 2/3  errdisable on        channel    
 2/4  errdisable on        channel    
----- ---------- --------- ----------- ------------------------- ----------

Switch-B> (enable) set port enable 2/1-4
Ports 2/1-4 enabled.
Switch-B> (enable) 
2000 Jan 17 23:15:22 %PAGP-5-PORTTOSTP:Port 2/1 joined bridg4
2000 Jan 17 23:15:22 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/1-4
2000 Jan 17 23:15:22 %PAGP-5-PORTTOSTP:Port 2/3 joined bridge port 2/1-4
2000 Jan 17 23:15:22 %PAGP-5-PORTTOSTP:Port 2/4 joined bridge port 2/1-4

Switch-B> (enable) show port channel
Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ---------- 
 2/1  connected  on        channel    
 2/2  connected  on        channel    
 2/3  connected  on        channel    
 2/4  connected  on        channel    
----- ---------- --------- ----------- ------------------------- ----------

Show What Happens When a Link Breaks and Is Restored

When a port in the channel goes down, any packets that are normally sent on that port are shifted over to the next port in the channel. You can issue the show mac command in order to verify that this occurs. In this test bed, Switch A sends ping packets to Switch B in order to determine which link the traffic uses. The procedure is:

  1. Clear the counters.

  2. Issue the show mac command.

  3. Send three pings.

  4. Issue the show mac command again in order to determine on which channel the ping responses were received.

    Switch-A> (enable) clear counters
    This command will reset all MAC and port counters reported in CLI and SNMP.
    Do you want to continue (y/n) [n]? y
    MAC and Port counters cleared.
    
    Switch-A> (enable) show port channel
    Port  Status     Channel   Channel     Neighbor                  Neighbor
                     mode      status      device                    port
    ----- ---------- --------- ----------- ------------------------- ---------- 
     2/1  connected  on        channel     WS-C5505    066509957(SW  2/1       
     2/2  connected  on        channel     WS-C5505    066509957(SW  2/2       
     2/3  connected  on        channel     WS-C5505    066509957(SW  2/3       
     2/4  connected  on        channel     WS-C5505    066509957(SW  2/4       
    ----- ---------- --------- ----------- ------------------------- ----------
    
    Switch-A> (enable) show mac
    Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
    -------- -------------------- -------------------- --------------------
     2/1                        0                   18                    0
     2/2                        0                    2                    0
     2/3                        0                    2                    0
     2/4                        0                    2                    0
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) show mac
    Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
    -------- -------------------- -------------------- --------------------
     2/1                        3                   24                    0
     2/2                        0                    2                    0
     2/3                        0                    2                    0
     2/4                        0                    2                    0

    At this point, ping responses are received on port 3/1. When the Switch B console sends a response to Switch A, the EtherChannel uses port 2/1.

  5. Shut down port 2/1 on Switch B.

  6. From Switch A, issue another ping and determine on which channel the response comes back.

    Note: Switch A sends on the same port to which Switch B is connected. Only the received packets from Switch B are shown because the transmit packets appear later in the show mac command output.

    1999 DEC 19 01:30:23 %PAGP-5-PORTFROMSTP:Port 2/1 left bridge port 2/1-4
    
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) show mac
    Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
    -------- -------------------- -------------------- --------------------
     2/1                        3                   37                    0
     2/2                        1                   27                    0
     2/3                        0                    7                    0
     2/4                        0                    7                    0

    Now that port 2/1 is disabled, EtherChannel automatically uses the next port in the channel, 2/2.

  7. Reenable port 2/1, and wait for it to join the bridge group.

  8. Issue two more pings.

    1999 DEC 19 01:31:33 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1-4
    
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) ping 172.16.84.17
    172.16.84.17 is alive
    Switch-A> (enable) show mac
    Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
    -------- -------------------- -------------------- --------------------
     2/1                        5                   50                    0
     2/2                        1                   49                    0
     2/3                        0                   12                    0
     2/4                        0                   12                    0

    Note: These pings are sent from port 2/1. When the link comes back up, EtherChannel again adds it to the bundle and uses it. All this is done transparently to the user.

Connectivity Issue with Channel Down after Supervisor Replacement

The EtherChannel can go down if the correct procedure is not followed while you replace a Supervisor module and the connected device has errdisable enabled. This usually happens when cables are connected to the new Supervisor module before it is configured for EtherChannel. Therefore, the connected device configured for errdisable detects the port channel misconfiguration and puts its ports in the errdisable state. This causes the connectivity issue. The channel does not come on again until you issue the set port enable command on the connected device.

In order to avoid the port channel misconfiguration, always follow these steps when you replace a Supervisor module that has EtherChannel configurations:

  1. Unplug all the cables from the Supervisor that you want to replace.

  2. Replace the Supervisor with the new Supervisor.

  3. Configure the new Supervisor module for EtherChannel.

  4. Connect the cables.

Bandwidth Is Limited to 1 Gbps When WS-X6148-GE-TX Ports Are Used in the Channel

The WS-X6148-GE-TX and WS-X6148V-GE-TX modules do not support more than 1 Gbps of traffic per EtherChannel. On these modules, there is a single 1-Gigabit Ethernet uplink from the port application-specific integrated circuit (ASIC) that supports eight ports. For EtherChannel, the data from all the links in a bundle go to the port ASIC, even though the data are destined for another link. These data consume bandwidth in the 1-Gigabit Ethernet link. For these modules, the sum total of all data on an EtherChannel cannot exceed 1 Gbps. As a result, they should only be used in port channels for link redundancy purposes. If they are included in any Gigabit EtherChannels, the entire channel is limited to 1 Gbps of bandwidth. You also see a warning message that is similar to this one:

Adding a WS-X6148-GE-TX port to a channel limits the channel's bandwidth to a 
maximum of 1Gig throughput

Commands Used in This Document

Commands to Set the Configuration

  • set port channel on—Turns on the EtherChannel feature.

  • set port channel auto—Resets the ports to their default mode of auto.

  • set port channel desirable—Sends PAgP packets to the other side that request that a channel be created.

  • set port enable—Enables the ports after the set port disable command is issued or after an errdisable state.

  • set port disable—Disables a port during other configuration settings.

  • set trunk desirable—Turns on trunking by causing this port to send to the other switch a request that this be a trunk link. Also, if the port is set to negotiate (the default setting), it requests to negotiate the type of trunking to use on the link (ISL or 802.1Q).

Commands to Verify the Configuration

  • show version—Displays the version of software that the switch runs.

  • show module—Displays the modules that are installed in the switch.

  • show port capabilities—Determines if the ports that you want to use have EtherChannel capabilities.

  • show port—Determines the status of the port (notconnect or connected) as well as the speed and duplex settings.

  • ping—Tests connectivity to the other switch.

  • show port channel—Shows the current status of the EtherChannel bundle.

  • show port channel mod/port —Provides a more detailed view of the channel status of a single port.

  • show spantree—Verifies that STP viewed the channel as one link.

  • show trunk—Shows the trunking status of ports.

Commands to Troubleshoot the Configuration

  • show port channel—Shows the current status of the EtherChannel bundle.

  • show port—Determines the status of the port (notconnect or connected) as well as the speed and duplex settings.

  • clear counters—Resets the switch packet counters to 0. The counters are visible with the show mac command.

  • show mac—Shows packets that the switch receives and sends.

  • ping—Tests connectivity to the other switch and generates traffic that appears in the show mac command output.

Commands to Help Create the Troubleshooting Scenarios

  • set vtp domain testDomain—Gives the switch a VTP domain, which is required in order to add VLANs on the switch.

  • set vlan 2 name vlan2—Creates VLAN 2 with a name of "vlan2".

  • set vlan 2 2/4—Moves port 2/4 into VLAN 2.

  • set port channel 2/1-4 desirable—Sends PAgP packets to the other side that request the creation of a channel.

  • set port channel 2/1-4 auto—Resets the ports to their default mode of auto.

  • set port channel 2/1-4 on—Sets the channel mode of these ports to on. No PAgP packets are sent to the other side. This side simply assumes that the other side has formed a channel as well.

  • set vlan 1 2/4—Moves port 2/4 into VLAN 1.

Command Summary

Because this document uses CatOS software version 4.5, the command syntax is taken from the Command Reference for Cisco Catalyst 5000 Series Switches.

Syntax: show version
As used in this document: show version
Syntax: show module [mod_num]
As used in this document: show module
Syntax: show port capabilities [mod_num[/port_num]]
As used in this document: show port capabilities
Syntax: show port [mod_num[/port_num]]
As used in this document: show port
Syntax: ping [-s] host[packet_size][packet_count]
As used in this document: ping 172.16.84.17
Syntax: show port channel [mod] [info | statistics] show port channel mod/port [info | statistics]
As used in this document: show port channel show port channel 2/1
Syntax: set port disable mod_num/port_num
As used in this document: set port disable 2/1-4
Syntax: set port channel mod/ports... [on | off | desirable | auto] set port channel admin_group [on | off | desirable | auto] set port channel admin_group mod/ports..[on | off | desirable | auto]
As used in this document: set port channel 2/1-4 on set port channel 2/1-4 auto set port channel 2/1-4 desirable
Syntax: set port enable mod_num/port_num
As used in this document: set port enable 2/1-4
Syntax: show spantree [vlan | mod_num/port_num] [active]
As used in this document: show spantree
Syntax: show trunk [mod_num[/port_num]] [detail]
As used in this document: show trunk 2
Syntax: set trunk mod_num/port_num [on | off | desirable | auto | nonegotiate] [vlan_range] [isl | dot1q | dot10 | lane | negotiate]
As used in this document: set trunk 2/1 desirable
Syntax: set vtp [domain domain_name] [mode {client | server | transparent}] [passwd passwd][pruning {enable | disable}] [v2 {enable | disable}]
As used in this document: set vtp domain testDomain
Syntax: set vlan vlan_num mod_num/port_list set vlan vlan_num [name name] [type {ethernet | fddi | fddinet | trcrf | trbrf}] [state {active | suspend}] [said said] [mtu mtu] [ring hex_ring_number ] [decring decimal_ring_number ] [bridge bridge_num] [parent vlan_num] [mode {srt | srb}] [stp {IEEE | ibm | auto}] [translation vlan_num] [backupcrf {off | on}] [aremaxhop hop_count] [stemaxhop hop_count]
As used in this document: set vlan 2 name vlan2 set vlan 2 2/4
Syntax: clear counters
As used in this document: clear counters
Syntax: show mac [mod_num[/port_num]]
As used in this document: show mac

Appendix A: Ethernet Crossover Cables

These cables are available from most computer stores. Also, you can make your own. These two images show the pinouts that are required for a switch-to-switch crossover cable:

6b.gif

6c.gif

Related Information

Updated: Aug 14, 2006
Document ID: 10024