Contents
- Configuring EtherChannels
- Information About EtherChannels
- Understanding EtherChannels
- Compatibility Requirements
- Load Balancing Using EtherChannels
- LACP Overview
- LACP ID Parameters
- Channel Modes
- LACP Marker Responders
- LACP-Enabled and Static EtherChannels Differences
- Configuring EtherChannels
- Creating an EtherChannel
- Adding a Port to an EtherChannel
- Configuring Load Balancing Using EtherChannels
- Configuring Hardware Hashing for Multicast Traffic
- Enabling LACP
- Configuring the Channel Mode for a Port
- Configuring the LACP Fast Timer Rate
- Configuring the LACP System Priority and System ID
- Configuring the LACP Port Priority
- Verifying EtherChannel Configuration
- Verifying the Load-Balancing Outgoing Port ID
Configuring EtherChannels
This chapter describes how to configure EtherChannels and to apply and configure the Link Aggregation Control Protocol (LACP) for more efficient use of EtherChannels in Cisco NX-OS. It contains the following sections:
- Information About EtherChannels
- Configuring EtherChannels
- Verifying EtherChannel Configuration
- Verifying the Load-Balancing Outgoing Port ID
Information About EtherChannels
An EtherChannel bundles up to 16 individual interfaces into a group to provide increased bandwidth and redundancy. Port channeling also load balances traffic across these physical interfaces. The EtherChannel stays operational as long as at least one physical interface within the EtherChannel is operational.
You create an EtherChannel by bundling compatible interfaces. You can configure and run either static EtherChannels or EtherChannels running the Link Aggregation Control Protocol (LACP).
Any configuration changes that you apply to the EtherChannel are applied to each member interface of that EtherChannel. For example, if you configure Spanning Tree Protocol (STP) parameters on the EtherChannel, the Cisco NX-OS applies those parameters to each interface in the EtherChannel.
You can use static EtherChannels, with no associated protocol, for a simplified configuration. For more efficient use of the EtherChannel, you can use the Link Aggregation Control Protocol (LACP), which is defined in IEEE 802.3ad. When you use LACP, the link passes protocol packets.
- Understanding EtherChannels
- Compatibility Requirements
- Load Balancing Using EtherChannels
- Understanding LACP
Understanding EtherChannels
Using EtherChannels, Cisco NX-OS provides wider bandwidth, redundancy, and load balancing across the channels.
You can collect up to 16 ports into a static EtherChannel or you can enable the Link Aggregation Control Protocol (LACP). Configuring EtherChannels with LACP requires slightly different steps than configuring static EtherChannels.
Note
Cisco NX-OS does not support Port Aggregation Protocol (PAgP) for EtherChannels.
An EtherChannel bundles individual links into a channel group to create a single logical link that provides the aggregate bandwidth of up to 16 physical links. If a member port within an EtherChannel fails, traffic previously carried over the failed link switches to the remaining member ports within the EtherChannel.
Each port can be in only one EtherChannel. All the ports in an EtherChannel must be compatible; they must use the same speed and operate in full-duplex mode. When you are running static EtherChannels, without LACP, the individual links are all in the on channel mode; you cannot change this mode without enabling LACP.
Note
You cannot change the mode from ON to Active or from ON to Passive.
You can create an EtherChannel directly by creating the port-channel interface, or you can create a channel group that acts to aggregate individual ports into a bundle. When you associate an interface with a channel group, Cisco NX-OS creates a matching EtherChannel automatically if the EtherChannel does not already exist. You can also create the EtherChannel first. In this instance, Cisco NX-OS creates an empty channel group with the same channel number as the EtherChannel and takes the default configuration.
Note
An EtherChannel is operationally up when at least one of the member ports is up and that port’s status is channeling. The EtherChannel is operationally down when all member ports are operationally down.
Compatibility Requirements
When you add an interface to a channel group, Cisco NX-OS checks certain interface attributes to ensure that the interface is compatible with the channel group. Cisco NX-OS also checks a number of operational attributes for an interface before allowing that interface to participate in the port-channel aggregation.
The compatibility check includes the following operational attributes:
Port mode
Access VLAN
Trunk native VLAN
Allowed VLAN list
Speed
802.3x flow control setting
MTU
The Cisco Nexus 5000 Series switch only supports system level MTU. This attribute cannot be changed on an individual port basis.
Broadcast/Unicast/Multicast Storm Control setting
Priority-Flow-Control
Untagged CoS
Use the show port-channel compatibility-parameters command to see the full list of compatibility checks that Cisco NX-OS uses.
You can only add interfaces configured with the channel mode set to on to static EtherChannels. You can also only add interfaces configured with the channel mode as active or passive to EtherChannels that are running LACP. You can configure these attributes on an individual member port.
When the interface joins an EtherChannel, the following individual parameters are replaced with the values on the EtherChannel:
The following interface parameters remain unaffected when the interface joins an EtherChannel:
Beginning with Cisco NX-OS Release 5.0(2)N2(1), after you enable forcing a port to be added to a channel group by entering the channel-group force command, the following two conditions occur:
Load Balancing Using EtherChannels
Cisco NX-OS load balances traffic across all operational interfaces in an EtherChannel by reducing part of the binary pattern formed from the addresses in the frame to a numerical value that selects one of the links in the channel. EtherChannels provide load balancing by default and the basic configuration uses the following criteria to select the link:
For a Layer 2 frame, it uses the source and destination MAC addresses.
For a Layer 3 frame, it uses the source and destination MAC addresses and the source and destination IP addresses.
For a Layer 4 frame, it uses the source and destination MAC addresses and the source and destination IP addresses.
Note
You have the option to include the source and destination port number for the Layer 4 frame.
You can configure the switch to use one of the following methods to load balance across the EtherChannel:
Destination MAC address
Source MAC address
Source and destination MAC address
Destination IP address
Source IP address
Source and destination IP address
Destination TCP/UDP port number
Source TCP/UDP port number
Source and destination TCP/UDP port number
Table 1 EtherChannel Load-Balancing Criteria Configuration
Layer 2 Criteria
Layer 3 Criteria
Layer 4 Criteria
Destination MAC
Destination MAC
Destination MAC
Destination MAC
Source MAC
Source MAC
Source MAC
Source MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Destination IP
Destination MAC
Destination MAC, destination IP
Destination MAC, destination IP
Source IP
Source MAC
Source MAC, source IP
Source MAC, source IP
Source and destination IP
Source and destination MAC
Source and destination MAC, source and destination IP
Source and destination MAC, source and destination IP
Destination TCP/UDP port
Destination MAC
Destination MAC, destination IP
Destination MAC, destination IP, destination port
Source TCP/UDP port
Source MAC
Source MAC, source IP
Source MAC, source IP, source port
Source and destination TCP/UDP port
Source and destination MAC
Source and destination MAC, source and destination IP
Source and destination MAC, source and destination IP, source and destination port
Fabric Extenders are not configurable individually. Fabric extender configurations are defined on the Nexus 5000 Series. In the case of the port-channel load balancing protocol, the table below illustrates which port-channel load balancing option is automatically configured on the fabric extender modules as a result of the configuration performed on the Nexus 5000 Series.
The following table shows the criteria used for each configuration:
Table 2 EtherChannel Load-Balancing Criteria for the Cisco Nexus 2232 and Cisco Nexus 2248 Fabric Extenders Configuration
Layer 2 Criteria
Layer 3 Criteria
Layer 4 Criteria
Destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Source and destination MAC
Destination IP
Source and destination MAC
Source and destination MAC, and source and destination IP
Source and destination MAC, and source and destination IP
Source IP
Source and destination MAC
Source and destination MAC, and source and destination IP
Source and destination MAC, and source and destination IP
Source and destination IP
Source and destination MAC
Source and destination MAC, and source and destination IP
Source and destination MAC, and source and destination IP
Destination TCP/UDP port
Source and destination MAC
Source and destination MAC, and source and destination IP
Source and destination MAC, source and destination IP , and source and destination port
Source TCP/UDP port
Source and destination MAC
Source and destination MAC, and source and destination IP
Source and destination MAC, source and destination IP , and source and destination port
Source and destination TCP/UDP port
Source and destination MAC
Source and destination MAC, source and destination IP
Source and destination MAC, source and destination IP , and source and destination port
Use the option that provides the balance criteria with the greatest variety in your configuration. For example, if the traffic on an EtherChannel is going only to a single MAC address and you use the destination MAC address as the basis of port-channel load balancing, the EtherChannel always chooses the same link in that EtherChannel; using source addresses or IP addresses might result in better load balancing.
LACP Overview
Note
You must enable the LACP feature before you can configure and use LACP functions.
The following figure shows how individual links can be combined into LACP EtherChannels and channel groups as well as function as individual links.
With LACP, just like with static port-channels, you can bundle up to 16 interfaces in a channel group.
Note
When you delete the EtherChannel, Cisco NX-OS automatically deletes the associated channel group. All member interfaces revert to their previous configuration.
You cannot disable LACP while any LACP configurations are present.
LACP ID Parameters
LACP uses the following parameters:
LACP system priority—Each system that runs LACP has an LACP system priority value. You can accept the default value of 32768 for this parameter, or you can configure a value between 1 and 65535. LACP uses the system priority with the MAC address to form the system ID and also uses the system priority during negotiation with other devices. A higher system priority value means a lower priority.
Note
The LACP system ID is the combination of the LACP system priority value and the MAC address.
LACP port priority—Each port configured to use LACP has an LACP port priority. You can accept the default value of 32768 for the LACP port priority, or you can configure a value between 1 and 65535. LACP uses the port priority with the port number to form the port identifier. LACP uses the port priority to decide which ports should be put in standby mode when there is a limitation that prevents all compatible ports from aggregating and which ports should be put into active mode. A higher port priority value means a lower priority for LACP. You can configure the port priority so that specified ports have a lower priority for LACP and are most likely to be chosen as active links, rather than hot-standby links.
LACP administrative key—LACP automatically configures an administrative key value equal to the channel-group number on each port configured to use LACP. The administrative key defines the ability of a port to aggregate with other ports. A port’s ability to aggregate with other ports is determined by these factors:
Channel Modes
Individual interfaces in EtherChannels are configured with channel modes. When you run static EtherChannels, with no protocol, the channel mode is always set to on. After you enable LACP globally on the device, you enable LACP for each channel by setting the channel mode for each interface to active or passive. You can configure either channel mode for individual links in the LACP channel group.
Note
You must enable LACP globally before you can configure an interface in either the active or passive channel mode.
The following table describes the channel modes.
Table 3 Channel Modes for Individual Links in an EtherChannel Channel Mode
Description
passive
LACP mode that places a port into a passive negotiating state, in which the port responds to LACP packets that it receives but does not initiate LACP negotiation.
active
LACP mode that places a port into an active negotiating state, in which the port initiates negotiations with other ports by sending LACP packets.
on
All static EtherChannels, that is, that are not running LACP, remain in this mode. If you attempt to change the channel mode to active or passive before enabling LACP, the device returns an error message.
You enable LACP on each channel by configuring the interface in that channel for the channel mode as either active or passive. When an LACP attempts to negotiate with an interface in the on state, it does not receive any LACP packets and becomes an individual link with that interface; it does not join the LACP channel group.
Both the passive and active modes allow LACP to negotiate between ports to determine if they can form an EtherChannel, based on criteria such as the port speed and the trunking state. The passive mode is useful when you do not know whether the remote system, or partner, supports LACP.
Ports can form an LACP EtherChannel when they are in different LACP modes as long as the modes are compatible as in the following examples:
A port in active mode can form an EtherChannel successfully with another port that is in active mode.
A port in active mode can form an EtherChannel with another port in passive mode.
A port in passive mode cannot form an EtherChannel with another port that is also in passive mode because neither port will initiate negotiation.
A port in on mode is not running LACP.
LACP Marker Responders
Using EtherChannels, data traffic may be dynamically redistributed due to either a link failure or load balancing. LACP uses the Marker Protocol to ensure that frames are not duplicated or reordered because of this redistribution. Cisco NX-OS supports only Marker Responders.
Creating an EtherChannel
SUMMARY STEPSYou can create an EtherChannel before creating a channel group. Cisco NX-OS automatically creates the associated channel group.
Note
If you want LACP-based EtherChannels, you need to enable LACP.
2. switch(config)# interface port-channel channel-number
3. switch(config)# no interface port-channel channel-number
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface port-channel channel-number
Specifies the port-channel interface to configure, and enters the interface configuration mode. The range is from 1 to 4096. Cisco NX-OS automatically creates the channel group if it does not already exist.
Step 3 switch(config)# no interface port-channel channel-number
Removes the EtherChannel and deletes the associated channel group.
Adding a Port to an EtherChannel
SUMMARY STEPSYou can add a port to a new channel group or to a channel group that already contains ports. Cisco NX-OS creates the EtherChannel associated with this channel group if the EtherChannel does not already exist.
Note
If you want LACP-based EtherChannels, you need to enable LACP.
2. switch(config)# interface type slot/port
3. (Optional) switch(config-if)# switchport mode trunk
4. (Optional) switch(config-if)# switchport trunk {allowed vlan vlan-id | native vlan vlan-id}
5. switch(config-if)# channel-group channel-number
6. (Optional) switch(config-if)# no channel-group
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface type slot/port
Specifies the interface that you want to add to a channel group and enters the interface configuration mode.
Step 3 switch(config-if)# switchport mode trunk
(Optional) Configures the interface as a trunk port.
Step 4 switch(config-if)# switchport trunk {allowed vlan vlan-id | native vlan vlan-id}
(Optional) Configures necessary parameters for a trunk port.
Step 5 switch(config-if)# channel-group channel-number
Configures the port in a channel group and sets the mode. The channel-number range is from 1 to 4096. Cisco NX-OS creates the EtherChannel associated with this channel group if the EtherChannel does not already exist. This is called implicit EtherChannel creation.
Step 6 switch(config-if)# no channel-group
(Optional) Removes the port from the channel group. The port reverts to its original configuration.
Configuring Load Balancing Using EtherChannels
SUMMARY STEPSYou can configure the load-balancing algorithm for EtherChannels that applies to the entire device.
Note
If you want LACP-based EtherChannels, you need to enable LACP.
2. switch(config)# port-channel load-balance ethernet {[destination-ip | destination-mac | destination-port | source-dest-ip | source-dest-mac | source-dest-port | source-ip | source-mac | source-port] crc-poly}
3. (Optional) switch(config)# no port-channel load-balance ethernet
4. (Optional) switch# show port-channel load-balance
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# port-channel load-balance ethernet {[destination-ip | destination-mac | destination-port | source-dest-ip | source-dest-mac | source-dest-port | source-ip | source-mac | source-port] crc-poly}
Specifies the load-balancing algorithm for the device. The range depends on the device. The default is source-dest-mac.
Beginning with Cisco NX-OS Release 5.0(3)N2(1), the Cisco Nexus 5500 Platform switches support 8 hash polynomials that can be used for compression on the hash-parameters. Depending on variations in the hash parameters for egress traffic flows from a port channel, different polynomials could provide different load distribution results. The default hash polynomial is CRC8a. The variable can be configured as follows:
Step 3 switch(config)# no port-channel load-balance ethernet
(Optional) Restores the default load-balancing algorithm of source-dest-mac.
Step 4 switch# show port-channel load-balance
(Optional) Displays the port-channel load-balancing algorithm.
This example shows how to configure source IP load balancing for EtherChannels:
switch# configure terminalswitch (config)# port-channel load-balance ethernet source-ip
Note
Before Release 4.0(1a)N1 of Cisco NX-OS, the source-dest-ip, source-dest-mac, and source-dest-port keywords were source-destination-ip, source-destination-mac, and source-destination-port, respectively.
Configuring Hardware Hashing for Multicast Traffic
SUMMARY STEPSBy default, ingress multicast traffic on any port in the switch selects a particular EtherChannel member to egress the traffic. You can configure hardware hashing for multicast traffic to reduce potential bandwidth issues and to provide effective load balancing of the ingress multicast traffic. Use the hardware multicast hw-hash command to enable hardware hashing. To restore the default, use the no hardware multicast hw-hash command.
2. switch(config)# interface port-channel channel-number
3. switch(config-if)# hardware multicast hw-hash
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface port-channel channel-number
Selects the EtherChannel and enters the interface configuration mode.
Step 3 switch(config-if)# hardware multicast hw-hash
Configures hardware hashing for the specified EtherChannel.
This example shows how to configure hardware hashing on an EtherChannel:
switch# configure terminalswitch (config)# interface port-channel 21switch(config-if)# hardware multicast hw-hashThis example shows how to remove hardware hashing from an EtherChannel:
switch# configure terminalswitch (config)# interface port-channel 21switch(config-if)# no hardware multicast hw-hashEnabling LACP
SUMMARY STEPSLACP is disabled by default; you must enable LACP before you begin LACP configuration. You cannot disable LACP while any LACP configuration is present.
LACP learns the capabilities of LAN port groups dynamically and informs the other LAN ports. Once LACP identifies correctly matched Ethernet links, it facilitates grouping the links into an EtherChannel. The EtherChannel is then added to the spanning tree as a single bridge port.
2. switch(config)# feature lacp
3. (Optional) switch(config)# show feature
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# feature lacp
Enables LACP on the switch.
Step 3 switch(config)# show feature
(Optional) Displays enabled features.
Configuring the Channel Mode for a Port
SUMMARY STEPSYou can configure the channel mode for each individual link in the LACP EtherChannel as active or passive. This channel configuration mode allows the link to operate with LACP.
When you configure EtherChannels with no associated protocol, all interfaces on both sides of the link remain in the on channel mode.
2. switch(config)# interface type slot/port
3. switch(config-if)# channel-group channel-number [force] [mode {on | active | passive}]
4. switch(config-if)# no channel-group number mode
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface type slot/port
Specifies the interface to configure, and enters the interface configuration mode.
Step 3 switch(config-if)# channel-group channel-number [force] [mode {on | active | passive}]
Specifies the port mode for the link in an EtherChannel. After LACP is enabled, you configure each link or the entire channel as active or passive.
force—Specifies that the LAN port be forcefully added to the channel group. This option is available in Cisco NX-OS Release 5.0(2)N2(1).
mode—Specifies the EtherChannel mode of the interface.
active—Specifies that when you enable LACP, this command enables LACP on the specified interface. The interface is in an active negotiating state in which the port initiates negotiations with other ports by sending LACP packets.
on—(Default mode) Specifies that all EtherChannels that are not running LACP remain in this mode.
passive—Enables LACP only if an LACP device is detected. The interface is in a passive negotiation state in which the port responds to LACP packets that it receives but does not initiate LACP negotiation.
When you run EtherChannels with no associated protocol, the channel mode is always on.
Step 4 switch(config-if)# no channel-group number mode
Returns the port mode to on for the specified interface.
This example shows how to set the LACP-enabled interface to active port-channel mode for Ethernet interface 1/4 in channel group 5:
switch# configure terminal switch (config)# interface ethernet 1/4 switch(config-if)# channel-group 5 mode activeThis example shows how to forcefully add an interface to the channel group 5:
switch(config)# interface ethernet 1/1 switch(config-if)# channel-group 5 force switch(config-if)#Configuring the LACP Fast Timer Rate
SUMMARY STEPSYou can change the LACP timer rate to modify the duration of the LACP timeout. Use the lacp rate command to set the rate at which LACP control packets are sent to an LACP-supported interface. You can change the timeout rate from the default rate (30 seconds) to the fast rate (1 second). This command is supported only on LACP-enabled interfaces.
2. switch(config)# interface type slot/port
3. switch(config-if)# lacp rate fast
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface type slot/port
Specifies the interface to configure and enters the interface configuration mode.
Step 3 switch(config-if)# lacp rate fast
Configures the fast rate (one second) at which LACP control packets are sent to an LACP-supported interface.
This example shows how to configure the LACP fast rate on Ethernet interface 1/4:
switch# configure terminalswitch (config)# interface ethernet 1/4switch(config-if)# lacp rate fastThis example shows how to restore the LACP default rate (30 seconds) on Ethernet interface 1/4.
switch# configure terminalswitch (config)# interface ethernet 1/4switch(config-if)# no lacp rate fastConfiguring the LACP System Priority and System ID
SUMMARY STEPS2. switch(config)# lacp system-priority priority
3. (Optional) switch# show lacp system-identifier
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# lacp system-priority priority
Configures the system priority for use with LACP. Valid values are 1 through 65535, and higher numbers have lower priority. The default value is 32768.
Step 3 switch# show lacp system-identifier
(Optional) Displays the LACP system identifier.
Configuring the LACP Port Priority
SUMMARY STEPS2. switch(config)# interface type slot/port
3. switch(config-if)# lacp port-priority priority
DETAILED STEPS
Command or Action Purpose Step 1 switch# configure terminal
Enters configuration mode.
Step 2 switch(config)# interface type slot/port
Specifies the interface to configure, and enters the interface configuration mode.
Step 3 switch(config-if)# lacp port-priority priority
Configures the port priority for use with LACP. Valid values are 1 through 65535, and higher numbers have lower priority. The default value is 32768.
Verifying EtherChannel Configuration
To display EtherChannel configuration information, perform one of the following tasks:
Command
Purpose
switch# show interface port-channel channel-number Displays the status of a EtherChannel interface.
switch# show feature Displays enabled features.
switch# show resource Displays the number of resources currently available in the system.
switch# show lacp {counters | interface type slot/port | neighbor | port-channel | system-identifier} Displays LACP information.
switch# show port-channel compatibility-parameters Displays the parameters that must be the same among the member ports in order to join an EtherChannel.
switch# show port-channel database [interface port-channel channel-number] Displays the aggregation state for one or more port-channel interfaces.
switch# show port-channel summary Displays a summary for the EtherChannel interfaces.
switch# show port-channel traffic Displays the traffic statistics for EtherChannels.
switch# show port-channel usage Displays the range of used and unused channel numbers.
switch# show port-channel database Displays information on current running of the EtherChannel feature.
switch# show port-channel load-balance Displays information about load-balancing using EtherChannels.
Verifying the Load-Balancing Outgoing Port ID
Command Guidelines
The show port-channel load-balance command allows you to verify which ports a given frame is hashed to on a port channel. You need to specify the VLAN and the destination MAC in order to get accurate results.
Note
Certain traffic flows are not subject to hashing, for example when there is a single port in a port-channel.The following example shows the output of the short port-channel load-balance command.
switch#show port-channel load-balance forwarding-path interface port-channel 10 vlan 1 dst-ip 1.225.225.225 src-ip 1.1.10.10 src-mac aa:bb:cc:dd:ee:ff l4-src-port 0 l4-dst-port 1
Missing params will be substituted by 0's. Load-balance Algorithm on switch: source-dest-port crc8_hash: 204 Outgoing port id: Ehernet1/1 Param(s) used to calculate load-balance:dst-port: 1
src-port: 0
dst-ip: 1.225.225.225
src-ip: 1.1.10.10
dst-mac: 0000.0000.0000
src-mac: aabb.ccdd.eeff