Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
Troubleshooting Load Balancing
Downloads: This chapterpdf (PDF - 158.0KB) The complete bookPDF (PDF - 3.41MB) | Feedback

Troubleshooting Bundles and Load Balancing

Table Of Contents

Troubleshooting Bundles and Load Balancing

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Verifying Routing Table Entries for Parallel Links

Verifying the CEF Database and Measuring Flows

Troubleshooting Problems with Link Bundles

Bundle Does Not Come Up

Bundle Member Not Distributing

Bundle Not Using MAC-Address From Backplane

Layer 3 Data Traffic Not Flowing

Regular Interface (No Subinterfaces)

Subinterface

Ping Failed over Bundle

Layer 3 Packets Not Synching Over Bundle

Layer 2 Traffic Not Flowing

Bundle Statistics

Troubleshooting Layer 2 Bundles and Load Balancing

Verifying the Bundle Status, IGP Route, and CEF Database

Viewing the Expected Paths and Measuring the Flows

Troubleshooting Layer 3 Bundles and Load Balancing


Troubleshooting Bundles and Load Balancing


This chapter explains the procedures for troubleshooting link bundles and load balancing on the Cisco ASR 9000 Aggregation Services Router.

A link bundle is a group of ports that are bundled together and act as a single link. The advantages of link bundles are:

Multiple links can span several LCs to form a single interface; thus, the failure of a single link does not cause a loss of connectivity.

Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available members of the bundle. Therefore, traffic can move onto another link if one of the links within a bundle fails. This allows you to add or remove bandwidth without interrupting packet flow.

This chapter contains the following sections:

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Troubleshooting Problems with Link Bundles

Troubleshooting Layer 2 Bundles and Load Balancing

Troubleshooting Layer 3 Bundles and Load Balancing

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Cisco Express Forwarding (CEF) uses the path information in the IP routing table to balance traffic over link bundles. For this reason, verifying correct load balancing with CEF begins with confirming the contents of the IP routing table and CEF database.

Verifying Routing Table Entries for Parallel Links

Verifying the CEF Database and Measuring Flows

Verifying Routing Table Entries for Parallel Links

Perform this procedure to verify the contents of the IP routing table.

SUMMARY STEPS

1. show route destination-address

2. configure

3. router ospf process

4. maximum paths number

5. end

6. show route destination-address

7. show ospf process interface brief

8. show running-config router ospf process

 
Command or Action
Purpose

Step 1 

show route destination-address

Example:

RP/0/RSP0/CPU0:router# show route 10.1.2.1

Displays the routes to a destination address. Use a destination address on another host that is reachable through the parallel links.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.

Step 2 

configure

Example:

RP/0/RSP0/CPU0:router# configure

Enters global configuration mode.

Step 3 

router ospf process

Example:

RP/0/RSP0/CPU0:router# router ospf 200

Enters configuration mode for the OSPF process.

Step 4 

maximum paths number

Example:

RP/0/RSP0/CPU0:router# maximum paths 3

Configures the maximum number of paths over which to load balance. By default, OSPF balances up to 4 equal-cost paths.

Step 5 

end

Example:

RP/0/RSP0/CPU0:router# end

Ends the configuration process. Enter yes at the prompt to commit the changes.

Step 6 

show route destination-address

Example:

RP/0/RSP0/CPU0:router# show route 10.1.2.1

Displays the routes to a destination address.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.

Step 7 

show ospf process interface brief

Example:

RP/0/RSP0/CPU0:router# show ospf 200 interface brief

Shows interface information for all routes to the destination address, which displays the cost metric. OSPF balances loads over equal-cost routes only, so verify that the interfaces have equal costs. To load balance over unequal paths, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as the IGP instead.

Step 8 

show running-config router ospf process

Example:

RP/0/RSP0/CPU0:router# show running-configuration router ospf process

Displays the running configuration for the OSPF process. This is another way to determine if the interfaces have different costs.

Verifying the CEF Database and Measuring Flows

Perform this procedure to verify the contents of the CEF database.

SUMMARY STEPS

1. show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

2. show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

3. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1 detail

Displays the CEF forwarding table. Verify that it contains the same interfaces that the routing table has for this destination.

Step 2 

show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

Example:

RP/0/RSP0/CPU0:router# show cef exact-route 192.168.254.1 10.1.2.1 protocol ospf source-port 5500 destination-port 80 ingress-interface gi0/6/5/4

Displays the exact route that a specific flow would take, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces.

Step 3 

show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:

RP/0/RSP0/CPU0:router# show interfaces accounting rates

Displays the traffic rates by interface. Use this command to verify that the simulated traffic takes the expected egress interface.

Troubleshooting Problems with Link Bundles

This section explains how to troubleshoot problems with link bundles. It contains the following subsections:

Bundle Does Not Come Up

Bundle Member Not Distributing

Bundle Not Using MAC-Address From Backplane

Layer 3 Data Traffic Not Flowing

Ping Failed over Bundle

Layer 3 Packets Not Synching Over Bundle

Layer 2 Traffic Not Flowing

Bundle Statistics

Bundle Does Not Come Up


Step 1 Ensure that the member port is not "shutdown". Ensure that the MAC burned-in address (BIA) of the port is valid.

RP/0/RSP0/CPU0:router# show interface 
 
   

Step 2 If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and receive accordingly. If LACP packets are not able to send and receive accordingly, check interface counters to identify at what stage packets are dropped.

RP/0/RSP0/CPU0:router# show lacp counters
 
   

Step 3 View LACP statistics.

RP/0/RSP0/CPU0:router# show lacp
 
   

Step 4 Ensure that the other side of the link is up (bundle and members).

RP/0/RSP0/CPU0:router# show bundle
 
   

Bundle Member Not Distributing


Step 1 Ensure that the member is up.

RP/0/RSP0/CPU0:router# show interface node-id

Step 2 Ensure that the remote side is up.

Step 3 Ensure that the LACP parameters are the same on both sides.

a. If LACP is enabled, check its status.

RP/0/RSP0/CPU0:router# show lacp bundle
 
   

b. Ensure that bundle members have the same characteristics.

RP/0/RSP0/CPU0:router# show running interface interface-name
 
   

c. If the bundle members have different characteristics, make them all the same.

d. Ensure that LACP packets are transmitted and received.

RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id
 
   

Workaround

If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in active mode. At least one side must be active.

Bundle Not Using MAC-Address From Backplane


Step 1 Ensure that the backplane MAC is programmed. Note that this command has to be run in the admin mode.

RP/0/RSP0/CPU0:router(admin)# show diag chassis eeprom-info
 
   

Step 2 Display the backplane information.

RP/0/RSP0/CPU0:router# show controllers backplane bpe-trace 
 
   

Layer 3 Data Traffic Not Flowing

Regular Interface (No Subinterfaces)


Step 1 View the Address Resolution Protocol (ARP).

RP/0/RSP0/CPU0:router# show arp

Step 2 Verify that the lag table is programmed properly in the hardware.

RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id

Step 3 View the running configuration information.

RP/0/RSP0/CPU0:router# show running-config

Step 4 View information about packets forwarded by CEF.

RP/0/RSP0/CPU0:router# show cef

Step 5 RP/0/RSP0/CPU0:router# show cef hardware ingress location node-id

Step 6 RP/0/RSP0/CPU0:router# show cef hardware egress location node-id


Subinterface


Step 1 Troubleshoot Layer 3 IPv4 traffic.

Step 2 Ensure that VLAN traffic coming in matches that on the incoming interface.


Ping Failed over Bundle


Step 1 View the ARP.

RP/0/RSP0/CPU0:router# show arp

Step 2 View the ARP information on the particular LC or RSP.

RP/0/RSP0/CPU0:router# show arp location node-id

Step 3 RP/0/RSP0/CPU0:router# show cef hardware detail location node-id ingress

Step 4 RP/0/RSP0/CPU0:router# show interface

Step 5 Use the hash calculator to determine which bundle member (interface) to test.

Step 6 Remove the interface from the bundle.

Step 7 Assign an IP address to the interface.

Step 8 Ping the interface.

Step 9 Ensure that the ARP is resolved between the router and the node being pinged.

Step 10 Ensure that the MAC address in the ARP table of the other side corresponds to that on the router.

Step 11 Ensure that the MAC address of the bundle is valid.

Step 12 Ensure that the routing and hardware routing table has an entry to the next hop.

Step 13 Check the interface counters to see if ping packets are transmitted and being received on the router member port of the bundle.

Step 14 Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the bundle.

Step 15 Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly programmed.

show cef adajacency bundle-type bundle-number hardware egress detail location location-id

show cef adajacency bundle-type bundle-number hardware ingress remote detail location location-id


Workaround

Try a different port.

Layer 3 Packets Not Synching Over Bundle


Step 1 View the interface information:

RP/0/RSP0/CPU0:router# show interface

Step 2 Turn on the debug of that protocol or look at the protocol counters to see if the protocol packets are being sent and received.

Step 3 If the protocol packets are not being sent or received, check the interface counters to see if interface indicates packets in and out.

Step 4 If the interface level indicates that packets are coming in and out but not reaching protocol, check the ucode counters to see if there are any drops.


Layer 2 Traffic Not Flowing

VPLS


Step 1 Verify that the AC is up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain

Step 2 Verify that the bridge domain is up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain

Step 3 Look for MTU mismatches.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail


VPWS


Step 1 View brief information on configured cross-connects.

RP/0/RSP0/CPU0:router# show l2vpn xconnect summary

Step 2 RP/0/RSP0/CPU0:router# show l2vpn xconnect state

Step 3 RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether bundle-id location node-id


Bundle Statistics

Layer 2 statistics are not supported in the show interface accounting command for bundle interfaces in the current release.

Troubleshooting Layer 2 Bundles and Load Balancing

This section describes how to troubleshoot Layer 2 bundles and load balancing. It includes the following topics:

Verifying the Bundle Status, IGP Route, and CEF Database

Viewing the Expected Paths and Measuring the Flows

Verifying the Bundle Status, IGP Route, and CEF Database

CEF uses the path information in the IP routing table to balance traffic over multiple links. For this reason, confirming proper CEF load balancing begins with confirming the contents of the IP routing table. When troubleshooting a bundle, verify that the bundle is up and that the IGP route to the desired destination includes the bundle interface.

SUMMARY STEPS

1. show running-config

2. show bundle {Bundle-Ether | Bundle-POS} interface-path-id

3. show interface {Bundle-Ether | Bundle-POS} bundle-id

4. show arm router-id

5. show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id

6. show route destination-address

7. show cef ipv4 prefix

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show running-config

Example:

RP/0/RSP0/CPU0:router# show running-config

Displays the running configuration.

Verify that the configuration related to the bundles are correct.

Step 2 

show bundle {Bundle-Ether | Bundle-POS} interface-path-id

Example:

RP/0/RSP0/CPU0:router# show bundle bundle-ether 12

Displays the bundle status.

Verify that the bundle has the expected number of links. If not, troubleshoot the bundle first.

Step 3 

show interface {Bundle-Ether | Bundle-POS} bundle-id

Example:

RP/0/RSP0/CPU0:router# show interface bundle-ether 12

Displays the interface status.

Verify that the interface is assigned to the bundle.

Step 4 

show arm router-id

Example:

RP/0/RSP0/CPU0:router# show arm router-id

Step 5 

show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id

Example:

RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether 12 location 0/4/CPU0

Step 6 

show route destination-address

Example:

RP/0/RSP0/CPU0:router# show route 10.1.2.1

Displays the routes to a destination address. Use a destination address on another host that is reachable through the bundle.

Verify that the route to the desalination address includes the bundle interface. If not, make sure that the bundle interface is included in the IGP process configuration.

Step 7 

show cef ipv4 prefix

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1

Displays the CEF forwarding table. Verify that it contains the same bundle interface that the routing table has for this subnet prefix.

Viewing the Expected Paths and Measuring the Flows

Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member links. This is an interactive tool prompts for the information that the load balancing algorithm uses to allocate flows to member links.

SUMMARY STEPS

1. bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

2. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

Example:

RP/0/RSP0/CPU0:router# bundle-hash bundle-ether 12

Launches the bundle-hash utility. This is an interactive utility that prompts for the necessary information.

Step 2 

show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:

RP/0/RSP0/CPU0:router# show interfaces tenGigE 0/5/0/1

Displays interface information, which includes the traffic rates. Use this command for each link in the bundle to verify that the simulated traffic takes the expected link. Use clear counters to make it easier to view the traffic allocation.

Troubleshooting Layer 3 Bundles and Load Balancing

This section provides commands for troubleshooting Layer 3 bundles and load balancing.


Step 1 RP/0/RSP0/CPU0:router# show arm router-id

Step 2 RP/0/RSP0/CPU0:router# show bundle

Step 3 RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id

Step 4 RP/0/RSP0/CPU0:router# show running-config

Step 5 RP/0/RSP0/CPU0:router# show arm router-ids

Step 6 Find out which member is carrying the traffic out.

RP/0/RSP0/CPU0:router# bundle-hash bundle-ether bundle-id

Step 7 View each member of the bundle to see which member is actually carrying the traffic out.

RP/0/RSP0/CPU0:router# show interface

Step 8 Display the exact route, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces.

show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]