Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
Troubleshooting Load Balancing
Downloads: This chapterpdf (PDF - 491.0KB) The complete bookPDF (PDF - 3.46MB) | Feedback

Troubleshooting Load Balancing

Table Of Contents

Troubleshooting Load Balancing

About Load Balancing with Cisco Express Forwarding

Load Balancing Function

Source Information for Load Balancing

Layer 2 Load Balancing

Terminology

Troubleshooting Layer 3 or Layer 4 Load Balancing

Verifying the Routing Table Entries for Parallel Links

Configuring Layer 4 Load Balancing

Verifying the CEF Database and Measuring Flows

Troubleshooting Layer 2 Load Balancing

Verifying the Bundle Status, IGP Route, and CEF Database

Configuring Layer 4 Load Balancing

Viewing the Expected Paths and Measuring the Flows

Configuration Examples for Troubleshooting Load Balancing

Troubleshooting Layer 3 or Layer 4 Load Balancing Example

Verifying Parallel Links in the Routing Table

Verifying IPv4/IPv6 Unicast Load Balancing

Troubleshooting Layer 2 Load Balancing Example

Verifying the Bundle Status

Checking the IGP Routing Table and CEF Database

Verifying Bundle Load Balancing


Troubleshooting Load Balancing


This chapter explains the troubleshooting procedures for load balancing on the Cisco CRS-1 Router. This chapter has the following sections:

About Load Balancing with Cisco Express Forwarding

Troubleshooting Layer 3 or Layer 4 Load Balancing

Troubleshooting Layer 2 Load Balancing

Configuration Examples for Troubleshooting Load Balancing

About Load Balancing with Cisco Express Forwarding

This document explains how Cisco IOS XR implements load balancing for Layer 2 (data link layer), Layer 3 (network layer) and Layer 4 (transport layer) flows across multiple parallel links when using Cisco Express Forwarding (CEF). This procedure explains load balancing on the Cisco CRS-1 Router only.

Load Balancing Function

Load balancing is a Cisco IOS XR software feature that improves the utilization of parallel links by distributing traffic flows among them. A traffic flow consists of packets that have common Layer 3 and Layer 4 characteristics, such as the same source and destination IP addresses. The concept of a flow also extends to Layer 2 bundles and to virtual links such as MPLS traffic-engineering tunnels.

Cisco IOS XR uses a hash algorithm to identify individual flows and to distribute them across multiple parallel links. Each flow uses a specific link, but there are many flows in a production environment. Balancing traffic flows is similar to the earlier method of per-destination balancing, which grouped all packets by destination. Grouping packets by flow provides a better degree of granularity, and ultimately balancing, because the hash algorithm considers up to seven packet characterizes instead of just the destination. Cisco IOS XRsoftware does not use the per-packet load balancing option, which rotates individual packets around the available links.

Source Information for Load Balancing

Per-flow load balancing in Cisco IOS XR has two options: 3-tuple (the default) and 7-tuple. The 3-tuple option uses Layer 3 packet information to identify flows, and the 7-tuple option uses Layer 3 and Layer 4 information. A unique flow consists of packets match all three of the 3-tuple values or all seven of the 7-tuple values.

With the 3-tuple option, the hash algorithm uses the following Layer 3 and platform-specific information to identify unique flows.

Layer 3 information from the IP header

Source IP address

Destination IP address

Platform-related information:

Router ID

With the 7-tuple option, the hash algorithm uses the following Layer 3, Layer 4, and platform-specific information to identify unique flows.

Layer 3 information from the IP header:

Source IP address

Destination IP address

Protocol

Layer 4 information from the TCP or UDP header

Source port

Destination port

Platform-related information

Router ID

Ingress interface handle

For Layer 3 packets that do not contain Layer 4 information, the 7-tuple option substitutes a replacement value or ignores the field.

Layer 2 Load Balancing

Load balancing at Layer 2 is also done by flow. Cisco IOS XR software provides load balancing for Layer 2 bundles only. Data is distributed to a link in proportion to the bandwidth of the link in relation to its bundle. For all IP traffic passing over a bundle interface, load balancing is done by the Forwarding Information Base (FIB) on the ingress and egress line cards. When a received packet is switched to a bundle interface, the FIB chooses which member link to use based on the source and destination IP address of the packets. The Adjacency Information Base (AIB) stores a list of adjacencies that the FIB uses to determine which member links are available for forwarding.

Layer 2 load balancing uses the 3-tuple algorithm, but uses only the source and destination IP addresses in the packet. A modulo operation on the hash result is subsequently performed using the number of entries in the load balancing table, which is driven by the link weighting.

For each member of bundle, the adjacency information includes a weight that balances the load among bundle members with different bandwidths. The theory behind the weight usage is:

The weight reflects the bandwidth ratios of all members. For example, if you have one OC192 member and one OC48 member as part of a bundle, OC192 member is given a relative weight of 4 while OC48 has a weight as 1.

If you have only one member in the bundle, it still gets a relative weight of 2 to initiate hashing.

If all members have the same bandwidth, each is given relative weight of 1.

If all members happen to be OC768, each member receives a relative weight of 2, which balances loads between two buffers.

There is nothing to configure for Layer 2 load balancing. Active links in the bundle are automatically eligible for load balancing.

Terminology

The following terms are applicable to load balancing.

Term
Description

Prefix

Describes a destination IP network, such as 192.16.10.0/24. Cisco IOS XR adds a destination IP prefix to the routing table using information obtained from exchanging messages using a dynamic routing protocol or by manual configuration of static routes.

Path

Describes a valid route to reach a destination prefix. Cisco IOS XR assigns a cost to each path. A set of active paths to a destination prefix may have equal or unequal costs. Loads can be balanced across equal-cost paths.

Session

Describes a unidirectional communication flow between two IP nodes. All packets in a session use the same source and destination IP address.

Flow

A network flow is a unidirectional sequence of packets that common header fields, which include the following:

Source IP address

Destination IP address

IP protocol

Source port (for example UDP or TCP port)

Destination port

Ingress interface

IP type of service


Troubleshooting Layer 3 or Layer 4 Load Balancing

This section describes how to troubleshoot load balancing for Layer 3 or Layer 4 unicast flows when Open Shortest Path First (OSPF) is the Interior Gateway Protocol (IGP).

Verifying the Routing Table Entries for Parallel Links

Configuring Layer 4 Load Balancing

Verifying the CEF Database and Measuring Flows

Verifying the Routing Table Entries for Parallel Links

Cisco Express Forwarding uses the path information in the IP routing table to balance traffic over multiple links. For this reason, verifying correct load balancing with Cisco Express Forwarding begins with confirming the contents of the IP routing table.


Note Unlike configuration procedures, troubleshooting is not a deterministic process. This section provides a typical procedure for troubleshooting load balancing when OPSF is the IGP. For a example of this procedure, see Troubleshooting Layer 3 or Layer 4 Load Balancing Example.


SUMMARY STEPS

1. show route destination-address

2. configure

3. router ospf process

4. maximum paths number

5. end

6. show route destination-address

7. show ospf process interface brief

8. show running-config router ospf process

9. configure

10. interface type interface-path-id

11. cost value

12. end

13. show route destination-address

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show route destination-address

Example:

RP/0/RP0/CPU0:Router# show route 10.1.2.1

Displays the routes to a destination address. Use a destination address on another host that is reachable through the parallel links.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.

Step 2 

configure

Example:

RP/0/RP0/CPU0:router# configure

Enters global configuration mode.

Step 3 

router ospf process

Example:

RP/0/RP0/CPU0:router# router ospf 200

Enters configuration mode for the OSPF process.

Step 4 

maximum paths number

Example:

RP/0/RP0/CPU0:router# maximum paths 3

Configures the maximum number of paths over which to load balance. By default, OSPF balances up to 4 equal-cost paths.

Step 5 

end

Example:

RP/0/RP0/CPU0:router# end

Ends the configuration process. Enter yes at the prompt to commit the changes.

Step 6 

show route destination-address

Example:

RP/0/RP0/CPU0:Router# show route 10.1.2.1

Displays the routes to a destination address.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.

Step 7 

show ospf process interface brief

Example:

RP/0/RP0/CPU0:Router# show ospf 200 interface brief

Shows interface information for all routes to the destination address, which displays the cost metric. OSPF balances loads over equal-cost routes only, so verify that the interfaces have equal costs. To load balance over unequal paths, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as the IGP instead.

Step 8 

show running-config router ospf process

Example:

RP/0/RP0/CPU0:Router# show running-configuration router ospf process

Displays the running configuration for the OSPF process. This is another way to determine if the interfaces have different costs.

Step 9 

configure

Example:

RP/0/RP0/CPU0:router# configure

Enters global configuration mode.

Step 10 

interface type interface

Example:

RP/0/RP0/CPU0:router# interface gi0/6/5/7

Enters interface configuration mode.

Step 11 

cost value

Example:

RP/0/RP0/CPU0:router# cost 1

Sets the cost of this interfaceto to the same value as the cost of the other parallel interfaces. If necessary, repeat the previous step and this step for other interfaces.

Step 12 

end

Example:

RP/0/RP0/CPU0:router# end

Ends the configuration process. Enter yes at the prompt to commit the changes.

Step 13 

show route destination-address

Example:

RP/0/RP0/CPU0:Router# show route 10.1.2.1

Displays the routes to a destination address.

Verify that number of routes in the routing table equals the number of parallel links.

Configuring Layer 4 Load Balancing

By default, the load balancing algorithm uses Layer 3 information only. This section describes how to configure load balancing to use both Layer 3 and Layer 4 information. Use the setting that is appropriate for your own system. Layer 4 distributes loads more evenly over the interfaces because it considers more packet parameters when defining unique flows.

SUMMARY STEPS

1. configure

2. cef load-balancing fields L4

3. end

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

configure

Example:

RP/0/RP0/CPU0:router# configure

Enters global configuration mode.

Step 2 

RP/0/RP0/CPU0:Router(config)# cef load-balancing fields [L3 |L4]

Example:

RP/0/RP0/CPU0:router(config)# cef load-balancing fields L4

Configures the fields of the IP header that the hash algorithm uses when balancing flows across parallel links.

Step 3 

end

Ends the configuration process. Enter yes at the prompt to commit the changes.

Verifying the CEF Database and Measuring Flows

Cisco Express Forwarding uses the path information in the Cisco Express Forwarding (CEF) database to balance traffic over multiple links. For this reason, confirming proper CEF load balancing begins with confirming the contents of the CEF database.

SUMMARY STEPS

1. show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

2. show cef [ipv4 | ipv6] exact-route source-address destination-address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

3. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

Example:

RP/0/RP0/CPU0:router# show cef ipv4 10.1.2.1 detail

Displays the CEF forwarding table. Verify that it contains the same interfaces that the routing table has for this destination.

Step 2 

RP/0/RP0/CPU0:Router# show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

Example:

RP/0/RP0/CPU0:router# show cef exact-route 192.168.254.1 10.1.2.1 protocol tcp source-port 5500 destination-port 80 ingress-interface gi0/6/5/4

Displays the exact route that a specific flow would take. Use this command for several flows to verify that they are distributed equally over the parallel interfaces.

Step 3 

show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:

RP/0/RP0/CPU0:router# show interfaces accounting rates

Displays the traffic rates by interface. Use this command to verify that the simulated traffic takes the expected egress interface.

Troubleshooting Layer 2 Load Balancing

This section describes how to troubleshoot load balancing at Layer 2. This procedure is specific to Layer 2 bundles.

Verifying the Bundle Status, IGP Route, and CEF Database

Configuring Layer 4 Load Balancing

Viewing the Expected Paths and Measuring the Flows

Verifying the Bundle Status, IGP Route, and CEF Database

Cisco Express Forwarding uses the path information in the IP routing table to balance traffic over multiple links. For this reason, confirming proper Cisco Express Forwarding load balancing begins with confirming the contents of the IP routing table. When troubleshooting a bundle, verify that the bundle us up and that the IGP route to the desired destination includes the bundle interface.


Note Unlike configuration procedures, troubleshooting is not a deterministic process. This section provides a typical procedure for troubleshooting load balancing within an Ethernet bundle. For a example of this procedure, see Troubleshooting Layer 2 Load Balancing Example.


SUMMARY STEPS

1. show bundle {Bundle-Ether | Bundle-POS} interface-path-id

2. show route destination-address

3. show cef ipv4 prefix

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show bundle {Bundle-Ether | Bundle-POS} interface-path-id

Example:

RP/0/RP0/CPU0:Router# show bundle bundle-ether 12

Displays the bundle status.

Verify that the bundle has the expected number of links. If not, troubleshoot the bundle first.

Step 2 

show route destination-address

Example:

RP/0/RP0/CPU0:Router# show route 10.1.2.1

Displays the routes to a destination address. Use a destination address on another host that is reachable through the bundle.

Verify that the route to the desalination address includes the bundle interface. If not, make sure that the bundle interface is included in the IGP process configuration.

Step 3 

show cef ipv4 prefix

Example:

RP/0/RP0/CPU0:router# show cef ipv4 10.1.2.1

Displays the CEF forwarding table. Verify that it contains the same bundle interface that the routing table has for this subnet prefix.

Configuring Layer 4 Load Balancing

Layer 2 load balancing is achieved by inspecting the Layer 3 and Layer 4 information in the encapsulated packet. By default, the load balancing algorithm uses Layer 3 information only. This section describes how to configure load balancing to use both Layer 3 and Layer 4 information. If your system only uses Layer 3 load balancing, retain the default load balancing setting.

For the configuration procedure, see Configuring Layer 4 Load Balancing.

Viewing the Expected Paths and Measuring the Flows

Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member links. This is an interactive tool prompts for the information that the load balancing algorithm uses to allocate flows to member links.

SUMMARY STEPS

1. bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

2. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

Example:

RP/0/RP0/CPU0:router# bundle-hash bundle-ether 12

Launches the bundle-hash utility. This is an interactive utility that prompts for the necessary information.

Step 2 

show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:

RP/0/RP0/CPU0:router# show interfaces Gi0/6/5/5

Displays interface information, which includes the traffic rates. Use this command for each link in the bundle to verify that the simulated traffic takes the expected link. Use clear counters to make it easier to view the traffic allocation.

Configuration Examples for Troubleshooting Load Balancing

This section shows a practical example of troubleshooting load balancing problems, in the following sections:

Troubleshooting Layer 3 or Layer 4 Load Balancing Example

Troubleshooting Layer 2 Load Balancing Example

Troubleshooting Layer 3 or Layer 4 Load Balancing Example

Table 13-1 shows the sample configuration that this section uses to describe the troubleshooting process. Two routers, Router A and Router B, connect back to back over three serial Gigabit Ethernet interfaces.

Table 13-1 Sample Configuration

Router A
Router B
interface GigabitEthernet0/6/5/5
 ipv4 address 10.12.48.1 255.255.255.0
!
interface GigabitEthernet0/6/5/6
 ipv4 address 10.12.44.1 255.255.255.0
!
interface GigabitEthernet0/6/5/7
 ipv4 address 10.12.40.1 255.255.255.0
 
        
router ospf 200
!
area 0
!
 interface GigabitEthernet0/6/5/5
 interface GigabitEthernet0/6/5/6
 interface GigabitEthernet0/6/5/7
  cost 10
interface GigabitEthernet0/6/5/5
 ipv4 address 10.12.48.2 255.255.255.0
!
interface GigabitEthernet0/6/5/6
 ipv4 address 10.12.44.2 255.255.255.0
!
interface GigabitEthernet0/6/5/7
 ipv4 address 10.12.40.2 255.255.255.0
 
        
router ospf 200
maximum-paths 1
area 0
!
 interface GigabitEthernet0/6/5/5
 interface GigabitEthernet0/6/5/6
 interface GigabitEthernet0/6/5/7
  cost 10

The bold text identifies those configurations that affect load balancing.

Verifying Parallel Links in the Routing Table

Troubleshooting of load balancing starts with the routing table, as described in the following sections:

Checking the IGP Routing Table

Checking the Maximum Paths

Checking the Route Metric

Checking the IGP Routing Table

This section shows how Router B selects one or more paths to reach Router A's loopback interface, 10.1.2.1.

By default, Open Shortest Path First (OSPF) supports four equal-cost paths to a destination. In this scenario, Router B is configured with maximum-paths equal to one, so it chooses only one path from the possible equal-cost paths. Use the show route command to view the current path to Router A:

RP/0/RP0/CPU0:RouterB# show route 10.1.2.1
 
   
Fri Sep 19 11:02:00.732 PST DST
 
   
Routing entry for 10.1.2.1/32
  Known via "ospf 200", distance 110, metric 2, type intra area
  Installed Sep 19 06:49:40.555 for 04:12:20
  Routing Descriptor Blocks
    10.12.48.1, from 10.1.2.1, via GigabitEthernet0/6/5/5
      Route metric is 2
  No advertising protos. 
 
   

This shows that Router B selected GigabitEthernet0/6/5/5 as the single path to 10.1.2.1. Use the show cef command to view the corresponding CEF path:

RP/0/RP0/CPU0:RouterB# show cef ipv4 10.1.2.1
 
   
Fri Sep 19 11:00:21.846 PST DST
 
   
10.1.2.1/32, version 0, internal 0x40040001 (0xa9482f74) [1], 0x0 (0xa902a9cc), 0x4400 
(0xa94f3300)  
 Updated Sep 19 06:49:40.562 
 remote adjacency to GigabitEthernet0/6/5/5
 Prefix Len 32, traffic index 0, precedence routine (0)
   via 10.12.48.1, GigabitEthernet0/6/5/5, 4 dependencies, weight 0, class 0
    next hop 10.12.48.1
    remote adjacency
     local label 16071      labels imposed {None}
 
   

This verifies that CEF also uses GigabitEthernet0/6/5/5 as the egress interface for traffic forwarded to 10.1.2.1. No load balancing can occur in this configuration because there are three parallel links, but only one is allowed.

Checking the Maximum Paths

Use the maximum-paths sub-command in the OSPF configuration mode to allow up to three paths in the routing table:

RP/0/RP0/CPU0:RouterB# configure
Fri Sep 19 11:33:27.451 PST DST
RP/0/RP0/CPU0:P2_CRS-8(config)# router ospf 200
RP/0/RP0/CPU0:P2_CRS-8(config-ospf)# maximum paths 3
RP/0/RP0/CPU0:P2_CRS-8(config-ospf)# end
 
   

Use the show route command to confirm that the routing table contains the desired number of paths:

RP/0/RP0/CPU0:RouterB# show route 10.1.2.1
 
   
Fri Sep 19 11:36:23.345 PST DST
 
   
Routing entry for 10.1.2.1/32
  Known via "ospf 200", distance 110, metric 2, type intra area
  Installed Sep 19 11:34:04.284 for 00:02:19
  Routing Descriptor Blocks
    10.12.48.1, from 10.1.2.1, via GigabitEthernet0/6/5/5
      Route metric is 2
    10.12.44.1, from 10.1.2.1, via GigabitEthernet0/6/5/6
      Route metric is 2
  No advertising protos. 
 
   

OSPF allows equal cost load balancing only, so one of the three paths must have a higher cost. To balance loads among links with unequal cost metrics, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as your Interior Gateway Protocol (IGP).

Checking the Route Metric

Although OSPF is configured to support three equal cost paths, only two active paths are displayed in the routing table. Use the show ospf interface command to determine the reason:

RP/0/RP0/CPU0:RouterB# show ospf 200 interface brief
 
   
Fri Sep 19 11:39:35.212 PST DST
 
   
* indicates MADJ interface
 
   
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi0/6/5/5    200   0               10.12.48.2/24      1     BDR   1/1
Gi0/6/5/6    200   0               10.12.44.2/24      1     BDR   1/1
Gi0/6/5/7    200   0               10.12.40.2/24      10    BDR   1/1
 
   

Gi0/6/5/7 has a higher cost thanGi0/6/5/6 and Gi0/6/5/7, and is therefore excluded. Use the show run command to confirm that Gi0/6/7 is configured with the cost 10 command:

RP/0/RP0/CPU0:RouterB# show run router ospf 200 
 
   
Fri Sep 19 11:46:36.142 PST DST
router ospf 200
maximum paths 3
 area 0
  interface Loopback1
   passive enable
  !
  ...
  interface GigabitEthernet0/6/5/7
   cost 10
 
   

Use the no cost 10 interface subcommand in the router ospf area configuration mode to restore the default cost to GigabitEthernet0/6/5/7. The show route command now displays three paths to the 194.168.20.0 network:

RP/0/RP0/CPU0:RouterB# show route 10.1.2.1
 
   
Fri Sep 19 11:52:46.942 PST DST
 
   
Routing entry for 10.1.2.1/32
  Known via "ospf 200", distance 110, metric 2, type intra area
  Installed Sep 19 11:52:34.487 for 00:00:12
  Routing Descriptor Blocks
    10.12.48.1, from 10.1.2.1, via GigabitEthernet0/6/5/5
      Route metric is 2
    10.12.44.1, from 10.1.2.1, via GigabitEthernet0/6/5/6
      Route metric is 2
    10.12.40.1, from 10.1.2.1, via GigabitEthernet0/6/5/7
      Route metric is 2
  No advertising protos. 
 
   

The next step is to look at how Cisco Express Forwarding uses the information in the routing table to forward packets.

Verifying IPv4/IPv6 Unicast Load Balancing

CEF load balancing for Layer 3 flows (3-tuple algorithm) is enabled by default, but Layer 4 flows are used in this example. Using Layer 4 information has the advantage of distributing flows more evenly over the parallel links, because it utilizes up to seven packet and platform parameters when calculating the egress interface. The same principles apply to Layer 3 flows, except that the load balancing algorithm checks only the source address, destination address, and router ID.

The troubleshooting process for Layer 3 and Layer 4 load balancing is described in the following sections:

Configuring Layer 4 Load Balancing

Checking the CEF Database

Verifying Load Balancing

Configuring Layer 4 Load Balancing

The default load balancing algorithm for Cisco IOS XR is Layer 3, so first enable Layer 4 (7-tuple algorithm):

RP/0/RP0/CPU0:P2_CRS-8# configure
RP/0/RP0/CPU0:P2_CRS-8(config)# cef load-balancing fields L4
RP/0/RP0/CPU0:P2_CRS-8 (config)# end

Checking the CEF Database

Verify that the CEF database has the expected interfaces to the test destination, 10.1.2.1:

RP/0/RP0/CPU0:P2_CRS-8# show cef ipv4 10.1.2.1 detail
 
   
Sun Sep 21 09:41:47.457 PST DST
10.1.2.1/32, version 0, internal 0x40040001 (0xa9482f74) [1], 0x0 (0xa902a9cc), 
 Updated Sep 19 11:52:34.493 
 remote adjacency to GigabitEthernet0/6/5/5
 Prefix Len 32, traffic index 0, precedence routine (0)
  gateway array (0xa8e9e1b4) reference count 2, flags 0x400d00, source lsd (2), 
                [0 type 4 flags 0x4101000 (0xa9524590) ext 0x0 (0x0)]
  LW-LDI[type=1, refc=1, ptr=0xa902a9cc, sh-ldi=0xa9524590]
   via 10.12.48.1, GigabitEthernet0/6/5/5, 4 dependencies, weight 0, class 0
    next hop 10.12.48.1
    remote adjacency
     local label 16071      labels imposed {None}
   via 10.12.44.1, GigabitEthernet0/6/5/6, 4 dependencies, weight 0, class 0
    next hop 10.12.44.1
    remote adjacency
     local label 16071      labels imposed {None}
   via 10.12.40.1, GigabitEthernet0/6/5/7, 4 dependencies, weight 0, class 0
    next hop 10.12.40.1
    remote adjacency
     local label 16071      labels imposed {None}
 
   
 
   
    Load distribution: 0 1 2 (refcount 0)
 
   
    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/6/5/5    remote         
    1     Y   GigabitEthernet0/6/5/6    remote         
    2     Y   GigabitEthernet0/6/5/7    remote         
 
   

This shows that the CEF database does contain the three parallel links to 10.1.2.1 and shows the associated hash bucket for each one. Therefore, CEF can correctly balance loads across these three interfaces.

Verifying Load Balancing

Now you can verify that the flows are actually distributed among the parallel interfaces. The show cef exact-route command displays the expected result of the hash calculation. If you are using the 3-tuple configuration, you only need the required parameters. The protocol, source-port, destination-port, and ingress-interface only apply to 7-tuple configurations.

In this example, a traffic generator is sending TCP packets to 10.1.2.1 from 192.168.254.1 through interface Gi0/6/5/4. The source TCP port is 5500 and the destination port is 80. The following command displays the calculated egress interface for this flow:

RP/0/RP0/CPU0:P2_CRS-8# show cef exact-route 192.168.254.1 10.1.2.1 protocol tcp 
source-port 5500 destination-port 80 ingress-interface gi0/6/5/4
 
   
Tue Sep 23 03:17:36.339 PST DST
10.1.2.1/32, version 0, internal 0x40040001 (0xa9482f74) [1], 0x0 (0xa902a9cc), 0x4400 
(0xa9db444c) 
 Updated Sep 19 11:52:34.493 
 remote adjacency to GigabitEthernet0/6/5/5
 Prefix Len 32, traffic index 0, precedence routine (0)
   via GigabitEthernet0/6/5/5
 
   

CEF load balancing sends this particular flow out of interface GigabitEthernet0/6/5/5. You can verify that a flow with the specified characteristics is acutally using GigabitEthernet0/6/5/5 using the following command:

RP/0/RP0/CPU0:P2_CRS-8# show interface accounting rates
 
   
Tue Sep 23 05:57:03.780 PST DST
GigabitEthernet0/6/5/4
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_UNICAST            43000               95            0                0
  IPV4_MULTICAST              0                0            0                0
 
   
GigabitEthernet0/6/5/5
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_UNICAST                0                0        43000               92
  IPV4_MULTICAST              0                0            0                0
          
GigabitEthernet0/6/5/6
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_MULTICAST              0                0            0                0
  ARP                         0                0            0                0
 
   
GigabitEthernet0/6/5/7
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_MULTICAST              0                0            0                0
 
   

This shows that the ingress data rate on GigabitEthernet0/6/5/4 is 43000 bps and that the egress data rate on interface GigabitEthernet0/6/5/5 is also 43000 bps, as expected.

Other flows usually use a different egress interface, because the hask algorithm produces a different hash value. For example, a second flow that is identical to the first one except for the destination port is forwarded out a different interface, GigabitEthernet0/6/5/7:

RP/0/RP0/CPU0:P2_CRS-8# show cef exact-route 192.168.254.1 10.1.2.1 protocol tcp 
source-port 5500 destination-port 23 ingress-interface gi0/6/5/4
 
   
Tue Sep 23 04:03:25.145 PST DST
10.1.2.1/32, version 0, internal 0x40040001 (0xa9482f74) [1], 0x0 (0xa902a9cc), 0x4400 
(0xa9db444c)  
 Updated Sep 19 11:52:34.493 
 remote adjacency to GigabitEthernet0/6/5/7
 Prefix Len 32, traffic index 0, precedence routine (0)
   via GigabitEthernet0/6/5/7
 
   

After the traffic counters stabilize, the display now shows that the ingress flow on GigabitEthernet0/6/5/4 has doubled and that that traffic is distributed equally between two egress interfaces, GigabitEthernet 0/6/5/5 and 0/6/5/7, as expected:

RP/0/RP0/CPU0:P2_CRS-8# show interface accounting rates
 
   
Tue Sep 23 05:57:03.780 PST DST
GigabitEthernet0/6/5/4
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_UNICAST            86000              192            0                0
  IPV4_MULTICAST              0                0            0                0
  ARP                         0                0            0                0
 
   
GigabitEthernet0/6/5/5
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_UNICAST                0                0        43000               96
  IPV4_MULTICAST              0                0            0                0
 
   
GigabitEthernet0/6/5/6
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_MULTICAST              0                0            0                0
 
   
GigabitEthernet0/6/5/7
                                Ingress                        Egress          
  Protocol             Bits/sec         Pkts/sec     Bits/sec         Pkts/sec
  IPV4_UNICAST                0                0        43000               92
  IPV4_MULTICAST              0                0            0                0

In an actual system, there is much more live traffic, so to the extent possible, reduce existing traffic so you can more easily view the test traffic. These results were obtained with a traffic generator that creates TCP flows, but you can obtain similar results using the ping command. Just make sure that the ping packets pass through the router with load balancing interfaces to reach the destination IP address.

Troubleshooting Layer 2 Load Balancing Example

Table 13-2 shows the sample configuration that this section uses to describe the troubleshooting process for Layer 2 load balancing. Two routers, Router A and Router B, connect back to back over three serial Gigabit Ethernet interfaces that are grouped into a bundle.

Table 13-2 Sample Configuration for Layer 2 Load Balancing

Router A
Router B
interface Bundle-Ether12
 description Connected to Router B
 ipv4 address 10.12.48.1 255.255.255.0
!
interface GigabitEthernet0/6/5/5
 bundle id 12 mode active
!
interface GigabitEthernet0/6/5/6
 bundle id 12 mode active
!
interface GigabitEthernet0/6/5/7 bundle id 
12 mode active
!
router ospf 200
!
area 0
!
 interface GigabitEthernet0/6/5/5
 interface GigabitEthernet0/6/5/6
 interface GigabitEthernet0/6/5/7
 
        
interface Bundle-Ether12
 description Connected to Router A
 ipv4 address 10.12.48.2 255.255.255.0
!
interface GigabitEthernet0/6/5/5
 bundle id 12 mode active
!
interface GigabitEthernet0/6/5/6
 bundle id 12 mode active
!
interface GigabitEthernet0/6/5/7
  bundle id 12 mode active
!
router ospf 200
area 0
!
 interface GigabitEthernet0/6/5/5
 interface GigabitEthernet0/6/5/6
 interface GigabitEthernet0/6/5/7
 
        

Verifying the Bundle Status

The configuration in Table 13-2 shows that a bundle is configured. Enter the show bundle command to verify the configuration:

RP/0/RP0/CPU0:RouterB# show bundle bundle-ether12
 
   
Fri Oct 24 02:31:14.331 PST DST
State: 0 - Port is Detached. 1 - Port is Waiting.
       2 - Port is Attached. 3 - Port is Collecting.
       4 - Port is Distributing.
 
   
Bundle-Ether12
  Bandwidth (Kbps)                          Min active          Max active
  Effective   Available    MAC address      Links  B/W (Kbps)   Links
  ----------  ----------   --------------   -----  ----------   -----
     3000000     3000000   0015.6358.b902       1           1      32
 
   
  Port           State  Port ID          B/W (Kbps)  MAC address
  ------------   -----  --------------   ----------  ---------------
  Gi0/6/5/5      4      0x8000, 0x0005      1000000  0015.6358.be38 
  Gi0/6/5/6      4      0x8000, 0x0006      1000000  0015.6358.be39 
  Gi0/6/5/7      4      0x8000, 0x0007      1000000  0015.6358.be3a 
 
   

This verifies that the bundle is active and is using the three expected interfaces (ports). If the bundle on your system is not active, troubleshoot that problem first. It might not have the minimum number of active links or minimum bandwidth to come up.

Checking the IGP Routing Table and CEF Database

This section shows how to verify that Router B uses Bundle-Ether12 to reach Router A's loopback interface, 10.1.2.1. Use the show route command to view the current path to RouterA:

RP/0/RP0/CPU0:RouterB# show route 10.1.2.1
 
   
Fri Oct 24 02:23:48.643 PST DST
 
   
Routing entry for 10.1.2.1/32
  Known via "ospf 200", distance 110, metric 2, type intra area
  Installed Oct 22 02:47:06.383 for 1d23h
  Routing Descriptor Blocks
    10.12.48.2, from 10.1.2.1, via Bundle-Ether12
      Route metric is 2
  No advertising protos.
 
   

This shows that Router B selected Bundle-Ether12 as the single path to 10.1.2.1. Use the show cef command to view the corresponding CEF path:

RP/0/RP0/CPU0:RouterB# show cef ipv4 10.1.2.1
 
   
Fri Oct 24 02:25:13.453 PST DST
10.1.2.1/32, version 0, internal 0x40040001 (0xa93159cc) [1], 0x0 (0xa909b448), 
 Updated Oct 22 02:47:06.391
 Prefix Len 32, traffic index 0, precedence routine (0)
   via 10.12.48.2, Bundle-Ether12, 2 dependencies, weight 0, class 0
    next hop 10.12.48.2
    local adjacency
     local label 16028      labels imposed {None}
 
   

This verifies that CEF also uses Bundle-Ether12 as the egress interface for traffic forwarded to 10.1.2.1. No Layer 3 or Layer 4 load balancing can occur in this configuration because there is only one link. Layer 2 load balancing can occur however, because the single link is a bundle of three separate interfaces.

Verifying Bundle Load Balancing

Cisco IOS XR software has a bundle utility that predicts how load balancing will distribute flows among the bundle links. This is an interactive utility that prompts for the information that the load balancing algorithm uses to assign flows to links. For more information, see Cisco IOS XR Interface and Hardware Component Command Reference.

This example shows how to use the bundle hash utility to determine the link that a specific Layer 4 flow would take, and the links that other flows in the subnet would take:

RP/0/RP0/CPU0:RouterB# bundle-hash bundle-ether 12
 
   
Fri Oct 24 07:03:09.163 PST DST
Specify load-balance configuration (L3/3-tuple or L4/7-tuple) (L3,L4): L4
Single SA:SP/DA:SP pair (IPv4,IPv6) or range (IPv4 only): S/R [S]: s
 
   
Enter bundle type IP V4 (1) or IP V6 (2): 1
Enter source IP V4 address: 192.168.254.1
Enter destination IP V4 address: 10.1.2.1
 
   
  Ingress interface --
  - physical interface format: [ POS | GigabitEthernet | TenGigE ]R/S/I/P
  - bundle interface format:   [ Bundle-Ether | Bundle-POS ]bundle-id
  Enter ingress interface: GigabitEthernet0/6/5/4
 
   
  Enter L4 protocol (TCP,UDP,SCTP,L2TPV3,NONE): TCP
  Enter src port: 5500
  Enter destination port: 80
Compute destination address set for all members? [y/n]: y
Enter subnet prefix for destination address set: 10.0.0.0
Enter bundle IP V4 address [192.168.254.1]: 10.12.48.1
 
   
S/D pair 192.168.254.1:5500/10.1.2.1:80 -- Link hashed to is GigabitEthernet0/6/5/5
 
   
Destination address set for subnet 10.0.0.0:
  S/D pair 192.168.254.1:5500/10.0.0.3:80 hashes to link GigabitEthernet0/6/5/5
  S/D pair 192.168.254.1:5500/10.0.0.9:80 hashes to link GigabitEthernet0/6/5/6
  S/D pair 192.168.254.1:5500/10.0.0.1:80 hashes to link GigabitEthernet0/6/5/7
 
   

This shows that an ingress flow from 192.168.254.2:5500 on interface GigabitEthernet0/6/5/4 to destination 10.1.2.1:80 would use egress interface GigabitEthernet0/6/5/5. You can verify that this occurs by viewing the traffic for each of the bundle links, as shown in the following excerpts from show interfaces:

RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/6
 
   
Tue Oct 28 03:43:18.783 PST DSTGigabitEthernet0/6/5/5 is up, line protocol is up 
  ...
     16196 packets output, 875154 bytes, 0 total output drops
     Output 0 broadcast packets, 12 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   
 
   
RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/6
 
   
Tue Oct 28 03:43:18.783 PST DST
GigabitEthernet0/6/5/6 is up, line protocol is up 
  ...
     3 packets output, 372 bytes, 0 total output drops
     Output 0 broadcast packets, 3 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   
 
   
RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/7
 
   
Tue Oct 28 03:43:23.482 PST DST
GigabitEthernet0/6/5/7 is up, line protocol is up 
  ...
     3 packets output, 372 bytes, 0 total output drops
     Output 0 broadcast packets, 3 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   

This verifies that the flow is using GigabitEthernet0/6/5/5, as expected. The other two links in the bundle have incidental traffic.

Repeat this procedure for other flows. For example, for a flow exactly the same as the first except that the destination port is 23, the bundle hash utility provides the following information:

RP/0/RP0/CPU0:RouterB# bundle-hash bundle-ether 12
 
   
Tue Oct 28 03:56:48.786 PST DST
Specify load-balance configuration (L3/3-tuple or L4/7-tuple) (L3,L4): L4
Single SA:SP/DA:SP pair (IPv4,IPv6) or range (IPv4 only): S/R [S]: S
 
   
Enter bundle type IP V4 (1) or IP V6 (2): 1
Enter source IP V4 address: 192.168.254.1
Enter destination IP V4 address: 10.1.2.1
 
   
  Ingress interface --
  - physical interface format: [ POS | GigabitEthernet | TenGigE ]R/S/I/P
  - bundle interface format:   [ Bundle-Ether | Bundle-POS ]bundle-id
  Enter ingress interface: GigabitEthernet0/6/5/4
 
   
  Enter L4 protocol (TCP,UDP,SCTP,L2TPV3,NONE): TCP
  Enter src port: 5500
  Enter destination port: 23
Compute destination address set for all members? [y/n]: y
Enter subnet prefix for destination address set: 10.0.0.0
Enter bundle IP V4 address [192.168.254.1]: 10.12.48.1
 
   
S/D pair 192.168.254.1:5500/10.1.2.1:23 -- Link hashed to is GigabitEthernet0/6/5/7
 
   
Destination address set for subnet 10.0.0.0:
  S/D pair 192.168.254.1:5500/10.0.0.2:23 hashes to link GigabitEthernet0/6/5/5
  S/D pair 192.168.254.1:5500/10.0.0.1:23 hashes to link GigabitEthernet0/6/5/6
  S/D pair 192.168.254.1:5500/10.0.0.4:23 hashes to link GigabitEthernet0/6/5/7
 
   

This shows that an ingress flow from 192.168.254.2:5500 on interface GigabitEthernet0/6/5/4 to destination 10.1.2.1:23 would use egress interface GigabitEthernet0/6/5/7. You can verify that this occurs by viewing the traffic for each of the bundle links, as shown in the following excerpts from show interfaces:

RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/5   
 
   
Tue Oct 28 04:01:57.568 PST DST
GigabitEthernet0/6/5/5 is up, line protocol is up 
  ...
     1 packets output, 94 bytes, 0 total output drops
     Output 0 broadcast packets, 1 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   
RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/6
 
   
Tue Oct 28 04:02:01.254 PST DST
GigabitEthernet0/6/5/6 is up, line protocol is up 
  ...
     0 packets output, 0 bytes, 0 total output drops
     Output 0 broadcast packets, 0 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   
RP/0/RP0/CPU0:RouterB# show interface Gi0/6/5/7
 
   
Tue Oct 28 04:02:04.704 PST DST
GigabitEthernet0/6/5/7 is up, line protocol is up 
  ...
     18190 packets output, 982260 bytes, 0 total output drops
     Output 0 broadcast packets, 0 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
 
   

This verifies that the flow to port 23 is using GigabitEthernet0/6/5/7, as expected.

For more information on implementing Cisco Express Forwarding on Cisco IOS XR Software, see Cisco IOS XR IP Addresses and Services Configuration Guide for the Cisco CRS-1 Router. For more information on Cisco Express Forwarding commands on Cisco IOS XR Software, see Cisco IOS XR IP Addresses and Services Command Reference for the Cisco CRS-1 Router.

For more information on Cisco IOS Load Balancing Troubleshooting, see Troubleshooting Load Balancing Over Parallel Links Using Cisco Express Forwarding.