Guest

Cisco IGX 8400 Series Switches

IGX MPLS & MPLS VPN

Table Of Contents

Reference Guide

Introduction

Hardware and Software Requirements

MPLS Hardware Setup

MPLS Software configuration

IGX (LSR) Software Config

Configuring the LSC (7200):

Configuring the E-LSR

MPLS Debug commands

MPLS-VPN Hardware configuration

MPLS-VPN Software Configuration

MPLS QoS

Further Reading Recommendations


Reference Guide


IGX MPLS & MPLS-VPN

Introduction

The purpose of this guide is to help reduce the time needed to deploy IGX-based Multiprotocol Label Switching (MPLS) and MPLS-Virtual Private Network (VPN) networks.

It is an implementation helper and the emphasis is put on setup and configuration, additional reading is highly recommended to gain in-depth knowledge (see last section).

Basic knowledge of MPLS technology, IGX and router operation is assumed.

Manuals are to be consulted for an exhaustive equipment and command reference.

Hardware and Software Requirements

Following a typical MPLS architecture, we will build a network made up of Label Switch Routers (LSR), Label Switch Controllers (LSC) and Edge-Label Switch Routers (E-LSR).

MPLS-VPNs are then composed of Customer Edge (CE) and Provider Edge (PE) equipment.

These architectures are illustrated below:

Figure 1 

While many platforms could be used as LSRs, we will focus on the IGX 8400 in this document.

With this assumption, the following table outlines the different options that are available for each network component:

 
Required Hardware
Required Software
LSR

IGX 8400 (focus of this document)

NPM-64 B required

UXM or UXM-E is required to connect LSC and E-LSRs

SWSW 9.3.10 or higher

UXM Firmware Model C

LSC

7200 Router

(FYI: 6400 and 7500 also feature the Tag Switch Controller software, but have not been tested as such, and therefore are not supported at this time, this may change in the future)

PA-A3 or PA-A1 ATM Adapter (PA-A3 does not support multi-VC mode at this time)

IOS 12.1.3(T) and higher

E-LSR

All platforms that support tag-switching (2600,3600,7200,7500,etc.)

ATM port adapter required

IOS 12.0 and higher

CE

Almost Any router

The software chosen should provide a compatible routing protocol with PE router where applicable (irrelevant if static routes are chosen)


MPLS Hardware Setup

As mentioned earlier, all MPLS-enabled IGX nodes must be equipped with NPM-64B processor modules.

Connectivity from the IGX to the LSC and E-LSR is established with UXM or UXM-E modules.

Asynchronous Transfer Mode (ATM) DS-3/E-3 or OC-3 is recommended between the IGX (LSR) and the 7200(LSC), so as to ensure fast transport of the Tag signalling protocol.

Connectivity between the IGX (LSR) and the E-LSRs can be accomplished with almost any ATM physical medium.

The diagram below illustrates these principles:

Figure 2 

MPLS Software configuration

This section features Configuration samples for each part of the network.

Almost any IP routing protocol can be used between the MPLS nodes, we have tested successfully in our lab OSPF and EIGRP.

The examples below will focus on OSPF.

What's more, a hierarchical OSPF implementation has been chosen here, as it is commonly found in Service Provider Networks.

The diagram below outlines the lab setup:

Figure 3 

Notes on Figure 3:

lb stands for Loopback interface

While the IGX does not feature layer 3 functionality at this moment, its physical ports are labeled with IP loopback interfaces because it is most practical for the illustration's sake. Indeed, in an MPLS architecture, the LSC "inherits" the LSR interfaces that are MPLS-enabled. These LSR interfaces become virtual interfaces of the LSC router and behave very much like any other router interface.

IGX (LSR) Software Config

Setup sequence:

1. Configure the LSC ATM port

- Upln
- Cnfln (optional)
- Upport
- cnfrsrc
- Cnfport (optional)

2. Add a VSIcontroller (= the LSC)

- addctrlr

3. Configure VSI-controlled ATM ports or trunks.

Ports
Trunks

Upln

Uptrk

Cnfln

Cnftrk

Upport

Addtrk (only if not VT)

Cnfport

cnfrsrc

Cnfrsrc

 

Important Note: VSI-controlled virtual trunks cannot carry Autoroute traffic, which results in the inability to add the trunk after enabling VSI, or the inability to enable VSI on a trunk that's been added. This only applies to virtual trunks and not to regular trunks.

Let's do this configuration on our sanjose IGX lab node:

1. configuring the LSC ATM port on UXM port 6.6:

upln 6.6

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 10 2000 14:54 PST

LN   6.6 Config       T3/636                UXM slot:6
Loop clock:           No
Line framing:         PLCP
Line length:          0-225 ft.
Idle code:            7F hex
HCS Masking:          Yes
Payload Scramble:     No
VC Shaping:           No

Last Command: cnfln 6.6

Upport 6.6

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 10 2000 15:01 PST

Line : 6.6
Maximum PVC LCNS: 1000            Maximum PVC Bandwidth: 48000
                                  (Reserved Port Bandwidth: 150)


              State   MinLCN   MaxLCN   StartVPI   EndVPI   MinBW    MaxBW
Partition 1:   E       0        1000     2          100      0        48000
Partition 2:   D
Partition 3:   D

Last Command: cnfrsrc 6.6

(Maximum PVC Bandwidth: BW reserved for non-vsi pvcs)
(Reserved BW : BW reserved for vsi controller traffic)

2. Add a VSI controller:

Addctrlr

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 10 2000 15:02 PST

                   VSI Controller Information

CtrlrId   PartId     ControlVC            Intfc    Type     CtrlrIP
                    VPI    VCIRange
   1         1       0      40-70          6.6      MPLS     6.6.6.6

Last Command: dspctrlrs

3. configure VSI-enabled port 7.8

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 11 2000 15:08 PST

Line : 7.8
Maximum PVC LCNS: 256             Maximum PVC Bandwidth: 0



              State   MinLCN   MaxLCN   StartVPI   EndVPI   MinBW    MaxBW
Partition 1:   E       0        1000     2          10       0        3622
Partition 2:   D
Partition 3:   D

Last Command: cnfrsrc 7.8

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 11 2000 15:07 PST

Port:       7.8     [ACTIVE  ]
Interface:          T1-IMA                CAC Override:          Enabled
Type:               UNI                   %Util Use:             Disabled
Speed:              3622 (cps)            GW LCNs:               200
SIG Queue Depth:    640                   Reserved BW:           0 (cps)
Alloc Bandwidth:    0 (cps)
Protocol:           NONE

Last Command: cnfport 7.8

4. Configure VSI-enabled trunk 11.1

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 11 2000 15:11 PST

Trunk : 11.1
Maximum PVC LCNS: 1000            Maximum PVC Bandwidth: 71378
                                  (Statistical Reserve: 5000)

              State   MinLCN   MaxLCN   StartVPI   EndVPI   MinBW    MaxBW
Partition 1:   E       0        1000     2          100      0        3622
Partition 2:   D
Partition 3:   D

Last Command: cnfrsrc 11.1

Configuring the LSC (7200):

tahiti lab router config:

tahiti#sho run
Building configuration...
 
Current configuration:
!
version 12.1
!
hostname tahiti
!
ip subnet-zero
!
ip cef
!
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
!
interface Loopback1
 ip address 6.6.6.6 255.255.255.255
!
interface ATM1/0
 description *** sanjose 6.1 (UXM) ***
 no ip address
 tag-control-protocol vsi
 atm framing cbitplcp
 no atm ilmi-keepalive
!
interface XTagATM78
 ip unnumbered Loopback0
 extended-port ATM1/0 vsi 0x00070800
 tag-switching atm vpi 2-10
 tag-switching ip
!
interface XTagATM111
 ip unnumbered Loopback1
 extended-port ATM1/0 vsi 0x000B0100
 tag-switching atm vpi 2-100
 tag-switching ip
!
router ospf 1
 log-adjacency-changes
 network 4.4.4.0 0.0.0.255 area 1
 network 6.6.6.0 0.0.0.255 area 0
!
ip classless
!
end
 
tahiti#

The "ip route-cache cef" command should be issued on all Xtag interfaces!!! (not displayed here as it is the default setting)

The bombay lab router config is similar to tahiti, and is provided below for your convenience:

bombay#sho run
Building configuration...
 
Current configuration:
!
version 12.1
!
hostname bombay
!
ip subnet-zero
!
ip cef
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
 no ip mroute-cache
!
interface Loopback1
 ip address 5.5.5.5 255.255.255.255
!
interface ATM1/0
 description *** tokyo 8.6 (UXM) ***
 no ip address
 no ip mroute-cache
 tag-control-protocol vsi id 2
 no atm scrambling cell-payload
 no atm ilmi-keepalive
!
!
interface XTagATM61
 ip unnumbered Loopback1
 no ip mroute-cache
 extended-port ATM1/0 vsi 0x00060100
 tag-switching atm vpi 2-10
 tag-switching ip
!
interface XTagATM81
 ip unnumbered Loopback0
 no ip mroute-cache
 extended-port ATM1/0 vsi 0x00080100
 tag-switching atm vpi 2-100
 tag-switching ip
!
router ospf 1
 log-adjacency-changes
 redistribute connected
 network 3.3.3.0 0.0.0.255 area 0
 network 5.5.5.0 0.0.0.255 area 2
!
ip classless
!
end
 
bombay#

Configuring the E-LSR

london lab router config:
london#sho run
Building configuration...
 
Current configuration:
!
version 12.1
!
hostname london
!
!
memory-size iomem 25
ip subnet-zero
!
!
ip cef
!
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface Loopback1
 ip address 7.7.7.7 255.255.255.255
!
interface ATM2/0
 description *** sanjose 7.8 (UXM) ***
 no ip address
 no atm ilmi-keepalive
 no scrambling-payload
!
interface ATM2/0.1 tag-switching
 ip unnumbered Loopback0
 tag-switching atm vpi 2-10
 tag-switching ip
!
!
router ospf 1
 network 1.1.1.0 0.0.0.255 area 1
!
ip classless
!
end
 
london#

The paris lab router config is provided below for your convenience:

paris#sho run
Building configuration...
 

Current configuration:
!
version 12.1
!
hostname paris
!
memory-size iomem 25
ip subnet-zero
!
ip cef
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface Loopback1
 ip address 8.8.8.8 255.255.255.255
!
interface ATM3/0
 description *** tokyo 6.1 (UXM) ***
 no ip address
 no atm ilmi-keepalive
 no scrambling-payload
!
interface ATM3/0.1 tag-switching
 ip unnumbered Loopback0
 tag-switching atm vpi 2-10
 tag-switching ip
!
!
router ospf 1
 network 2.2.2.0 0.0.0.255 area 2
!
ip classless
!
end
 

paris#  q

MPLS Debug commands

Before doing extensive debugging, we have found the following problems to occur frequently:

Payload scrambling mismatch between the router and the IGX: Both have different default settings, which results in a default mismatch. (not MPLS-specific, but related)

LVC vpi-vci range mismatch. There is unfortunately no mechanism today to automatically match LVC vpi-vci ranges between router and IGX, or IGX and IGX (the Cisco Wan Manager team is working on a tool to compare each side of a MPLS-enabled UXM trunk). This problem can be identified by turning on tag-switching TDP debug.

"ip route-cache cef" has not been enabled on all Tag interfaces.

While there are many debug commands available for MPLS, we have found the following to be most useful:

On the IGX side:

Dspvsich:

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 11 2000 16:38 PST

                             VSI lcns for Slot 6

    lcn       type                     dest slot   dest lcn   vpi     vci
     259       ctrl chan - mstr end     local       -          0       44
     260       ctrl chan - mstr end     7           8159       0       45
     264       ctrl chan - mstr end     11          8159       0       49
     8133      interslave               7           8132       -       -
     8137      interslave               11          8132       -       -

Last Command: dspvsich 6

Dspvsipartinfo:

sanjose        TN    Cisco           IGX 8430  9.3.1T    Sep. 11 2000 16:40 PST

            VSI Resources Status for port  6.6 Partition 1           Snapshot

Minimum Lcns       :       0     Minimum BW   (cps) :       0
Maximum Lcns       :    1000     Maximum BW   (cps) :   48000
Used Lcns          :      12     Used BW      (cps) :       0
Available Lcns     :     988     Available BW (cps) :   48000
Start VPI          :       2     End VPI            :     100

Last Command: dspvsipartinfo 6.6 1

sanjose        TN    Cisco           GX 8430þ 9.3.1Tþþþ Sep. 20 2000 15:39 PST
 
VSI Partitions on this node
 
 Interface (slot.port)þ Part 1þ Part 2þ Part 3
 Trunkþ 6.1þþþþþþþþþþþþþþ Dþþþþþþ Dþþþþþþ D
 Lineþþ 6.6þþþþþþþþþþþþþþ Eþþþþþþ Dþþþþþþ D
 Trunkþ 7.1þþþþþþþþþþþþþþ Dþþþþþþ Dþþþþþþ D
 Lineþþ 7.8þþþþþþþþþþþþþþ Eþþþþþþ Dþþþþþþ D
 Trunkþ 11.1þþþþþþþþþþþþþ Eþþþþþþ Dþþþþþþ D
 
Last Command: dsprsrc

sanjoseþþþþþþþ TNþþþ Ciscoþþþþþþþþþþ IGX 8430þ 9.3.1Tþþþ Sep. 20 2000 15:40 PST
 
Channel Information:
(a - available, c - configured, ms - mstr-slv, u - used, mn - min, mx - max, nw - networking cha
nnels, gw - gateway channels)
þþþþþþþþ totalþþþ availþþþ pvc_cþþþ nw_cþþþþ gw_cþþþþ gw_aþþþþ vsi_msþþ vsi_c
card 6:þþ 8000þþþþ 5444þþþþ 1256þþþþ 270þþþþþ 670þþþþþ 3330þþþþ 30þþþþþþ 1000
card 7:þþ 8000þþþþ 6218þþþþ 512þþþþþ 270þþþþþ 670þþþþþ 3330þþþþ 0þþþþþþþ 1000
card 11:þ 8000þþþþ 5730þþþþ 1000þþþþ 270þþþþþ 470þþþþþ 3530þþþþ 0þþþþþþþ 1000
 
Last Command: dspchuse

On the IOS side (LSC):

Show controllers vsi descriptor:

tahiti#show controllers vsi descriptor
 
Phys desc: 0.6.6.0
Log intf:  0x00060600 (0.6.6.0)
Interface: switch control port
IF status: n/a                   IFC state: ACTIVE
Min VPI:   2                     Maximum cell rate:  48000
Max VPI:   100                   Available channels: 1000
Min VCI:   32                    Available cell rate (forward):  48000
Max VCI:   65535                 Available cell rate (backward): 48000
 
Phys desc: 0.7.8.0
Log intf:  0x00070800 (0.7.8.0)
Interface: XTagATM78
IF status: up                    IFC state: ACTIVE
Min VPI:   2                     Maximum cell rate:  3622
Max VPI:   10                    Available channels: 1000
Min VCI:   32                    Available cell rate (forward):  3622
Max VCI:   65535                 Available cell rate (backward): 3622
 
Phys desc: 0.11.1.0
Log intf:  0x000B0100 (0.11.1.0)
Interface: XTagATM111
IF status: up                    IFC state: ACTIVE
Min VPI:   2                     Maximum cell rate:  3622
Max VPI:   100                   Available channels: 1000
Min VCI:   32                    Available cell rate (forward):  3622
Max VCI:   65535                 Available cell rate (backward): 3622
 
tahiti#

This command is extremely useful to verify that the Xtag interfaces created manually on the router did bind with the advertised interfaces from the IGX...

This is also a great command to find out which interfaces have been MPLS-enabled on the IGX from a router's perspective.

IF status should be up, except for the LSC control port, which should be n/a.

Show controllers vsi session:

tahiti#show controllers vsi session
Interface    Session  VCD    VPI/VCI    Switch/Slave Ids   Session State
ATM1/0       0        1      0/40       0/0                UNKNOWN
ATM1/0       1        2      0/41       0/0                UNKNOWN
ATM1/0       2        3      0/42       0/0                UNKNOWN
ATM1/0       3        4      0/43       0/0                UNKNOWN
ATM1/0       4        5      0/44       0/6                ESTABLISHED
ATM1/0       5        6      0/45       0/7                ESTABLISHED
ATM1/0       6        7      0/46       0/0                UNKNOWN
ATM1/0       7        8      0/47       0/0                UNKNOWN
ATM1/0       8        9      0/48       0/0                UNKNOWN
ATM1/0       9        10     0/49       0/11               ESTABLISHED
ATM1/0       10       11     0/50       0/0                UNKNOWN
ATM1/0       11       12     0/51       0/0                UNKNOWN
ATM1/0       12       13     0/52       0/0                UNKNOWN
ATM1/0       13       14     0/53       0/0                UNKNOWN
tahiti#

Some sessions should be established, if not, there is a problem. (thank you, Sherlock).

MPLS-VPN Hardware configuration

There is no limitation to the physical medium that can be used to connect the CE and PE routers.

MPLS-VPN Software Configuration

No specific changes are required on the IGX to add VPNs to the MPLS network. The LSC(7200) configuration can also remain untouched.

The following diagram illustrates the lab MPLS-VPN setup:

Figure 4 

The following IP routing scheme has been implemented in this setup:

Figure 5 

Revised E-LSR 3640 (PE) configs:

paris#sho run
Building configuration...
 
Current configuration:
!
version 12.1
!
hostname paris
!
memory-size iomem 25
ip subnet-zero
!
ip vrf vpn-tme
 rd 1:1
 route-target export 1:1
 route-target import 1:1
ip cef
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface Loopback1
 ip address 8.8.8.8 255.255.255.255
!
interface Serial1/1
 description *** antwerp se1 ***
 ip vrf forwarding vpn-tme
 ip unnumbered Loopback1
 clockrate 2015232
!
interface ATM3/0
 description *** tokyo 6.1 (UXM) ***
 no ip address
 no atm ilmi-keepalive
 no scrambling-payload
!
interface ATM3/0.1 tag-switching
 ip unnumbered Loopback0
 tag-switching atm vpi 2-10
 tag-switching ip
!
!
router ospf 1
 network 2.2.2.0 0.0.0.255 area 2
!
router bgp 1
 neighbor 1.1.1.1 remote-as 1
 !
 address-family ipv4 vrf vpn-tme
 redistribute static
 no auto-summary
 no synchronization
 exit-address-family
 !
 address-family vpnv4
 neighbor 1.1.1.1 activate
 neighbor 1.1.1.1 send-community extended
 exit-address-family
!
ip classless
ip route vrf vpn-tme 10.10.10.0 255.255.255.0 Serial1/1
!
end
 
paris# 

london#sho run
Building configuration...
 
Current configuration:
!
hostname london
!
memory-size iomem 25
ip subnet-zero
!
ip vrf vpn-tme
 rd 1:1
 route-target export 1:1
 route-target import 1:1
ip cef
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface Loopback1
 ip address 7.7.7.7 255.255.255.255
!
interface ATM2/0
 description *** sanjose 7.8 (UXM) ***
 no ip address
 no atm ilmi-keepalive
 no scrambling-payload
!
interface ATM2/0.1 tag-switching
 ip unnumbered Loopback0
 tag-switching atm vpi 2-10
 tag-switching ip
!
interface Serial3/1
 description *** osaka se1 ***
 ip vrf forwarding vpn-tme
 ip unnumbered Loopback1
 clockrate 2015232
!
router ospf 1
 network 1.1.1.0 0.0.0.255 area 1
!
router bgp 1
 neighbor 2.2.2.2 remote-as 1
 !
 address-family ipv4 vrf vpn-tme
 redistribute static
 no auto-summary
 no synchronization
 exit-address-family
 !
 address-family vpnv4
 neighbor 2.2.2.2 activate
 neighbor 2.2.2.2 send-community extended
 exit-address-family
!
ip classless
ip route vrf vpn-tme 9.9.9.0 255.255.255.0 Serial3/1
!
end
 
london#

Customer Edge (CE) 3810 Configs:
osaka#sho run
Building configuration...
 
Current configuration:
!
version 12.0
!
hostname osaka
!
ip subnet-zero
!
interface Serial1
 description *** london se3/1 ***
 ip unnumbered Loopback0
 no ip directed-broadcast
no ip mroute-cache
!
ip classless
ip route 0.0.0.0 0.0.0.0 Serial1
ip route 10.10.10.10 255.255.255.255 Serial1
!
end
 
osaka#

antwerp#sho run
Building configuration...
 
Current configuration:
!
version 12.0
!
hostname antwerp
!
ip subnet-zero
!
interface Loopback0
 ip address 10.10.10.10 255.255.255.255
 no ip directed-broadcast
!
interface Serial1
 description *** paris se1/1 ***
 ip unnumbered Loopback0
 no ip directed-broadcast
 no ip mroute-cache
!
ip classless
ip route 0.0.0.0 0.0.0.0 Serial1
ip route 9.9.9.9 255.255.255.255 Serial1
!
!
end
 
antwerp#

MPLS QoS

This lab test does not cover MPLS multi-VC mode.

Multi-VC mode requires the use of PA-A1 port adapters on the 7200 routers (as opposed to PA-A3).

The IGX supports multi-vc mode by way of its user-configurable qbins (cnfqbin,...), which can be left at their default settings for most networks.

Multi-vc mode needs only to be enabled on the E-LSR (no special configuration required on IGX or LSC).

When Multi-VC mode is enabled, IP packets are differentiated in the E-LSR by the IP Type of Service (TOS) bits, and then sent on different parallel permanent virtual circuits (PVCs). These PVCs will use different user-configurable queues end-to-end (qbins), which helps maintain different Quality of Service (QoS) levels for each of them.

Further Reading Recommendations

Update to the Cisco IGX 8400 Series Reference Guide (MPLS related IGX documentation):

http://www.cisco.com/en/US/products/hw/switches/ps988/tsd_products_support_reference_guides.html

Your feedback is welcome, let us know how useful this document has been for you, e-mail Frederic Laruelle at flaruell@cisco.com.