Troubleshooting PVC Failures When Using OAM Cells and PVC Management

Introduction

If a communication problem occurs on a PVC (no traffic going one way or the other), the permanent virtual circuit (PVC) remains UP on the end-devices. Therefore, routing entries that were pointing to that PVC remain in the routing table for a certain time and as a result, packets will be lost. The solution to this problem is to use Operation and Maintenance (OAM) to detect such failures and allow the PVC to disconnect if it is disrupted along its path.

You can view a sample configuration on using OAM for PVC management by clicking here.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

There are no specific prerequisites for this document.

Components Used

OAM and PVC management are supported since Cisco IOS® version 11.1(22)CC and in Cisco IOS version 12.0 and later.

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Network Diagram

This document is based on the following setup:

• 1/116 is the VPI/VCI allocated to the virtual circuit (VC) on the complete path.

• The ATM switches are running Cisco IOS 12.0. The ATM switches have been configured to send the Alarm Indication Signal/Remote Defect Indicator (AIS/RDI) upon link failure, as explained in this document.

• You can produce failures by shutting down the (sub)interface on Guilder and observing what happens on Bernard. We enabled service timestamps debug datetime msec in the configurations for all of the debugs in this document. This allows us to see the time of each events in msec.

Detecting Failures

We will only consider F5 OAM (VC level) cells for this document since these are the only ones used by Cisco end-devices (routers) to detect failures. In order to detect a failure along the PVC path on an end-device, OAM uses these specific cells:

• Loopback cells

• Continuity Check (CC) cells

• Alarm Indication Signal (AIS) cells

• Remote Detection Indication (RDI) cells

There are three conditions to declare a PVC UP:

• The router receives a configured number of successive end-to-end F5 OAM loopback cell replies.

• The router does not receive F5-AIS cells for 3 seconds.

• The router does not receive F5-RDI cells for 3 seconds.

The next section describes these cells and outputs showing their effects.

OAM Loopback Cells

At regular intervals, end-devices (such as routers) configured for OAM send loopback cells which must be looped in the network. This looping point can be the machine at the end of the PVC (end-to-end loopback cells) or an equipment on the path (segment loopback cells).

Identifiers in the loopback cell indicate which device(s) should loop the cell. A Cisco device that terminates a VC when receiving such a cell on a PVC will loop it even if it is not configured for OAM. Also, each of these cells will contain a "direction" indicator (to identify if it is a command or response cell) and a sequence number (called Correlation tag or CTag in the debugs). The "command" loopback cell and the "response" loopback cell will have the same sequence number.

The following diagram illustrates loopback (LB) cells:

Sample Debug Output

The following shows the debugs (debug atm oam) that illustrates the loopback cells on Bernard:

Mar 30 14:22:39.050: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:17128
Tries:0
Mar 30 14:22:39.050: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:42E9
Mar 30 14:22:39.050: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0CTag:42E9
Mar 30 14:22:48.958: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:17129
Tries:0
Mar 30 14:22:48.958: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:42EA
Mar 30 14:22:48.958: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0CTag:42EA

Comments on Sample Debug Output

• The first line indicates that the timer used to identify when a loopback cell is to be emitted on a (sub)interface has expired.

• A command loopback cell is then sent out on the corresponding interface (second line of the debugs). The CTag value displayed on this line is the hexadecimal value of the first line CTag plus one.

• A looped loopback cell is then received with a LoopInd equal to zero.

Note: LoopInd=1 indicates a command cell and LoopInd=0 indicates a response (looped) cell. LoopInd=1 does not display in the debugs, but would appear on a sniffer trace.

Sample Debug Output (if loopback cells are lost)

Consider a device (using PVCs) configured to send OAM cells and using PVC management. If this equipment loses a certain number of loopback cells, it puts the PVC in a Down state. See the following debugs:

Mar 30 14:48:31.704: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 
Status:2 CTag:17284
Tries:0
Mar 30 14:48:31.704: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:4385


At this point, the sub-interface corresponding to PVC 1/116 on Guilder is shut down


Mar 30 14:48:41.684: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17285
Tries:0
Mar 30 14:48:41.684: atm_oam_setstate - VCD#4, VC 1/116: newstate = Down Retry <-no reply to the loopback cell just sent
Mar 30 14:48:41.684: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:4386
Mar 30 14:48:42.680: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17286
Tries:1
Mar 30 14:48:42.680: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:4387
Mar 30 14:48:43.680: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17287
Tries:2
Mar 30 14:48:43.680: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:4388
Mar 30 14:48:44.680: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17288
Tries:3
Mar 30 14:48:44.680: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:4389
Mar 30 14:48:45.676: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17289
Tries:4
Mar 30 14:48:45.676: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:438A
Mar 30 14:48:46.676: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:17290
Tries:5 <- the router makes 5 retries before declaring the PVC down
Mar 30 14:48:46.676: atm_oam_setstate - VCD#4, VC 1/116: newstate = Not Verified 
<-5 retries and no answers -> PVC declared down
Mar 30 14:48:46.676: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM2/0/0.116,changed state to down
Mar 30 14:48:46.676: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:438B

You can configure the amount of lost cells needed to put the PVC down. The following show atm pvc vpi/vci command explains the previous debugs.

Bernard# sh atm pvc 1/116
ATM2/0/0.116: VCD: 4, VPI: 1, VCI: 116
UBR, PeakRate: 155000
AAL5-LLC/SNAP, etype:0x0, Flags: 0xC20, VCmode: 0x0
OAM frequency: 10 second(s), OAM retry frequency: 1 second(s)
OAM up retry count: 3, OAM down retry count: 5
OAM Loopback status: OAM Sent
OAM VC state: Not Verified
ILMI VC state: Not Managed
VC is managed by OAM.
InARP frequency: 15 minutes(s)
InPkts: 4, OutPkts: 4, InBytes: 280, OutBytes: 300
InPRoc: 2, OutPRoc: 0, Broadcasts: 5
InFast: 0, OutFast: 0, InAS: 2, OutAS: 0
InPktDrops: 0, OutPktDrops: 364240961
CrcErrors: 0, SarTimeOuts: 0, OverSizedSDUs: 0
Out CLP=1 Pkts: 0
OAM cells received: 9
F5 InEndloop: 9, F5 
InSegloop: 0, F5 InAIS: 0, F5 InRDI: 0
F4 InEndloop: 0, F4 InSegloop: 0, F4 InAIS: 0, F4 InRDI: 0
OAM cells sent: 18
F5 OutEndloop: 18, F5 OutSegloop: 0, F5 OutRDI: 0
F4 OutEndloop: 0, F4 OutSegloop: 0, F4 OutRDI: 0
OAM cell drops: 0
Status: DOWN, State: NOT_VERIFIED

As you can see, F5 loopbacks were sent, but not answered (18 F5 OutEndloop but only 9 F5 InEndloop; therefore, 9 F5 looped loopback cells were lost.). This caused the PVC to go down (as PVC management is configured). F5 OutEndloop represents the number of loopback cells sent and F5 InEndloop represents the number of F5 loopback cells received.

As you can also see, F4 OAM cell counters are present, but nothing is being recorded since only F5 cells are considered here. From the show command output above, other interesting information can be collected regarding loopback cells:

• OAM cells are sent every 10 seconds regardless of whether the PVC is up or down.

• If the PVC is up but the other end is not responding, the router tries to send OAM cellsevery second until it receives an answer or until 5 OAM cells have not been answered. Then the PVC goes down (see debugs above).

• On the other end, if the PVC is down and it suddenly receives a valid looped cell, it will try to re-send LB cells every second until 3 valid looped loopbacks cells in a row are received. Then the PVC will go up again. See the debugs below.

Mar 31 12:40:10.154: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM2/0/0.116, changed state to down
Mar 31 12:40:20.074: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:1 CTag:25267
Tries:6
Mar 31 12:40:20.074: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:62B4
Mar 31 12:40:20.074: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0 CTag:62B4
Mar 31 12:40:20.074: atm_oam_setstate - VCD#4, VC 1/116: newstate = Up Retry 

! PVC was down and suddenly receives a valid response loopback cell

Mar 31 12:40:21.070: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:25268
Tries:0
Mar 31 12:40:21.070: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:62B5
Mar 31 12:40:21.070: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0 CTag:62B5 

! first looped LB cell

Mar 31 12:40:22.066: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:25269
Tries:0
Mar 31 12:40:22.066: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:62B6
Mar 31 12:40:22.066: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0 CTag:62B6 

! second looped LB cell in a row

Mar 31 12:40:23.062: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:25270
Tries:0
Mar 31 12:40:23.062: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:62B7
Mar 31 12:40:23.062: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0 CTag:62B7 

! third looped LB cell in a row

Mar 31 12:40:23.062: atm_oam_setstate - VCD#4, VC 1/116: newstate = Verified 

! PVC is declared up again

Mar 31 12:40:23.062: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM2/0 0.116, changed state to up

As you can see, the sub-interface (hence the PVC) was brought up again after the reception of three valid response loopback cells in a row.

Note: The user can configure all of the parameters described above, as well as use the show atm pvc vpi/vci command to check the parameters.

Alarm Indication Signal/Remote Defect Indicator (AIS/RDI)

Upon detection of a failure, a device configured for OAM sends AIS frames downstream and sends RDI frames upstream.

The following example illustrates the AIS and RDI cells. Suppose that the Rx signal disappears on a switch. The failure in this case is called a Loss of Signal (LOS). The switch that detected it sends an AIS downstream compared to the failure and an RDI upstream compared to the failure.

When receiving such cells, an end-device configured for PVC management brings down the affected PVC(s). These AIS and RDI cells are sent using the same VPI/VCI as the user cells on the PVC. Furthermore, the device sends these cells every second until the failure disappears.

Sample Debug Output

You can detect a failure in several ways:

• A lower OAM level (F1 AIS, Loss Of Signal, and so on) reports it.

• The reception of an AIS or RDI triggers it.

• The device no longer receives CC cells.

A Continuity Check (CC) cell is a cell which devices configured for OAM regularly send and use to check the "link" integrity. Cisco routers don't send these cells so they are not discussed here. For more information regarding OAM CC cells, refer to ITU-T I.610.

The following debug shows what happens on a router configured for PVC management upon reception of an AIS/RDI cell:

Mar 31 13:11:18.990: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:25470
Tries:0
Mar 31 13:11:18.990: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:637F
Mar 31 13:11:18.990: ATM OAM LOOP(ATM2/0/0) I: VCD#4 VC 1/116 LoopInd:0 CTag:637F

At this point, the PVC on Bernard goes down (main interface on Guilder is shut down):

Mar 31 13:11:28.894: ATM OAM(ATM2/0/0.116): Timer: VCD#4 VC 1/116 Status:2 CTag:25471
Tries:0
Mar 31 13:11:28.894: ATM OAM LOOP(ATM2/0/0.116) O: VCD#4 VC 1/116 CTag:6380
Mar 31 13:11:29.806: atm_oam_ais(ATM2/0/0): AIS signal, failure=0x6A, VC 1/116
Mar 31 13:11:29.806: atm_oam_setstate - VCD#4, VC 1/116: newstate = AIS/RDI
Mar 31 13:11:29.806: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM2/0/0.116, changed state to down
Mar 31 13:11:30.806: atm_oam_ais(ATM2/0/0): AIS signal, failure=0x6A, VC 1/116
Mar 31 13:11:31.806: atm_oam_ais(ATM2/0/0): AIS signal, failure=0x6A, VC 1/116
Mar 31 13:11:32.806: atm_oam_ais(ATM2/0/0): AIS signal, failure=0x6A, VC 1/116

You can check the new PVC state with the following command:

Bernard# sh atm pvc 1/116
ATM2/0/0.116: VCD: 4, VPI: 1, VCI: 116
UBR, PeakRate: 155000
AAL5-LLC/SNAP, etype:0x0, Flags: 0xC20, VCmode: 0x0
OAM frequency: 10 second(s), OAM retry frequency: 1 second(s)
OAM up retry count: 3, OAM down retry count: 5
OAM Loopback status: OAM Sent
OAM VC state: AIS/RDI
ILMI VC state: Not Managed
VC is managed by OAM.
InARP frequency: 15 minutes(s)
InPkts: 4, OutPkts: 2, InBytes: 140, OutBytes: 60
InPRoc: 0, OutPRoc: 0, Broadcasts: 0
InFast: 0, OutFast: 0, InAS: 4, OutAS: 2
InPktDrops: 0, OutPktDrops: 0
CrcErrors: 0, SarTimeOuts: 0, OverSizedSDUs: 0
Out CLP=1 Pkts: 0
OAM cells received: 14
F5 InEndloop: 0, F5 InSegloop: 0, F5 InAIS: 14,
F5 InRDI: 0
F4 InEndloop: 0, F4 InSegloop: 0, F4 InAIS: 0, F4 InRDI: 0
OAM cells sent: 15
F5 OutEndloop: 1, F5 OutSegloop: 0, F5 OutRDI: 14
F4 OutEndloop: 0, F4 OutSegloop: 0, F4 OutRDI: 0
OAM cell drops: 0
Status: DOWN, State: NOT_VERIFIED

As you can see, the PVC went down because it received an F5 AIS or RDI signal (in this particular case an AIS). You can also see that the router generated F5 RDI cells upon reception of the F5 AIS cells.

The following example illustrates activity on the two switches on the path:

• On LS1010-1:

  1d03h: % OAM Pkt Rcv
  1d03h: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-END-LPBK 
  
  ! OAM LB cell
  
  1d03h: % OAM Pkt Sent
  1d03h: % Intf: 0/0/1 VPI: 1 VCI: 116 OAM: F5-END-LPBK 
  
  ! OAM LB cell
    

  At this point, PVC goes down on Guilder:

  1d03h: % OAM Pkt Rcv
  1d03h: % Intf: 0/0/1 VPI: 1 VCI: 116 OAM: F5-AIS 
  
  ! AIS cell sent downstream by LS1010-2 upon detection of the failure
  
  1d03h: % OAM Pkt Sent
  1d03h: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-AIS
  1d03h: % OAM Pkt Rcv
  1d03h: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-RDI 
  
  ! RDI sent by Bernard upstream compared to the failure
  
  1d03h: % OAM Pkt Sent
  1d03h: % Intf: 0/0/1 VPI: 1 VCI: 116 OAM: F5-RDI 
  
  ! Bernard's RDI forwarded upstream
  
  1d03h: % OAM Pkt Rcv
  1d03h: % Intf: 0/0/1 VPI: 1 VCI: 116 OAM: F5-AIS
  
  1d03h: % OAM Pkt Sent
  1d03h: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-AIS

  And so on until the failure is eliminated.

• On LS1010-2:

  Upon detection of the failure (in this case Rx-signal disappears on int atm 1/1/2 connected to Guilder), AIS cells are sent downstream to LS1010-1:

  Mar 31 13:17:09.847: % OAM Pkt Sent
  Mar 31 13:17:09.847: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-AIS
  
  Mar 31 13:17:10.847: % OAM Pkt Sent
  Mar 31 13:17:10.847: % Intf: 0/0/0 VPI: 1 VCI: 116 OAM: F5-AIS

  As you can also see in all of the debugs so far, all of the F5 OAM cells are sent on VPI 1 VCI 116, which is the VPI/VCI used by the user's cells.

debug and show Commands

• debug atm oam (on routers)

• show atm pvc vpi/vci with 12.0 and 12.0T

• show atm vc <vcd> with 11.1CC

• show int atm x[/y/[z]]].w (we recommend you use show atm pvc when possible instead of show int atm x) with 12.0

Related Information

• Using OAM for PVC Management
• ATM Technology Support Pages
• CRC Troubleshooting Guide for ATM Interfaces
• Troubleshooting PVC Failures When Using OAM Cells and PVC Management
• Tools & Resources
• Technical Support & Documentation - Cisco Systems