Troubleshooting ATM-3-FAILCREATEVC Messages on the PA-A3 [Cisco Port Adapters]

Document ID: 10407

Introduction
Prerequisites
      Requirements
      Components Used
      Conventions
Understanding PA-A3 Architecture
      Hardware Architecture
      Software Architecture
Understanding ATM VC Creation
Known Issues with FAILCREATEVC
Known Bug IDs with the PA-A3
      CSCdt90054: VIP Runs at 99% CPU on RSP Console
Troubleshooting
Related Information

Introduction

In rare circumstances, typically when the ATM router interface is supporting a large number of switched virtual connections (SVCs), your router outputs this log message to report an error or failure condition:

ATM-3-FAILCREATEVC: ATM failed to create VC
(VCD=[dec], VPI=[dec], VCI=[dec]) on Interface[chars],
(Cause of the failure: [chars])

The ATM Messages section of the Cisco IOS Software System Error Messages lists several unrelated reasons why this message may appear. One reason is the driver failing to accept the virtual connection (VC).

%ATM-3-FAILCREATEVC: ATM failed to create VC
(VCD=3503, VPI=0, VCI=1076) on Interface ATM1/0,
(Cause of the failure: Failed to have the driver to accept the VC)

This document describes known causes of this error condition on the PA-A3 ATM port adapter. It also provides an overview of Cisco IOS® Software architecture and explains the driver to which the message is referring.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the Cisco PA-A3 ATM port adapter.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Understanding PA-A3 Architecture

Before reviewing the causes of the FAILCREATEVC message, you first need to understand PA-A3 hardware and software architecture.

Hardware Architecture

Each PA-A3 supports separate transmit and receive segmentation and reassembly (SAR) chips to provide sufficient processing power for high-speed communication over digital signal level 3 (DS3) and optical carrier 3 (OC-3) links. The SAR consists, internally, of hardware functional modules. These modules include the ATM processing unit (APU), which is a “mini” reduced instruction set computing (RISC) chip with customized logic for ATM-specific extensions.

Software Architecture

All ATM interfaces use a software architecture that consists of multiple blocks. Processors on a dedicated CPU module or on the ATM interface itself handle a block of routines. Before walking through these software blocks, you must first understand Cisco IOS Software drivers and the PCI bus architecture inside your router.

Cisco IOS Software contains device drivers for a range of devices, from flash cards and NVRAM to network interfaces. Cisco IOS Software network interface device drivers provide the primary intelligence for packet operations in and out of interfaces.

The PA-A3 supports a peripheral component interconnect (PCI) host driver that allows the SAR’s processor to interface with the PCI buses that run the length of the 7200 and 7400 series, as well as the versatile interface processor (VIP). The PCI host driver and the APU—which runs the ATM firmware—communicate with each other using a messaging mailbox.

Drivers provide additional code that performs platform-dependent or platform-specific functions. In other words, they allow Cisco IOS Software to offer a base set of code and then add drivers to support specific, installed hardware. This process is called hardware abstraction in software development terminology.

Now consider how Cisco IOS Software uses these software components when creating an ATM VC.

Understanding ATM VC Creation

The PCI host driver communicates with the firmware running on the processor (the APU) on the PA-A3. The PCI host driver instructs the firmware to perform a specific task, such as set up a VC. This command is stored in a command buffer. The PA host interface reads the command from the command buffer and processes it. When complete, it returns a “command-done” interrupt across the PCI bus to the host driver.

Similarly, the CPU in the Route/Switch Processor (RSP) communicates with the CPU in the VIP using command control blocks (CCB). These commands are guaranteed, meaning that the VIP platform driver acknowledges the completion of the request.

Consider an example of this communication using the debug atm events command on a 7200 series router. These messages differ slightly when using an RSP and VIP platform.

caution Caution: Before issuing debug commands, refer to Important Information on Debug Commands. The debug atm events command may print a large amount of disruptive debug output on a production router. The amount depends on the number of VCs for which it needs to report statistics and on the amount of VC-related events.

7200(config)# interface atm 5/0.2 multi

7200(config-subif)# pvc 2/2

7200(config-if-atm-vc)# vbr-nrt 10000 9000

7200(config-if-atm-vc)# exit

5d08h: atmdx_setup_vc(ATM5/0): vc:3 vpi:2 vci:2 state:2 config_status:0
5d08h: atmdx_pas_vc_setup(ATM5/0): vcd 3, atm hdr 0x00200020, mtu 4482
5d08h: VBR: pcr 23597, scr 21237, mbs 94
5d08h:  vc tx_limit=353, rx_limit=117

7200(config-subif)# no pvc 2/2

7200(config-subif)#

5d10h: atmdx_teardown_vc(ATM5/0): idb state 4 vcd 3 state 4
5d10h: atmdx_pas_teardown_vc(ATM5/0): vcd 3

7200(config-subif)#

Known Issues with FAILCREATEVC

The PA-A3 gives each SAR a small amount of onboard memory to store data structures like the VC table. Cisco bug ID CSCdp18492 resolves the symptoms that lead to %ATM-3-FAILCREATEVC messages when the transmit SAR misses a command interrupt during heavy set up and tear down of SVCs, such as in a LANE environment. During this condition, the transmit and receive SARs may fall out of sync about the status of a VC, because the receive SAR has configured the VC in its local memory but the transmit SAR has not.

The PCI host driver is responsible for VC table synchronization. Both SARs need to execute a command from the platform driver. The host driver waits for a positive status message from both SARs before returning a positive acknowledgment. If the positive status messages are not received within a short time period, the host driver returns a fail feedback message.

In some cases, your router may reference the same VPI/VCI pair in several %ATM-3-FAILCREATEVC log messages. The likely reason has to do with how the connected ATM switch selects VPI/VCI values to allocate for a new signaled data virtual circuit.

The ATM switch—not the ATM end-system—assigns the VPI/VCI value when it returns a CALL PROCEEDING signaling message to the calling party. Some ATM switches allocate VC values in blocks. These switches scan the VC table for unused values that may be lower than the highest value currently in use. They try to allocate the same unused resource and simply move to the next highest free value when the signaling layer on the router returns a failure message.

For example, suppose an ATM switch creates 100 SVCs with VPI/VCI values between 0/32 and 0/131. If SVCs 0/50 and 0/68 are torn down, the switch will look to reuse these values for a new SVC. ATM router interfaces build a VC table indexed by a virtual circuit descriptor (VCD). If the switch repeatedly loops while trying to assign the same VPI/VCI which the ATM driver did not fully tear down in the original table row, the PA-A3 can get stuck in a loop trying to update its VCD table.

Known Bug IDs with the PA-A3

This section lists known Cisco bug IDs filed against FAILCREATEVC messages with the PA-A3.

Cisco bug ID CSCdt42998—ATM failed to create a VC already in use.

The router reports the FAILCREATEVC messages against the same VPI/VCI pair. Setup or teardown of this VC does not occur completely, and the VC remains stuck in an intermediate state that the “free-VC” software routines cannot correct.
Cisco bug ID CSCdp49223—Deluxe: Tx SAR may miss command interrupt from host driver.

Resolves FAILCREATEVC error conditions caused by the PA-A3’s transmit SAR missing command interrupts from the host CPU. Cisco bug ID CSCdp18492 implements a workaround for CSCdp49223. Cisco bug ID CSCdp42529 fixes a side effect of CSCdp49223.
Cisco bug ID CSCdt90054—%ATM-3-FAILCREATEVC when VIP uses 99 percent of the CPU on the RSP console.

Also refer to Cisco bug ID CSCdt77918.

CSCdt90054: VIP Runs at 99% CPU on RSP Console

In rare cases, when the VIP CPU is running at 99 percent utilization, it cannot allocate processor cycles to receive and execute commands from the RSP. During these conditions, the RSP console may report ATM-3-FAILCREATEVC messages pointing to the failure of the driver to accept the VC. During troubleshooting of Cisco bug ID CSCdt90054, Cisco found that these log messages appeared when the set of ATM VCs shown in driver-level commands on the VIP console was not consistent with the VCs displayed in the output of show atm vc on the RSP console.

It is important to note that 99 percent utilization on a VIP is common. The VIP implements the 7500 series’ distributed architecture, in which both enhanced services like distributed traffic shaping and packet processing are offloaded to the VIP’s CPU. The VIP reaches 99 percent CPU utilization easily with moderate traffic. Thus, VIP CPU utilization is not an accurate gauge of whether the VIP is exhausting its performance.

Receiver-side (Rx-side) buffering is the process that occurs when the outbound interface is congested and when the queueing strategy of the outbound interface is first-in, first-out (FIFO). Rather than dropping the packet immediately, the inbound VIP buffers the packet in its shared RAM until buffers are available for the outgoing interface. A consequence of Rx-side buffering is that the VIP runs at 99 percent CPU utilization. The VIP continuously monitors the status of the transmit queue (txqueue) of the outbound interface and, as soon as there is a free buffer, copies the packet over the CyBus into the txqueue. Refer to Understanding VIP CPU Running at 99% and Rx-Side Buffering for more information.

The difference in VIP functionality between 99 percent and 100 percent CPU utilization is significant. A classic symptom of higher than 99 percent utilization is a slow response (up to several seconds) to commands at the VIP CPU.