Guest

Cisco ONS 15454 Series Multiservice Provisioning Platforms

Cisco ONS 15454 ML Card Troubleshooting

Cisco - Cisco ONS 15454 ML Card Troubleshooting

Document ID: 59863

Updated: Nov 14, 2005

   Print

Introduction

This document uses a simple test topology to describe how to troubleshoot Multi-Layer (ML) cards on Cisco ONS 15454. The Appendix section provides some basic configuration commands, and detailed topology information.

The test uses an empirical approach to understand the network faults associated with ML cards. The test injects known faults or configurations in order to capture and analyze expected results. The Fault Isolation Case Studies present these findings.

The document follows the typical troubleshooting methodologies. The document presents a symptom, and discusses the relevant fault isolation steps, and also provides generic troubleshooting procedures.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

  • Cisco ONS 15454

  • Cisco ONS 15454 ML-Series Ethernet Cards

  • Cisco IOS

  • Bridging and IP Routing

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Router 7603 that runs Cisco IOS® Software Release 12.1(13)E13

  • Cisco ONS 15454 that runs Cisco ONS Release 4.1.3

  • ML (bundled as part of the ONS 4.1.3 release) that runs Cisco IOS Software Release 12.1(19)EO1

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Basic ML Architecture

The Cisco ML-Series cards for the ONS 15454 platform provide 10/100/1000 Mbps Ethernet connectivity over SONET/SDH at layer 2 and layer 3. Each ML card in the chassis runs one independent IOS image. Creation of a cross-connect circuit in Cisco Transport Controller (CTC) between ML ports creates virtual backend Packet over SONET (POS) ports. In software releases 4.6 and later, creation of POS ports always occurs, but the ports come up only when a cross-connect circuit creation occurs in CTC.

The ML1000-2 card has two POS ports (0 and 1). Each port has up to Synchronous Transport Signal (STS)-24c bandwidth and a total of STS-48c per card. Each POS port supports subinterfaces to allow VLAN trunking. Physical mapping of a POS port to an optical port occurs during the circuit creation phase, and can change during optical span change. Thus, two POS ports on two ends of the circuit are peers, and their configurations need to match.

The mapping between an Ethernet port and a POS port depends on the topology requirement. Layer 2 switching topology ties these two types of ports together with the same bridge-group number. Layer 3 topology routes packets between these interfaces.

Basic Test Topology

Figure 1 represents the test topology:

Figure 1 – Test Topology

f1_59863.gif

In order to set up the test topology:

  1. Connect two Cisco 7603 routers to ONS nodes over Gigabit Ethernet, and ensure that both ports on the two routers are on the same IP subnet. Here, each ONS node has one ML1000-2 card in slot 12.

  2. Configure a bridge-group 100 for Gig0 and POS0 on both ONS nodes.

    Note: You do not need to use POS1 in this test.

    The circuit between the two ML POS0 ports is STS-12c.

  3. Disable IP routing on ML cards.

  4. Provision OC12 1+1 protection between the two ONS nodes. See Figure 1 for the relevant information.

    Note: Both ONS nodes run Cisco ONS Release 4.1.3.

Fault Insertion Case Studies

This section examines the results of various known faults and some common operations. Each case study describes the operation, and the results on ML and ONS.

General Information

Some Relevant Commands to Troubleshoot Issues on ML IOS

show ons alarm
show ip interface brief
clear counters
show interface summary
show interface <gig/pos>
show controller pos
show cdp neighbor
show bridge verbose
show vlans <vlan-id>
show sdm l2-switching forwarding
show ons provisioning-agent message ports
show running
show log
show tech-support

Ensure the usage of a correct time stamp for buffer logging, and check whether the Timing Communication and Control (TCC) is set with the correct date and time. Here is a sample configuration output on ML:

service timestamps debug uptime
 service timestamps log datetime msec localtime
 logging buffered 4096 debugging 

These alarms automatically trigger POS link status change:

PAIS
PLOP
PTIM 
PUNEQ 
PRDI 
PPLM 
PPDI 
BER_SF_B3

Note: The ONS 15454 platform uses two formats to report alarms. For example, PAIS appears in IOS (ML), while AIS-P appears in CTC. PAIS and AIS-P represent the same type of alarm.

Common Verifications in CTC

Alarms
Conditions
History
Circuit
Inventory
Port PM counters
Diagnostics file
Audit trail
  • On ML card:

    • Maintenance/Performance Ether Ports: check for errors.

    • Maintenance/Performance POS Ports: check for errors.

  • On the OC12 Working card:

    • Enable IPPM on Provisioning/SONET STS.

    • Performance: check for errors.

Case Studies

This section describes various potential failure points, and explains how to capture the correct information for problem resolution.

Ethernet Disconnect

This alarm appears on .225 when you pull the Ethernet cable:

.225ML12#show ons alarm
Equipment Alarms
Active: None 
Port Alarms 
  POS0  Active: None 
  POS1  Active: None 
  GigabitEthernet0  Active: CARLOSS 
  GigabitEthernet1  Active: None 

POS0
Active Alarms : None
Demoted Alarms: None 

POS1 Interface not provisioned 

Note: If you force up the ML GigE interface, ML does not notice that the link is down.

The same alarm appears in CTC of .225 (see Figure 2).

Figure 2 – Alarm in CTC

f2_59863.gif

The loss of Cisco Discovery Protocol (CDP) neighbor to 7603a confirms the problem.

Note: The status of GigE 0 does not affect the POS 0 interface (the interface is still Up/Up).

OC12 Protection Switch

OC12 Protection Switch does not create any alarms or errors.

Both OC12 Out of Service (OOS)

When both OC12 ports on .252 node change to OOS, .225 reports AIS-P, which causes the POS 0 interface to go down, and lead to TPTFAIL.

.225ML12#show ons alarm
Equipment Alarms
Active: RUNCFG-SAVENEED
  
Port Alarms
  POS0  Active: TPTFAIL
  POS1  Active: None
  GigabitEthernet0  Active: None  
  GigabitEthernet1  Active: None 

POS0 
Active Alarms : PAIS 
Demoted Alarms: None
 
POS1
Interface not provisioned

XC Side Switch

This log entry appears on the ML of the node that XC is switched. Note that XCON B is slot 10 XC.

May 24 09:55:27.402: %CARDWARE-5-XCON_SWITCH: Switched XCON to B
May 24 09:55:27.406: %CARDWARE-6-BTC_DRV: Init BTC, BTC Rev = 2, Backplane = 0, 
Port = 0

TCC Side Switch

Figure 3 displays the registered alarm.

Figure 3 – TCC Side Switch Alarm

f3_59863.gif

Note: If you use CTC or reverse telnet to connect to the ML card, you lose the connection to the ML card.

After a few minutes, the alarm must clear. These log entries appear in ML:

May 24 10:29:09.258: %CARDWARE-5-SOCKET_INFO: closed socket to TCC: 
changed active TCC 
May 24 10:29:09.766: %ONS-6-VTY: All Vty lines cleared 
May 24 10:29:14.762: %CARDWARE-5-SOCKET_INFO: cannot connect socket to TCC: B 
May 24 10:29:20.270: %CARDWARE-5-SOCKET_INFO: cannot connect socket to TCC: B 
May 24 10:29:25.770: %CARDWARE-5-SOCKET_INFO: cannot connect socket to TCC: B 
May 24 10:29:31.270: %CARDWARE-5-SOCKET_INFO: cannot connect socket to TCC: B 
May 24 10:29:36.370: %CARDWARE-5-SOCKET_INFO: open socket to TCC: B 
May 24 10:29:41.166: %CARDWARE-6-BTC_DRV: Init BTC, BTC Rev = 2, Backplane = 0, 
Port = 0.

The current active TCC also appears in this output. Slot 11 TCC is TCC B, while slot 7 is TCC A.

.252ML12#show ons equipment-agent status
EQA ---- phySlot: 12, eqptType: EQPT_L2SC, eqptID: 0x2403 ----
     curTCC: Tcc B
linkStatus:    Full    dbReq/Recv: 7  /  7   msgVerToEQM: 2
   socketFd:       0    pipeMsgAct:   No     hdrSizeToEQM: 28
  connTries:       0 connTimerFast:   No   hdrSizeFromEQM: 28
timingProv:   No
clock auto 1 

Circuit Removal and Creation

Removal of the cross-connect circuit creates these log entries:

May 27 17:40:48.459: %VIRTUAL_PA-6-PAREMOVED: 
POS interface [0] has been removed due to circuit deletion 
May 27 17:40:48.511: %CARDWARE-6-BTC_DRV: Init BTC, BTC Rev = 2, 
Backplane = 0, Port = 0.  

Port configuration is changed as you view it from ML.

.225ML12#show ons provisioning-agent m ports all
  ----- Backend Port (00) Data ----- 
  prov:  no   sts: xx   vt: xx   type: xxx   name: xxxxx 
  ----- Backend Port (01) Data -----   
  prov:  no   sts: xx   vt: xx   type: xxx   name: xxxxx 

Creation of an STS3c circuit updates the port information on ML. Circuit size also appears in POS 0 controller output.

.225ML12#show ons provisioning-agent m ports all
  ----- Backend Port (00) Data -----
  prov: yes   sts: 00   vt: 255   type: DOS   name:                                  
  ----- STS (00) Term Strip -----
  Admin State: IS        Direction: TX_RX_EQPT
  Type: 3 Sf: 1E-4 Sd: 1E-7 C2 tx/exp: 0x01 / 0x01
  PathTrace Format: 64Byte  Mode: OFF 
   expected: (not valid)
   send: valid: "\000\000\000\000" 
  ----- VT (255) Term Strip not provisioned ----- 
  ----- STS (00) Xc Strip -----   
  rate: 3  Admin: IS  
  Src  Port/STS: 0x09/0x00  STS Eqpt: 0x01
  Dest Port/STS: 0x06/0x00  UPSR STS Cont Dest: 0x00 
  Prev STS Stich Dest Port/STS: 0xFF/0x00 
  Next STS Stich Dest Port/STS: 0xFF/0x00 

  ----- Backend Port (01) Data -----  
  prov:  no   sts: xx   vt: xx   type: xxx   name: xxxxx 

These log entries appear:

May 27 17:47:08.711: %VIRTUAL_PA-6-PAPLUGGEDIN: 
POS interface [0] has been created due to circuit creation
May 27 17:47:08.715: %CARDWARE-6-BTC_DRV: Init BTC, BTC Rev = 2, 
Backplane = 0, Port = 0. 
May 27 17:47:08.915: %LINK-3-UPDOWN: 
Interface POS0, changed state to up 
May 27 17:47:09.927: %LINEPROTO-5-UPDOWN: 
Line protocol on Interface POS0, changed state to up

Loopbacks

Application of a facility loop to the active OC12 port on the .225 causes the .225 ML to report TPTFAIL alarm. This alarm also appears on the ML alarm lists.

Note: If you enable loopbacks on an active path, traffic loss occurs.

.225ML12#show ons alarm
Equipment Alarms
Active: None 
Port Alarms
  POS0  Active: TPTFAIL    
  POS1  Active: None
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None 
POS0 
Active Alarms : PAIS
Demoted Alarms: None 

POS1
Interface not provisioned

Note: When you use resilient packet ring (RPR) instead of the 1+1 OC-12 as in this test, shut down POS interfaces before you enable loopbacks. Such a loopback on RPR causes traffic loss, because the protection path does not reroute the traffic.

Date and Time Change

Incorrect date and time settings on the TCC create this entry in the log:

2d23h: %CARDWARE-5-CLOCK_ERR: cannot set time-of-day, 
(invalid IOS time set on TCC) 

When you change the date and time, this entry appears in the ML log.

2d23h: %CARDWARE-5-CLOCK_INFO: system clock, timezone, 
and summertime configured 

An automatic update occurs on the IOS system clock based on the clock from TCC. You can verify this update through the show clock command.

Note: You can use the service timestamps command to configure debug and log time stamps to use the new clock information.

One POS Interface Shut Down

When POS 0 interface on .225 ML is shut down, some alarms and conditions occur (see Figure 4).

Figure 4 – Alarms and Conditions that Occur when POS 0 Interface is Shut Down

f4_59863.gif

AIS-P occurs for both OC12 ports on .252. Then TPTFAIL occurs for ML on .252. On the return path, .225 reports Path Payload Defect Indication (PPDI, also called PDI-P), for both OC-12 ports, and RFI-P for the working OC-12 port.

On .225 ML, these alarms appear:

.225ML12#show ons alarm
Equipment Alarms
Active: RUNCFG-SAVENEED 
 
Port Alarms  
  POS0  Active: None  
  POS1  Active: None 
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None
 
POS0
Active Alarms : PRDI PPDI 
Demoted Alarms: None 

POS1 
Interface not provisioned 

These log entries appear on .225 also:

May 24 10:52:01.802: %LINK-5-CHANGED: Interface POS0, 
changed state to administratively down 
May 24 10:52:02.801: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS0, 
changed state to down 
May 24 10:52:04.021: %SONET-4-ALARM:  POS0: PRDI    
May 24 10:52:04.269: %SONET-4-ALARM:  POS0: PPDI  

On .252, these alarms occur:

.252ML12#show ons alarm
Equipment Alarms
Active: None 

Port Alarms
  POS0  Active: TPTFAIL 
  POS1  Active: None 
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None
 
POS0 
Active Alarms : PAIS
Demoted Alarms: None 

POS1
Interface not provisioned 

Similarly, logs entries on .252 indicate the reason for POS 0 down event is PAIS. This is consistent with the alarms or conditions that CTC reports.

May 24 10:51:48.969: %VIRTUAL_PA-6-UPDOWN: 
POS0 changed to down due to PAIS defect trigger changing state 
May 24 10:51:49.169: %LINK-3-UPDOWN: 
Interface POS0, changed state to down 
May 24 10:51:50.169: %LINEPROTO-5-UPDOWN: 
Line protocol on Interface POS0, changed state to down 
May 24 10:51:51.169: %SONET-4-ALARM:  POS0: PAIS 

You can confirm this fact through this output:

.252ML12#show contro pos 0 | inc Active
Active Alarms : PAIS
Active Defects: PAIS  

When you bring up the POS 0 interface, these log entries appear on .252 ML:

May 24 11:16:17.509: %VIRTUAL_PA-6-UPDOWN: 
 POS0 changed to up due to PAIS defect trigger changing state
 May 24 11:16:17.709: %LINK-3-UPDOWN:
 Interface POS0, changed state to up
 May 24 11:16:18.709: %LINEPROTO-5-UPDOWN:
 Line protocol on Interface POS0, changed state to up
 May 24 11:16:27.309: %SONET-4-ALARM: 
 POS0: PAIS  cleared  

These are the log entries on .225 ML:

May 24 11:16:30.607: %VIRTUAL_PA-6-UPDOWN: 
POS0 changed to up due to PPDI defect trigger changing state 
May 24 11:16:30.807: %LINK-3-UPDOWN: 
Interface POS0, changed state to up 
May 24 11:16:31.555: %SYS-5-CONFIG_I: 
Configured from console by vty0 (127.0.0.100) 
May 24 11:16:31.807: %LINEPROTO-5-UPDOWN: 
Line protocol on Interface POS0, changed state to up
May 24 11:16:40.175: %SONET-4-ALARM:  POS0: PRDI  cleared 
May 24 11:16:40.415: %SONET-4-ALARM:  POS0: PPDI  cleared 

Now traffic returns to normal.

POS CRC Mismatch

When CRC does not match on both POS ports of the same circuit (for instance, one side 16 bits, while the other side 32 bits), no alarms occur on TCC, nor on ML. Both POS ports are still up, but traffic does not flow. Here are a few symptoms:

  1. Both POS interface input error counters increment with 100% due to CRC. In this case, CRC changes to 16 bits on .225 ML while .252 ML still has the default 32 bits CRC. The POS0 interface on .252 ML displays a similar input and CRC error count.

    .225ML12#show int pos 0
    POS0 is up, line protocol is up  
      Hardware is Packet/Ethernet over Sonet, 
    address is 000f.2475.8c00 (bia 000f.2475.8c00) 
      MTU 1500 bytes, BW 622080 Kbit, DLY 100 usec,   
         reliability 149/255, txload 1/255, rxload 1/255   
    Encapsulation ONS15454-G1000, crc 16, loopback not set   
    Keepalive set (10 sec)   
    Scramble enabled    
    ARP type: ARPA, ARP Timeout 04:00:00   
    Last input 00:06:57, output never, output hang never   
    Last clearing of "show interface" counters 00:04:28   
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0   
    Queueing strategy: fifo   
    Output queue: 0/40 (size/max)   
    5 minute input rate 0 bits/sec, 0 packets/sec   
    5 minute output rate 0 bits/sec, 0 packets/sec   
       0 packets input, 11190 bytes  
       Received 0 broadcasts (0 IP multicast)   
       0 runts, 0 giants, 0 throttles             
                0 parity      138 input errors, 
       138 CRC, 0 frame, 0 overrun, 0 ignored    
       0 input packets with dribble condition detected   
       178 packets output, 15001 bytes, 0 underruns  
       0 output errors, 0 applique, 0 interface resets    
       0 babbles, 0 late collision, 0 deferred   
       0 lost carrier, 0 no carrier  
       0 output buffer failures, 0 output buffers swapped out  
       0 carrier transitions 
  2. POS controller input CRC error counts increment.

    .225ML12#show contro pos 0 | inc input
    8841 total input packets,  46840204 post-HDLC bytes 
    0 input short packets,  46840993 pre-HDLC bytes 
    0  input long packets , 3893  input runt packets 
    2165  input CRCerror packets , 0  input drop packets 
    0 input abort packets 
    0 input packets dropped by ucode 
  3. CDP neighbor across the optical path drops. Even though POS0 is up and CDP works, the neighbor across POS0 does not show up.

    225ML12#show cdp neighbor
    Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge    
                      S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone 
    Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID 
    7603a            Gig 0              170         R S I     Cat 6000  Gig 1/1 
    
    .225ML12#show cdp int | be POS0
    POS0 is up, line protocol is up 
      Encapsulation  
      Sending CDP packets every 60 seconds 
      Holdtime is 180 seconds 
    

POS Scrambling Mismatch

With PPP encapsulation, you can enable SPE scrambling (by default, SPE scrambling is disabled). In this example, .225ML POS0 has scramble enabled while .252ML POS0 has the default setting.

.225ML12#show int pos 0 | in Scramble
   Scramble enabled

Mismatch of scrambling changes the C2 value. If you enable scrambling, POS interfaces use a C2 value of 0x16. If you disable scrambling, POS interfaces use a C2 value of 0xCF. When you enable scrambling on .252 POS 0 port, here is the result (the .225 POS 0 configuration does not change):

.252ML12#show contr pos 0 | in C2
C2 (tx / rx)           : 0x16 / 0xCF

On the .252 node, PLM-P occurs against the active OC12 port in CTC, and then POS0 port. This triggers POS0 port to go down, which raises TPTFAIL alarm.

.252ML12#show ons alarm
Equipment Alarms 
Active: RUNCFG-SAVENEED  

Port Alarms 
  POS0  Active: TPTFAIL    
  POS1  Active: None  
  GigabitEthernet0  Active: None 
  GigabitEthernet1  Active: None 
POS0 
Active Alarms : PPLM  
Demoted Alarms: None 

POS1
Interface not provisioned 
 

On the .225 node, PDI-P occurs for both OC12 ports in CTC. This alarm is the result of POS0 down in .252. The same alarm (called Path Payload Defect Indication [PPDI] in IOS) occurs for POS0, which is because the interface receives the C2 value of 0xFC (more information on this follows later in the document).

.225ML12#show control pos 0 | inc C2
C2 (tx / rx)           : 0xCF / 0xFC

The PPDI alarm brings down the POS0 interface. The down POS0 interface then raises TPTFAIL.

.225ML12#show ons alarm
Equipment Alarms 
Active: RUNCFG-SAVENEED  
Port Alarms  
  POS0  Active: TPTFAIL    
  POS1  Active: None 
  GigabitEthernet0  Active: None 
  GigabitEthernet1  Active: None 

POS0 
Active Alarms : PPDI  
Demoted Alarms: None 

POS1 
Interface not provisioned 

POS C2 Byte Change

The default C2 value is 0x01 for LEX encapsulation (the default encapsulation for POS) and 0xCF for PPP/HDLC encapsulation. If you change this value inconsistently to any other value, the PLM-P and TPTFAIL alarms can occur, which affect service. Both POS ports on the same circuit can use the same C2 value. The exception is 0xFC. A value of 0xFC indicates a path Payload Defect. So even if the C2 values do match (0xFC/0xFC), PDI-P occurs.

You can change the POS C2 value with this command:

pos c2 flag <value in decimal>

You can represent the actual C2 values as shown here (they are in hexadecimal formats):

.225ML12#show contro pos 0 | inc C2
C2 (tx / rx)           : 0x16 / 0x16 

In this case, both C2 values match. Therefore, no alarm occurs.

Change Circuit State to OOS

When you change the OC-12 circuit to OOS, no alarms may occur immediately on TCC or on ML. The circuit state displays OOS on the circuit window in CTC. Log entries are inserted into ML:

.225ML12#show log …
May 27 14:22:15.114: %CARDWARE-6-CIRCUIT_STATE: 
Circuit state on POS 0 change from IS to OOS_AS 
May 27 14:22:15.114: %CARDWARE-6-BTC_DRV: 
Init BTC, BTC Rev = 2, Backplane = 0, Port = 0.  

The POS ports can change to the Up/Down state. As a result, TPTFAIL alarm occurs on both ends. Traffic does not flow, as you can expect.

Stuck PDI-P Alarm

Sometimes an alarm gets stuck, and does not clear automatically, even after the condition that caused the alarm clears. A PPDI (or PDI-P) example is shown here:

May 27 18:41:15.339: %CARDWARE-6-CIRCUIT_STATE: 
Circuit state on POS 0 change from IS to OOS_AS 
May 27 18:42:11.871: %LINEPROTO-5-UPDOWN: 
Line protocol on Interface POS0, changed state to down 
May 27 19:17:48.507: %SYS-5-CONFIG_I: 
Configured from console by vty2 (127.0.0.100) 
May 28 11:57:33.387: %CARDWARE-6-CIRCUIT_STATE: 
Circuit state on POS 0 change from OOS_AS to IS 
May 28 11:57:33.391: %CARDWARE-6-BTC_DRV: 
Init BTC, BTC Rev = 2, Backplane = 0, Port = 0.  
May 28 11:57:35.879: %VIRTUAL_PA-6-UPDOWN: 
POS0 changed to down due to PPDI defect trigger changing state 
May 28 11:57:36.079: %LINK-3-UPDOWN: 
Interface POS0, changed state to down 
May 28 11:57:36.279: %SONET-4-ALARM:  
POS0: PPDI

When a previous circuit state changes to OOS, .225 POS reports PPDI even after the circuit returns to the In-Service (IS) state. So POS0 interface stays down. CTC also reports PDI-P on .225 node. The PM counters of the OC12 interfaces on .225 show no errors, and indicate that the OC-12 path is clean.

This output reports PPDI as stuck:

.225ML12#show contro pos 0
Interface POS0 
Hardware is Packet/Ethernet over Sonet  
PATH 
  PAIS      = 0         PLOP      = 0         PRDI    = 0         PTIM = 0 
  PPLM      = 0         PUNEQ     = 0         PPDI    = 0        
  BER_SF_B3 = 0         BER_SD_B3 = 0         BIP(B3) = 0          REI = 0 
  NEWPTR    = 0         PSE       = 0         NSE     = 0        
Active Alarms : PPDI  
Demoted Alarms: None 
Active Defects: PPDI  
Alarms reportable to CLI: 
PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3 
Link state change defects: 
PAIS PLOP PTIM PUNEQ PRDI PPLM PPDI BER_SF_B3  

Link state change time   : 200 (msec)
 
DOS FPGA channel number  : 0 
Starting STS (0 based)   : 0 
VT ID (if any) (0 based) : 255 
Circuit size           : STS-3c 
RDI Mode               : 1 bit 
C2 (tx / rx)           : 0xCF / 0xFC 
Framing                : SONET

Recall from earlier in this document, the C2 value 0xFC causes POS to report PPDI.

Note: When .252 node is free of alarms and errors, and has the matching C2 values of 0xCF/0xCF for POS0, you must consider a stuck alarm problem. If you reset the POS0 interface on .225 node, the alarm clears, which includes the PDI-P reported in CTC. This anomaly shall be fixed in a later release.

May 28 14:34:16.967: %LINK-5-CHANGED: 
Interface POS0, changed state to administratively down 
May 28 14:34:18.675: %LINK-3-UPDOWN: 
Interface POS0, changed state to down 
May 28 14:34:18.939: %VIRTUAL_PA-6-UPDOWN: 
POS0 changed to up due to PPDI defect trigger changing state 
May 28 14:34:19.139: %LINK-3-UPDOWN: 
Interface POS0, changed state to up 
May 28 14:34:20.127: %SYS-5-CONFIG_I: 
Configured from console by vty2 (127.0.0.100) 
May 28 14:34:20.147: %LINEPROTO-5-UPDOWN: 
Line protocol on Interface POS0, changed state to up 
May 28 14:34:28.739: %SONET-4-ALARM:  
POS0: PPDI  cleared 

Now the C2 values match, and the node is alarm-free.

.225ML12#show control pos 0 
Interface POS0 
Hardware is Packet/Ethernet over Sonet  
PATH  
  PAIS      = 0         PLOP      = 0         PRDI    = 1         PTIM = 0 
  PPLM      = 0         PUNEQ     = 0         PPDI    = 0     
  BER_SF_B3 = 0         BER_SD_B3 = 0         BIP(B3) = 0          REI = 16 
  NEWPTR    = 0         PSE       = 0         NSE     = 0         
Active Alarms : None 
Demoted Alarms: None 
Active Defects: None 
Alarms reportable to CLI: 
PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3 
Link state change defects: 
PAIS PLOP PTIM PUNEQ PRDI PPLM PPDI BER_SF_B3  
Link state change time: 200 (msec) 

DOS FPGA channel number  : 0 
Starting STS (0 based)   : 0 
VT ID (if any) (0 based) : 255 
Circuit size           : STS-3c 
RDI Mode               : 1 bit 
C2 (tx / rx)           : 0xCF / 0xCF 
Framing                : SONET 

Note: Sometimes, one or more alarms can also be stuck on optical cards. You need to reset the active TCC in order to clear these stuck alarms. Consequently, the standby TCC becomes active, and the operation is a hitless one (i.e. there is no traffic impact), though you can lose the management traffic (CTC session, for example) for a few minutes.

Bridge-group Number Mismatch

This test uses the same 100 bridge-group on both ONS ML cards. However, the bridge-groups do not have to be the same, as long as POS 0 and GigE 0 are on the same ML, or in the same bridge-group. For example, a change to bridge-group 101 on .252 ML does not affect traffic.

.252ML12#show bridge ver
Total of 300 station blocks, 298 free 
Codes: P - permanent, S - self 

Maximum dynamic entries allowed: 1000 
Current dynamic entry count: 0 

Flood ports  
        
Maximum dynamic entries allowed: 1000 
Current dynamic entry count: 2 
BG Hash      Address      Action  Interface    VC    Age   RX count   TX count 
101 02/0   000b.45b0.484a forward  Gi0                  -     
101 BC/0   0009.b7f4.76ca forward  POS0 
                -     
Flood ports          
GigabitEthernet0     
POS0                 

A Partial List of Outstanding ML Bugs

Here is a partial list of bugs that apply to the configuration in this document:

Note: These bugs are documented as part of the release notes on cisco.com.

DDTS ID Status Release found Release fixed ********************Release*Notes********************
CSCeb56287 V 4.1 4.6 When you provision the state of an ML-series circuit from In-Service (IS) to Out-of-Service (OOS), and then back to IS, data traffic does not recover. In order to avoid this issue, before you change the state from IS, set the POS port to shutdown on the CLI. After you change the state back to IS from OOS, set the POS port to no shutdown.
CSCeb24757 V 4.1 4.6 If you disconnect a transmit fiber on an ML1000 port, only the adjacent port takes the link down. Ideally, both ports must identify that the link went down so that upper layer protocols can re-route the traffic to a different port. In order to work around this situation, issue shutdown and no shutdown to the port that has the disconnected or faulty transmit fiber.
CSCdy31775 V 4 4.6 No discard count includes packets that are discarded due to output queue congestion. This issue occurs under either of these conditions:
  • Traffic on ML-series cards between Ethernet and SONET ports, with oversubscription of available circuit bandwidth configured, which leads to output queue congestion.
  • Traffic from SONET to Ethernet, with oversubscription of the available Ethernet bandwidth.
CSCdz49700 C 4 - The ML-series cards always forward Dynamic Trunking Protocol (DTP) packets between connected devices. If DTP is enabled on connected devices (which can be the default setting), DTP might negotiate parameters, for example, ISL, that the ML-series cards do not support. The ML-series card counts all packets on a link negotiated to use ISL as multicast packets, and STP and CDP packets are bridged between connected devices that use ISL without being processed. In order to avoid this issue, disable DTP and ISL on connected devices. This functionality is as designed.
CSCdz68649 C 4 - Under certain conditions, the flow-control status can indicate that flow control is functioning, when flow-control does not work. Flow-control on the ML-series cards only functions when you configure a port-level policer. A port-level policer is a policer on the default and only class of an input policy-map. Flow-control also functions only to limit the source rate to the configured policer discard rate. Flow-control does not prevent packet discards due to output queue congestion. Therefore, if you do not have a port-level policer, or if output queue congestion occurs, policing does not function. However, policing can still mistakenly appear as enabled under these conditions. In order to avoid this issue, configure a port-level policer and prevent output queue congestion.
CSCdz69700 C 4 - If you issue a shutdown/no shutdown command sequence on an ML1000 port, the counters clear. This is a normal part of the startup process and this functionality will not change.
CSCea11742 V 4 4.6 When you provision a circuit between two ML POS ports as OOS, one of the ports can erroneously report TPTFAIL. This issue exists for both ML100T-12 and ML1000-2 cards. If this issue occurs, open a console window to each ML card and configure the POS port to shutdown.
CSCea20962 V 4 5 No warning appears when you apply OOS to ML drop ports on the circuit provisioning window.
CSCdy47284 C 4 - ML-100 FastEthernet MTU is not enforced. However, frames larger than 9050 bytes can be discarded and cause Rx and Tx errors.
Status Codes:
  • V – Verified (the fix is verified in the lab)
  • C – Closed (there will be no fix for the bug due to various reasons)
  • Release found: the software release that the bug was first reported
  • Release fixed: the software release that the bug was fixed

Troubleshoot and Isolate Faults

With the information presented so far, this section aims to build fault isolation cases. Based on the symptoms that the system reports, this section provides step-by-step tips to troubleshoot the problem. These case studies relate to some common symptoms associated with the ML card on ONS 15454.

Typically, you must follow these steps to troubleshoot an issue:

  • Collect general information and fault symptoms.

  • Analyze the information.

  • Isolate the problem.

  • Identify the problem.

  • Resolve the problem.

Some of theses steps are iterated multiple times.

General Information

Basic Information Collection

Gather information before you reload or reset the ML card due to an error. A manual reload discards potentially valuable information. Manual reloads reset all counters, and you lose any logs stored in memory. Cisco recommends that you issue the show tech-support command, and any other data collection commands to recover log information before you issue any troubleshooting commands on the router. If you reboot or reset the ML card, you can lose console/telnet access, and also the relevant information.

Console logs that lead up to the event can provide a picture of what led to the error or crash. When an error occurs, you must attempt to save any messages logged to the console or buffer. These last console messages could prove vital to discover the problem. Depending on the type of problem, not all messages are written to the SYSLOG server.

Use the show tech-support command to collect a wide variety of data. This command is often the best tool to get the state of the router, after the error at a given point in time.

Here is a basic list of the commands that the show tech-support command executes. What you capture varies, based on the IOS version, hardware, and the options you select.

show version
show running-config 
show stacks 
show interfaces 
show controllers 
show file systems 
dir nvram: 
show flash: all 
show process memory 
show process cpu 
show context 
show sdm internal all-regions 
show sdm ip-adjacency all 
show sdm ip-mcast all 
show sdm ip-prefix all 
show sdm l2-switching forwarding 
show sdm l2-switching interface-macs 
show sdm qos all 
show ons alarm defect 
show ons alarm failure 
show ons hwp defects 
show ons hwp reframe 
show ons hwp tci 
show ons hwp xcon 
show ons equipment-agent status 
show ons provisioning-agent message ports all 
show ons provisioning-agent message node-element 
test mda conn dump connections 
test mda ppe global reg dump 0 
test mda ppe global reg dump 1 Mempool statistics 
show region 
show buffers

In addition to these commands, capture other command outputs that have special relevance to the ML card as described in the previous sections of this document. For example, show log, show ons alarm and so on. From CTC, capture and export relevant information as described previously, for example, alarms, conditions, circuits, inventory, and PM counters.

Some Useful IOS Tools to Troubleshoot Issues

After you gather the required information, you need to decipher the information for errors. This task can be difficult with the output of a show-tech command. These are tools that can decipher the output of the show-tech command, and many other commands.

  • Output Interpreter Tool (registered customers only) : Paste the output from the show tech-support command into this tool. This tool will provide a quick summary of any problems found. This is a great tool that provides a quick summary of the more straightforward problems you encounter. This tool interprets a variety of input. You can use the Technology menu drop-down box to browse. However, the tool is not perfect, and still requires interpretation to validate the information.

  • Command Lookup Tool: Select any one of these reference guides to lookup a command and the syntax:

    • IOS Command reference

    • IOS Configuration guide

    • Catalyst Command reference

    • PIX firewall command reference

  • Error Message Decoder: This tool helps you to research and resolve error messages for Cisco IOS Software, Catalyst Switches Software, and Cisco Secure PIX Firewall Software. Paste the error messages from the log files, and ensure that you check the suggest related documents within results check box.

  • Bug Toolkit: Search for results based on one or more of these options:

    • IOS Version.

    • Features or Components.

    • Keywords.

    • Bug Severity (you can select a specific severity, or specify a range).

  • TAC Case Collection: You can interactively diagnose common problems that involve hardware, configuration, and performance issues with solutions that TAC engineers provide.

Note: Some tools are not 100% compatible for the ML card.

Case Studies

This section describes some of the common fault conditions, and possible steps you can take to isolate the conditions. Refer to Cisco ONS 15454 Troubleshooting Guide, Releases 4.1.x and 4.5 for detailed alarm information.

CARLOSS Alarm Reported on an ML Ethernet Port

Major (MJ), and Service-Affecting (SA), a Carrier Loss alarm on the ML-series Ethernet (traffic) card is the data equivalent of the "LOS (OC-N)" alarm. The Ethernet port has lost the link, and does not receive a valid signal.

A CARLOSS alarm occurs when the Ethernet port has been configured from IOS CLI as a no shutdown port, and one of these conditions is also met:

  • The cable is not properly connected to the near or far port.

  • Auto-negotiation fails.

  • The speed (for 10/100 ports only) is set incorrectly.

As seen in this test between 7603b and .252 node ML card, disable auto-negotiation to bring up the ports.

TPTFAIL Alarm Reported for POS

This is a major alarm (MJ), and is service affecting (SA). The TPT Layer Failure alarm indicates a break in the end-to-end POS link integrity feature of the ML-series POS cards. TPTFAIL indicates a far-end condition or incorrect configuration of the POS port.

The TPTFAIL alarm indicates a problem on either the SONET path, the remote POS port, or a misconfiguration of the POS port that prevents the complete end-to-end POS path from working.

If any SONET path alarms, for example, the "AIS-P", the "LOP-P", the "PDI-P", or the "UNEQ-P" exist on the circuit that the POS port uses, the affected port can report a TPTFAIL alarm.

If the far-end ML-series POS port is administratively disabled, the port inserts an "AIS-P" condition that the near-end port detects. The near-end port can report TPTFAIL in this event. The far-end POS port reports PRDI and PPDI. You can view all these alarms with the show ons alarm command. If the POS port is incorrectly configured at the IOS CLI level, the misconfiguration will cause the port to go down, and report TPTFAIL.

Complete these steps in order to clear the TPTFAIL (ML-Series) Alarm:

  1. If no SONET alarms occur against the POS port circuit, verify whether you have configured both POS ports properly.

  2. If only the "PLM-P" alarm occurs against the POS port circuit, verify whether you have configured both POS ports properly.

  3. If only the "PDI-P" condition occurs against the POS port circuit, and the circuit is terminated by a G-series card, verify whether a "CARLOSS (G-Series Ethernet)" alarm occurs against the G-series card. If so, complete the "Clear the CARLOSS (G-Series Ethernet) Alarm" procedure.

  4. If the "AIS-P" alarm, the "LOP-P" alarm, or the "UNEQ-P" alarm is present, troubleshoot the SONET path (the path between the two POS interfaces over the same circuit) to clear those alarms.

Gigabit Ethernet Interface Stays Down

See CARLOSS Alarm Reported on an ML Ethernet Port.

POS Interface Reports CRC Errors

This issue is typically due to CRC mismatch on POS configurations.

POS Reports PPDI

PDI-P is a set of application-specific codes contained in the STS path overhead (POH) that the ONS node generates. The alarm indicates to downstream equipment that there is a defect in one or more of the directly mapped payloads contained in that STS synchronous payload envelope

A PDI-P condition on the port of an OC-N card that supports an ML-series card circuit can result from the end-to-end Ethernet link integrity feature of the ML-series card. If the issue is due to link integrity, the "TPTFAIL (G-Series Ethernet)" alarm, or alarm reported against one or both POS ports terminating the circuit also occurs. If TPTFAIL occurs against one or both of POS ports, troubleshoot the alarm that accompanies TPTFAIL, to clear the PDI-P condition. The PDI-P alarm can also be a symptom of a stuck alarm.

Here is an example of alarms that occur due to POS0 administratively down on .225:

.225 POS 0 (shut) .252 POS 0
PPDI, PRDI PAIS, TPTFAIL

In this example, PAIS indicates that the root of the problem is the .225 node. If you clear PAIS, the TPTFAIL, PPDI, and PRDI also clear.

POS Reports PRDI

PRDI indicates that the problem is in the far end. This issue can occur because the far-end receives the AIS alarm. See POS Reports PPDI for more information.

POS Reports PAIS

The AIS Path condition means that this node detects AIS in the incoming path.

Generally, any AIS is a special SONET signal that tells the receiver node that the sender node has no valid signal available to send. AIS is not an error. The receiver node raises the fault condition AIS on each input where the node sees the signal AIS instead of a real signal. In most cases when this condition occurs, an upstream node raises an alarm to indicate a signal failure; all downstream nodes only raise some type of AIS. This condition clears when you resolve the problem on the upstream node.

POS Reports PPLM

This issue is Critical (CR) and Service-Affecting (SA)

A Path Payload Label Mismatch alarm on a node indicates that the incoming signal does not match the locally provisioned label. The condition occurs due to an invalid C2 byte value in the SONET path overhead. Scrambling and encapsulation can change C2 values.

POS Interface Stays Down

A variety of alarms can bring down the POS interface. By default, these alarms cause POS link to go down: PAIS, PLOP, PTIM, PUNEQ, PRDI, PPLM, PPDI, BER_SF_B3. In order to modify the list, use the pos trigger defects interface command. When POS interface goes up or down, the cause is logged (show log). You can retrieve all active alarms or defects with the show ons alarm command. Troubleshoot the cause to bring up the POS interface. When the POS interface goes down, TPTFAIL alarm occurs.

When you connect to other vendors POS interfaces, ensure that these items match on both ends:

  1. Scrambling

  2. C2 value

  3. CRC

POS Interface Reports Input Errors

Input errors that accumulate on a POS interface (show interface POS and CTC PM counters) indicate that the inbound packets are malformed. A variety of causes can lead to input error packets.

Troubleshoot alarms if they exist.

If CRC errors increment along input errors, CRC errors can be the cause of input errors. Troubleshoot CRC configurations.

Verify POS interface configurations.

Troubleshoot the path components between the two POS ports. If input errors increment without a corresponding increment in any other component errors, consider a hardware issue. Before hardware replacement, perform these steps on both sides of the circuit (one at a time) to see if the problem persists:

  • TCC side switch

  • XC side switch

  • Protection switch on the SONET ports, if protection exists

  • ML card soft reset

  • ML card reseat

CDP Neighbor Does Not Show Up

Verify whether you have enabled CDP on both interfaces.

Troubleshoot alarms and interface errors if they exist.

No End-to-end Traffic Flow

Verify configurations on the two end devices.

Troubleshoot alarms and errors if they exist.

Appendix: Basic Test Configuration and Command Information

This section captures the basic configuration information for all the devices in this test, which is used as a baseline to troubleshoot issues.

7603a

7603a#show run
Building configuration...

Current configuration : 3136 bytes
!
version 12.1
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname 7603a
!
!
ip subnet-zero
!
!
!
mls flow ip destination
mls flow ipx destination
spanning-tree extend system-id
!
redundancy
 mode rpr-plus
 main-cpu
  auto-sync running-config
  auto-sync standard
!
!
!
interface GigabitEthernet1/1
 ip address 10.0.0.1 255.0.0.0
!
router ospf 1
 log-adjacency-changes
 network 10.0.0.1 0.0.0.0 area 0
!
ip classless
no ip http server
!
!
!
!
line con 0
line vty 0 4
!
end

7603a#show ip int bri
Interface              IP-Address     OK? Method Status                 Protocol
Vlan1                  unassigned     YES unset  administratively down  down

GigabitEthernet1/1     10.0.0.1       YES manual up                     up


7603a#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default, U - per-user static route, o - ODR
       P - periodic downloaded static route

Gateway of last resort is not set

7603a#show int gigabitEthernet 1/1
GigabitEthernet1/1 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0009.b7f4.76ca (bia 0009.b7f4.76ca)
  Internet address is 10.0.0.1/8
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex mode, link type is autonegotiation, media type is SX
  output flow-control is unsupported, input flow-control is unsupported, 1000Mb/s
  Clock mode is auto
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:45, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  L2 Switched: ucast: 5482 pkt, 516472 bytes - mcast: 1 pkt, 64 bytes
  L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 0 pkt, 0 bytes
     5145 packets input, 405866 bytes, 0 no buffer
     Received 5107 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     332 packets output, 111641 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

7603a#show ip ospf neig

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.0.0.2          1   FULL/DR         00:00:38    10.0.0.2        GigabitEtherne
t1/1

7603b

7603b#show run
Building configuration...

Current configuration : 1102 bytes
!
version 12.1
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname 7603b
!
enable password cisco
!
ip subnet-zero
!
!
!
mls flow ip destination
mls flow ipx destination
spanning-tree extend system-id
!
redundancy
 mode rpr-plus
 main-cpu
  auto-sync running-config
  auto-sync standard
!
!
!
interface GigabitEthernet1/1
 ip address 10.0.0.2 255.0.0.0
 speed nonegotiate
!
router ospf 1
 log-adjacency-changes
 network 10.0.0.2 0.0.0.0 area 0
!
ip classless
no ip http server
!
!
!
!
line con 0
line vty 0 4
 no login
!
end

Note that if GigE link does not come up, auto-negotiation may not be working. 
Auto-negotiation can be turned off to force the link to come up. 
Ensure both sides of the link are matching.

7603b#show ip int bri
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  unassigned      YES NVRAM  administratively down down

GigabitEthernet1/1     10.0.0.2        YES manual up                    up

7603b#show int gig 1/1
GigabitEthernet1/1 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 000b.45b0.484a (bia 000b.45b0.484a)
  Internet address is 10.0.0.2/8
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex mode, link type is force-up, media type is SX
  output flow-control is unsupported, input flow-control is unsupported, 1000Mb/s
  Clock mode is auto
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:04, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  L2 Switched: ucast: 5695 pkt, 534143 bytes - mcast: 3 pkt, 192 bytes
  L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 0 pkt, 0 bytes
     5319 packets input, 395772 bytes, 0 no buffer
     Received 5172 broadcasts, 4 runts, 0 giants, 0 throttles
     4 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     413 packets output, 139651 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

7603b#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default, U - per-user static route, o - ODR
       P - periodic downloaded static route

Gateway of last resort is not set

C    10.0.0.0/8 is directly connected, GigabitEthernet1/1

7603b#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

.225ML

.225ML12#show run
Building configuration...
Current configuration : 580 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname .225ML12
!
logging buffered 4096 debugging 
enable password cisco
!
ip subnet-zero
no ip routing
no ip domain-lookup
!
!
bridge 100 protocol ieee
!
!
interface GigabitEthernet0
 no ip address
 no ip route-cache
 bridge-group 100
!
interface GigabitEthernet1
 no ip address
 no ip route-cache
 shutdown
!
interface POS0
 no ip address
 no ip route-cache
 crc 32
 bridge-group 100
!
ip classless
no ip http server
!
!
!
!
line con 0
line vty 0 4
 exec-timeout 0 0
 no login
!
end

.225ML12#show ip int bri
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet0       unassigned      YES unset  up                    up      
GigabitEthernet1       unassigned      YES unset  administratively down down    
POS0                   unassigned      YES unset  up                    up      

.225ML12#show int gig 0
GigabitEthernet0 is up, line protocol is up 
  Hardware is xpif_port, address is 000f.2475.8c04 (bia 000f.2475.8c04)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, 1000BaseSX, Auto-negotiation
  output flow-control is off, input flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:53, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     336 packets input, 111810 bytes
     Received 1 broadcasts (0 IP multicast)
     1 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 244 multicast
     0 input packets with dribble condition detected
     5369 packets output, 422097 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

.225ML12#show int pos 0
POS0 is up, line protocol is up 
  Hardware is Packet/Ethernet over Sonet, address is 000f.2475.8c00 
(bia 000f.2475.8c00)
  MTU 1500 bytes, BW 622080 Kbit, DLY 100 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ONS15454-G1000, crc 32, loopback not set
  Keepalive set (10 sec)
  Scramble enabled 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:32, output never, output hang never
  Last clearing of "show interface" counters 02:16:40
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     152 packets input, 26266640 bytes
     Received 0 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     4250 packets output, 351305 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

.225ML12#show ons alarm
Equipment Alarms
Active: None
Port Alarms
  POS0  Active: None
  POS1  Active: None
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None

POS0
Active Alarms : None
Demoted Alarms: None

POS1
Interface not provisioned

This command shows all the defects that can be reported to CLI and TCC (via CTC).

.225ML12#show ons alarm defect
Equipment Defects
Active: None
Reportable to TCC/CLI: CONTBUS-IO-A CONTBUS-IO-B CTNEQPT-PBWORK
 CTNEQPT-PBPROT EQPT RUNCFG-SAVENEED ERROR-CONFIG 
Port Defects
  POS0 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  POS1 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  GigabitEthernet0 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  GigabitEthernet1 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 

POS0
Active Defects: None
Alarms reportable to CLI: PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3

POS1
Interface not provisioned

This command shows all the active alarms.

.225ML12#show ons alarm failure
Equipment Alarms
Active: None
Port Alarms
  POS0  Active: None
  POS1  Active: None
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None
POS0
Active Alarms : None
Demoted Alarms: None
POS1
Interface not provisioned

.225ML12#show control pos 0
Interface POS0
Hardware is Packet/Ethernet over Sonet 
PATH
  PAIS      = 0         PLOP      = 0         PRDI    = 0         PTIM = 0
  PPLM      = 0         PUNEQ     = 0         PPDI    = 0       
  BER_SF_B3 = 0         BER_SD_B3 = 0         BIP(B3) = 0          REI = 0
  NEWPTR    = 0         PSE       = 0         NSE     = 0       
Active Alarms : None
Demoted Alarms: None
Active Defects: None
Alarms reportable to CLI: PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3
Link state change defects: PAIS PLOP PTIM PUNEQ PRDI PPLM PPDI BER_SF_B3 
Link state change time   : 200 (msec)

DOS FPGA channel number  : 0
Starting STS (0 based)   : 0
VT ID (if any) (0 based) : 255
Circuit size           : STS-12c
RDI Mode               : 1 bit
C2 (tx / rx)           : 0x01 / 0x01
Framing                : SONET

Path Trace
 Mode            : off          
 Transmit String : 
 Expected String : 
 Received String : 
 Buffer          : Unstable
 Remote hostname :                         
 Remote interface:               
 Remote IP addr  :                 
B3 BER thresholds: 
SFBER = 1e-4,   SDBER = 1e-7

231 total input packets,  26294392 post-HDLC bytes
0 input short packets,  26294465 pre-HDLC bytes
0  input long packets , 0  input runt packets
1  input CRCerror packets , 0  input drop packets
0 input abort packets
0 input packets dropped by ucode

6392  total output packets, 527660  output pre-HDLC bytes
527812 output post-HDLC bytes

Carrier delay is 200 msec

.225ML12#show cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
.252ML12         POS0               148           T       ONS-ML1000POS0
7603a            Gig 0              121         R S I     Cat 6000  Gig 1/1

The following command  shows the detail bridge table. Note that 000b.45b0.484a
 is the address of Gig0 on 7603b.

.225ML12#show bridge ver
Total of 300 station blocks, 298 free
Codes: P - permanent, S - self
Maximum dynamic entries allowed: 1000
Current dynamic entry count: 2

BG Hash      Address      Action  Interface      VC    Age   RX count   TX count
100 02/0   000b.45b0.484a forward  POS0                 -    
100 BC/0   0009.b7f4.76ca forward  Gi0                  -    

Flood ports         
GigabitEthernet0    
POS0                

This command shows the same type of info as the above.

.225ML12#show sdm l2-switching forwarding bridge-group 100
MAC-Address   B-Group l3_int punt_da Out-int SPR-NodeId CAM-ADDR  STATE
-----------   ------- ------ ------- ------- ---------- --------  -----
0009B7F476CA  100     0      0       Gi0     ***        11        Used   
000B45B0484A  100     0      0       PO0     ***        12        Used   

.225ML12#show interface summary
 *: interface is up
 IHQ: pkts in input hold queue     IQD: pkts dropped from input queue
 OHQ: pkts in output hold queue    OQD: pkts dropped from output queue
 RXBS: rx rate (bits/sec)          RXPS: rx rate (pkts/sec)
 TXBS: tx rate (bits/sec)          TXPS: tx rate (pkts/sec)
 TRTL: throttle count

  Interface           IHQ   IQD  OHQ   OQD  RXBS RXPS  TXBS TXPS TRTL
---------------------------------------------------------------------
* GigabitEthernet0      0     0    0     0     0    0     0    0    0
  GigabitEthernet1      0     0    0     0     0    0     0    0    0
* POS0                  0     0    0     0     0    0     0    0    0

NOTE:No separate counters are maintained for subinterfaces
     Hence Details of subinterface are not shown

.225ML12#show ons equipment-agent status 
EQA ---- phySlot: 12, eqptType: EQPT_L2SC, eqptID: 0x2403 ----
    curTCC: Tcc B
linkStatus:    Full    dbReq/Recv: 1  /  4   msgVerToEQM: 2
  socketFd:       0    pipeMsgAct:   No     hdrSizeToEQM: 28
 connTries:       0 connTimerFast:   No   hdrSizeFromEQM: 28
timingProv:   No
clock auto 1

.225ML12#show ons provisioning-agent message ports all
  ----- Backend Port (00) Data -----
  prov: yes   sts: 00   vt: 255   type: DOS   name:                                 
  
  ----- STS (00) Term Strip -----
  Admin State: IS        Direction: TX_RX_EQPT
  Type: 12 Sf: 1E-4 Sd: 1E-7 C2 tx/exp: 0x01 / 0x01
  PathTrace Format: 64Byte  Mode: OFF       
   expected: (not valid)
   send: valid: "\000\000\000\000"

  ----- VT (255) Term Strip not provisioned -----

  ----- STS (00) Xc Strip -----
  rate: 12  Admin: IS     
  Src  Port/STS: 0x09/0x00  STS Eqpt: 0x01
  Dest Port/STS: 0x06/0x00  UPSR STS Cont Dest: 0x00
  Prev STS Stich Dest Port/STS: 0xFF/0x00
  Next STS Stich Dest Port/STS: 0xFF/0x00

  ----- Backend Port (01) Data -----
  prov:  no   sts: xx   vt: xx   type: xxx   name: xxxxx

The following command retrieves the ONS provisioning information
 that is done via CTC.

.225ML12#show ons provisioning-agent message node-element 
  ----- NE Data -----
  Node Name: R27-15454c
  MAC Addr : 00 10 CF D2 70 92
  IP Addr      : 10.89.244.225
  Sub Net Mask : 255.255.255.192
  Dflt Router  : 10.89.244.193
  Lan IP Addr  : 10.89.244.225
  Lan Sub Mask : 255.255.255.192
  Day Savings  : 0x01
  Min from UTC : 480
  Node ID      : 0xFF
  Sync Msg Ver : 0x01
  Sync Msg Res Delta : -1
  Sync Msg Res Quality : 0x06
  XConA Eqpt ID : 0x00000201
  XConB Eqpt ID : 0x00000201   OSPF Node ID  : 0xCFD27092
  SDH Mode      : SONET

.252ML12

The auto negotiation was turned off on Gig0 (see later).

.252ML12#show run
Building configuration...
Current configuration : 643 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname .252ML12
!
logging buffered 4096 debugging
enable password cisco
!
ip subnet-zero
no ip routing
no ip domain-lookup
!
!
bridge 100 protocol ieee
!
!
interface GigabitEthernet0
 no ip address
 no ip route-cache
 no speed
 no negotiation auto
 bridge-group 100
!
interface GigabitEthernet1
 no ip address
 no ip route-cache
 shutdown
!
interface POS0
 no ip address
 no ip route-cache
 crc 32
 bridge-group 100
!
ip classless
no ip http server
!
!
!
!
line con 0
line vty 0 4
 exec-timeout 0 0
 no login
!
end

.252ML12#show ip int brie
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet0       unassigned      YES manual up                    up      
GigabitEthernet1       unassigned      YES NVRAM  administratively down down    
POS0                   unassigned      YES unset  up                    up      

The Gig0 interface showed carrier loss until it was forced up by turning off
 auto negotiation.

.252ML12#show int gig 0
GigabitEthernet0 is up, line protocol is up 
  Hardware is xpif_port, address is 000f.2475.8c4c (bia 000f.2475.8c4c)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, 1000BaseSX, Force link-up
  output flow-control is off, input flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:06, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     391 packets input, 125375 bytes
     Received 1 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 282 multicast
     0 input packets with dribble condition detected
     8489 packets output, 637084 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

.252ML12#show int pos 0
POS0 is up, line protocol is up 
  Hardware is Packet/Ethernet over Sonet, address is 000f.2475.8c48
 (bia 000f.2475.8c48)
  MTU 1500 bytes, BW 622080 Kbit, DLY 100 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ONS15454-G1000, crc 32, loopback not set
  Keepalive set (10 sec)
  Scramble enabled 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 03:58:02
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     7396 packets input, 608413 bytes
     Received 0 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     267 packets output, 96676 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

.252ML12#show ons alarm
Equipment Alarms
Active: None
Port Alarms
  POS0  Active: None
  POS1  Active: None
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None

POS0
Active Alarms : None
Demoted Alarms: None

POS1
Interface not provisioned

.252ML12#show ons alarm defect
Equipment Defects
Active: None
Reportable to TCC/CLI: CONTBUS-IO-A CONTBUS-IO-B CTNEQPT-PBWORK
 CTNEQPT-PBPROT EQPT RUNCFG-SAVENEED ERROR-CONFIG 
Port Defects
  POS0 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  POS1 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  GigabitEthernet0 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 
  GigabitEthernet1 
  Active: None
  Reportable to TCC: CARLOSS TPTFAIL 

POS0
Active Defects: None
Alarms reportable to CLI: PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3

POS1
Interface not provisioned

.252ML12#show ons alarm failure
Equipment Alarms
Active: None
Port Alarms
  POS0  Active: None
  POS1  Active: None
  GigabitEthernet0  Active: None
  GigabitEthernet1  Active: None

POS0
Active Alarms : None
Demoted Alarms: None

POS1
Interface not provisioned

.252ML12#show contro pos 0
Interface POS0
Hardware is Packet/Ethernet over Sonet 
PATH
  PAIS      = 0         PLOP      = 0         PRDI    = 0         PTIM = 0
  PPLM      = 0         PUNEQ     = 0         PPDI    = 0       
  BER_SF_B3 = 0         BER_SD_B3 = 0         BIP(B3) = 0          REI = 0
  NEWPTR    = 0         PSE       = 0         NSE     = 0       
Active Alarms : None
Demoted Alarms: None
Active Defects: None
Alarms reportable to CLI: PAIS PRDI PLOP PUNEQ PPLM PTIM PPDI BER_SF_B3 BER_SD_B3
Link state change defects: PAIS PLOP PTIM PUNEQ PRDI PPLM PPDI BER_SF_B3 
Link state change time   : 200 (msec)

DOS FPGA channel number  : 0
Starting STS (0 based)   : 0
VT ID (if any) (0 based) : 255
Circuit size           : STS-12c
RDI Mode               : 1 bit
C2 (tx / rx)           : 0x01 / 0x01
Framing                : SONET

Path Trace
 Mode            : off          
 Transmit String : 
 Expected String : 
 Received String : 
 Buffer          : Unstable
 Remote hostname :                         
 Remote interface:               
 Remote IP addr  :                 

B3 BER thresholds: 
SFBER = 1e-4,   SDBER = 1e-7

7425 total input packets,  610493 post-HDLC bytes
0 input short packets,  610501 pre-HDLC bytes
0  input long packets , 0  input runt packets
1  input CRCerror packets , 0  input drop packets
0 input abort packets
0 input packets dropped by ucode

268  total output packets, 97061  output pre-HDLC bytes
97061 output post-HDLC bytes

Carrier delay is 200 msec

.252ML12#show cdp neigh
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
.225ML12         POS0               168           T       ONS-ML1000POS0
7603b            Gig 0              158         R S I     Cat 6000  Gig 1/1

.252ML12#show bridge verbose
Total of 300 station blocks, 300 free
Codes: P - permanent, S - self

Total of 300 station blocks, 298 free Codes: P - permanent, S – self

Maximum dynamic entries allowed: 1000 Current dynamic entry count: 2 
BG Hash       Address        Action  Interface    VC    Age   RX count   TX count 
100 02/0      000b.45b0.484a forward  Gi0             -     
100 BC/0      0009.b7f4.76ca forward  POS0            -      
Flood ports   GigabitEthernet0        POS0

.252ML12#show sdm l2-switching forwarding bridge-group 100
MAC-Address   B-Group l3_int punt_da Out-int SPR-NodeId CAM-ADDR  STATE
-----------   ------- ------ ------- ------- ---------- --------  -----
000B45B0484A  100     0      0       Gi0     ***        11        Used   
0009B7F476CA  100     0      0       PO0     ***        16        Used   

.252ML12#show int summ
 *: interface is up
 IHQ: pkts in input hold queue     IQD: pkts dropped from input queue
 OHQ: pkts in output hold queue    OQD: pkts dropped from output queue
 RXBS: rx rate (bits/sec)          RXPS: rx rate (pkts/sec)
 TXBS: tx rate (bits/sec)          TXPS: tx rate (pkts/sec)
 TRTL: throttle count
  Interface           IHQ   IQD  OHQ   OQD  RXBS RXPS  TXBS TXPS TRTL
---------------------------------------------------------------------
* GigabitEthernet0      0     0    0     0     0    0     0    0    0
  GigabitEthernet1      0     0    0     0     0    0     0    0    0
* POS0                  0     0    0     0     0    0     0    0    0
NOTE:No separate counters are maintained for subinterfaces
     Hence Details of subinterface are not shown

.252ML12#show ons equipment-agent status
EQA ---- phySlot: 12, eqptType: EQPT_L2SC, eqptID: 0x2403 ----
    curTCC: Tcc A
linkStatus:    Full    dbReq/Recv: 1  /  5   msgVerToEQM: 2
  socketFd:       0    pipeMsgAct:   No     hdrSizeToEQM: 28
 connTries:       0 connTimerFast:   No   hdrSizeFromEQM: 28
timingProv:   No
clock auto 1

.252ML12#show ons provisioning-agent message ports all
  ----- Backend Port (00) Data -----
  prov: yes   sts: 00   vt: 255   type: DOS   name:                                 
  
  ----- STS (00) Term Strip -----
  Admin State: IS        Direction: TX_RX_EQPT
  Type: 12 Sf: 1E-4 Sd: 1E-7 C2 tx/exp: 0x01 / 0x01
  PathTrace Format: 64Byte  Mode: OFF       
   expected: (not valid)
   send: valid: "\000\000\000\000"

  ----- VT (255) Term Strip not provisioned -----

  ----- STS (00) Xc Strip -----
  rate: 12  Admin: IS     
  Src  Port/STS: 0x09/0x00  STS Eqpt: 0x01
  Dest Port/STS: 0x06/0x00  UPSR STS Cont Dest: 0x00
  Prev STS Stich Dest Port/STS: 0xFF/0x00
  Next STS Stich Dest Port/STS: 0xFF/0x00

  ----- Backend Port (01) Data -----
  prov:  no   sts: xx   vt: xx   type: xxx   name: xxxxx

.252ML12#show ons provisioning-agent message node-element 
  ----- NE Data -----
  Node Name: r26-15454a
  MAC Addr : 00 10 CF D2 40 52
  IP Addr      : 10.89.244.252
  Sub Net Mask : 255.255.255.192
  Dflt Router  : 10.89.244.193
  Lan IP Addr  : 10.89.244.252
  Lan Sub Mask : 255.255.255.192
  Day Savings  : 0x01
  Min from UTC : 480
  Node ID      : 0xFF
  Sync Msg Ver : 0x01
  Sync Msg Res Delta : 0
  Sync Msg Res Quality : 0x00
  XConA Eqpt ID : 0x00000201
  XConB Eqpt ID : 0x00000201
  OSPF Node ID  : 0xCFD24052
  SDH Mode      : SONET

Related Information

Updated: Nov 14, 2005
Document ID: 59863