Guest

Cisco ONS 15800 Series DWDM Platforms

CMP On-Field Troubleshooting Guide

Cisco - CMP On-Field Troubleshooting Guide

Document ID: 18925

Updated: Dec 02, 2005

   Print

Introduction

This document describes how to collect Control and Monitoring Processor (CMP) information to help Cisco Technical Assistance Center (TAC) troubleshoot on-field failed CMPs.

Prerequisites

Requirements

There are no specific prerequisites for this document.

Components Used

The information in this document is based on the software and hardware versions below.

  • A laptop PC with Microsoft Windows 95, Windows 98 or Windows 2000. This procedure requires access to the CMP through the LAN port and the PPP serial connection. If the CMP is running the TL1 Agent version 1.1.2, then, to get access to the CMP through the serial connection, you need a Windows 95 PC. Using Windows 98 and Windows 2000 does not work.

  • An Ethernet British Naval Connector (BNC) coaxial cable to connect the PC to the CMP local Ethernet port.

  • A hub to connect the CMP to the PC or to the LAN.

  • A serial cable connection to connect to the CMP serial local port.

  • A ping application on the PC.

  • A Telnet application on the PC (verify that your Telnet application enables you to log your session and all your commands in a log file).

    Note: Telnet applications from Windows 2000 do not have this ability. If you are running Windows 2000 on your PC, then you must obtain a Telnet application from a Windows 98 or Windows 95 machine.

  • Cisco Photonics Local Terminal (CPLT) or Cisco Photonics Toolkit (CPTK) installed on the PC (compatible with the agent version you are running).

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Background

This process applies to any CMP failure, however, depending on the type of failure, not all steps may be applicable. The three main phases are:

  1. Site inventory, in order to collect information on the site hardware and software configuration.

  2. CMP visual inspection, in order to take an assesment of the site at the arrival time.

  3. CMP software inspection, aimed at investigating possible software blocks or failures.

One of these activities requires testing the CMP from the Network Element (NE) local LAN connection. This involves opening the LAN connector and inserting coaxial cable to connect the local PC (QA or QB connector).

Note: Only conduct this operation for troubleshooting purposes, and do it as quickly as possible to minimize the impact on the local NE LAN.

Note: The NE under test in this document is not visible from the Element Manager application during the test.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

CMP Troubleshooting

This section explains how to perform CMP troubleshooting. There are several steps starting with site inventory and ping tests, and finishing with software inspection of the board CMP using a Telnet session and CPLT or CPTK. Depending on the type of problem, some steps are not performed.

Site Inventory

Site inventory is an activity aimed at collecting information regarding the site hardware and software configuration. Information that should be collected during this step includes:

General Site Information

  • Site Type: Terminal Site, Optical Line Amplification (OLA), Optical Add/Drop Multiplexing (OADM), Regenerating site.

  • Number of sub-racks managed by the CMP.

  • Number and type of boards for sub-rack.

  • Firmware version of all the boards present in the site.

CMP Card Details

The following are details about the CMP card:

  • Serial Number

  • Part Number

  • Q3/TL1 Agent version

  • Switch Control Card (SCC) master version

CMP Visual Inspection

This section includes the visual analysis done on the CMP and on the site as well. During this phase, you should collect the following information:

  • Number displayed on the CMP LED display.

  • Check whether the number is blinking or fixed, or if the CMP is continuously rebooting.

  • Check whether alarms are present on the site boards and report where and with which severity.

  • Check whether the CMP alarm LED in the front panel is red.

CMP Software Inspection

This phase involves getting information on basic and advanced CMP functionality.

Note: Depending on the CMP failure condition, some of the steps during this phase are not applicable. If this is the case, please go on to the next step.

  1. Connect the Laptop to the Ethernet local port and perform the following test:

    • Ping the local CMP with default size packet.

    • Ping the local CMP with 10.000 size packet.

    • Telnet to the local CMP and get access with login and password; then exit.

    • Ping any remote CMP with default size packet.

    • Ping any remote CMP with 10.000 size packet.

    • Telnet to any remote CMP and get access with login and password; then exit.

    Note: After the execution of this test, remember to disconnect the coaxial cable and put the LAN terminator back in place on the LAN port (QA or QB connectors).

  2. Connect the Laptop to the CMP serial port and perform the following test:

    • Ping the local CMP with default size packet.

    • Ping the local CMP with 10.000 size packet.

    • Telnet to the local CMP and get access with login and password; then exit.

    • Ping any remote CMP with default size packet.

    • Ping any remote CMP with 10.000 size packet.

    • Telnet to any remote CMP and get access with login and password; then exit.

  3. Telnet to the CMP using either the Ethernet or the serial port. After login remember to activate the log to file option of your Telnet application.

    General Information

    • Run the ls -il command.

    • Change the directory using cd 9.0.1/ and perform ls -il again. Directory 9.0.1 represents the DOS partition of the CMP flash disk.

    • Run the netstat command to get information about the network configuration.

    • Run the PSOS ps command to get information about the processes running on the CMP.

    • Run the PSOS vinfo command to get information about the volume and its occupancy.

    Q3 Agent Specific Information (applicable only if the CMP is running a Q3 agent application)

    Get information about the following files, such as the file size, or the content of the file, or the download of the file itself for further analysis.

    • param - IP configuration of CMP.

    • pwd.fil - Security password to access the CMP.

    • netosi.cfg - Open Systems Interconnection (OSI) configuration and network identifier (Only Agent Q3).

    • moPers.dat - The general persistency of Agent Q3.

    • moPers.idx - The index file of Agent Q3 persistency.

    • genPers.dat - The generic persistency of Agent Q3.

    • genPers.idx - The index file of Agent Q3 persistency.

    • cfg.fil - The board slot configuration of the site.

    • bpool.cfg - The memory buffer pool for Agent Q3 only.

    TL1 Agent Specific Information (applicable only if the CMP is running a TL1 agent application)

    Get information about the following files. Information could include either the file size, the content of the file, or the file download for further analysis

    • param

    • pwd.fil

    • cfg.fil

    • wd.fil

    • BOARDS.DAT - The list of the all board's state (only for the TL1 Agent).

    • BOARDS.DAT - The list of the all board's state (only for the TL1 Agent).

  4. Telnet to the CMP using either the Ethernet or the serial port. You need to Telnet on port 5678 (telnet <IP address> 5678).

    After login, remember to activate the log to file option of your Telnet application.

    This step is aimed at activating trace information coming from the CMP processes. The traces are collected on the Telnet log file.

    After login, at the prompt command type help. The firmware and software versions of the software running in the CMP are shown, along with a list of possible commands

  5. . Type the option allon. It is now possible to collect the trace information.

  6. Connect to the CMP with the CPLT (the connection can be established using the same port as for step 3).

    • Get a snapshot of the local node (that is the boards visible from CPLT).

    • Get the firmware versions of all the boards (if this information is not available during the site inventory phase).

    • Verify empty slot presence shown in the CPLT that do not correspond to what installed in the site (that is, CPLT shows empty slot while the card is inserted in the slot).

    • Verify whether some board's serial numbers are partially shown or missing.

CMP Reset Inspection

This step is used to check the date and time of the last CMP restart.

  1. From a Telnet session run the date command to see the actual time and date.

  2. (Q3 Agent from Blue Devil onwards) From a Telnet session run cat Q3INFO.

    You should have output similar to this:

    INFO SECTION
    1.5(1)M
    PART_NUMBER SECTION
    85-03204-02
    TIME_STAMP SECTION
    date:[20/08/2001] time:[17:15:12]
    

    Note: The TIME_STAMP SECTION shows the last restart of the CMP.

  3. This pertains to the TL1 Agent only.

    1. Run telnet <ip address> 1000 to connect to the TL1 Agent and then set up a TL1 session with the NE through the activate user command:

      ACTIVATE-USER::<user_ID>:<ctag>::<password>;

      Ask the customer network administrator about a NE account with read only privileges. The default setting has both <user_ID> and <password> equal to USER_3. The correlation tag <ctag> is a string of up to six characters. For instance, the command for the default account could be:

      ACTIVATE-USER::USER_3:CONNEC:USER_3;

      If the command is successfully executed, you get a COMPLD response.

    2. Issue the retrieve equipment command:

      RETRIEVE-EQPT::ALL:EQTLIS;

      If the command is successfully executed, you get a COMPLD response and the list of all configured boards on the CMP. For example:

      DEFAULT 01-08-20 09:40:29
      M LLL COMPLD
      "TPA_R-01-01-01:IS-NR"
      "TPA_B-01-01-02:IS-NR"
      "EMPTY_SLOT-01-01-03:OOS-MT-MTCE"
      "24WD_LLR-01-01-04:IS-NR"
      "8WD_B-01-01-07:IS-NR"
      "RBA_10G-01-01-09:IS-NR"
      "BBA_10G-01-01-10:IS-NR"
      "RBU_W-01-01-11:IS-NR"
      "EMPTY_SLOT-01-01-13:OOS-MT-MTCE"
      "CMP_W-01-01-15:IS-NR"
      "IOC_W-01-01-16:OOS-MT"
      "SCF_W-01-01-17:IS-NR"
      
    3. Select the CMP entry and enter this command:

      RETRIEVE-UPTIME::CMP_W-01-01-15:LLL;

      If the command is successfully executed, you get a COMPLD response and the time of the last restart. For example:

      00-06-19 14:55:56
      M LLL COMPLD
      "CMP_W-01-01-15:00-06-19 14:37:46" 
      

PSOS Resource Usage

Among the set of relevant information for the troubleshooting are those regarded to the PSOS Operating System. They comprise the volatile memory exploiting and the current process list. Lets examine how to retrieve them:

  • OS memory usage: connect to the CMP through a Telnet session on the port 5678, then enable the trace logging and type the command monon (which enables PSOS object tracing). The operator is displayed with a recurrence of ten seconds, with a set of meaningful values, including memory fragmentation, message queues usage and so on. Send the obtained trace file to the Cisco TAC for further examination.

  • Once connected, enable trace logging within Telnet, and enter ps to list all processes running on the CMP.

Related Information

Updated: Dec 02, 2005
Document ID: 18925