Table Of Contents
Troubleshooting
K.1 Overview
K.2 CTM Troubleshooting Tools
K.3 Server Problems
K.3.1 Conditions that Affect CTM Server Performance
K.3.2 CTM Server Does Not Respond
K.3.3 Cannot Connect to the CTM Server
K.3.4 How Do I Restart the CTM Server When the Network Contains Many NEs?
K.3.5 NE Connection State Is Listed as Unavailable
K.3.6 Launching Tables Results in Database Errors
K.3.7 SNMP Traps Are Not Forwarded from NEs
K.3.8 Trap Port Is Unavailable
K.3.9 Problems with ONS 15530 and ONS 15540 Configuration
K.3.10 Performance Monitoring Data Is Not Displayed
K.3.11 CRS-1 and XR 12000 Communication Problems
K.3.12 CTC-Based NE Is Not Discovered
K.3.13 CTC-Based NE Is Not Reachable
K.3.14 ONS 1530x Communication Problems
K.3.15 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable
K.3.16 ONS 1580x NE Is Not Discovered
K.3.17 Changing the NE Operational State from the UNIX CLI
K.3.18 Circuits Are Not Displayed
K.3.19 Circuit State for Monitor Circuits Reads "Duplicate ID"
K.3.20 NE Model Type Appears as Unknown
K.3.21 Cannot Copy CTC Binary to the CTM Server
K.3.22 Memory Backup, Memory Restore, or Software Download Fails
K.3.23 Memory Autobackup, Software Commit, or Software Revert Fails
K.3.24 The getinfo.sh Script Fails
K.4 Client Connectivity Problems
K.4.1 Database Is Not Available
K.4.2 Database Timeout Occurs
K.4.3 Are the CTM Client and the CTM Server Connected?
K.4.4 Cannot Log In as Provisioner or Operator
K.4.5 Cannot Authenticate User Message Appears
K.5 Client Operational Problems
K.5.1 Model Index Is Unknown
K.5.2 Cannot Delete an NE
K.5.3 CTC Fails to Start
K.5.4 Added a New Software Version to the Wrong NE
K.5.5 CRS-1 and XR 12000 NE Explorer Applications Are Disabled
K.5.6 Cannot Delete a Subnetwork
K.5.7 Cannot Move an NE Between Subnetworks
K.5.8 NEs Change Subnetworks Rapidly
K.5.9 Cannot Schedule Jobs
K.5.10 CTM Does Not Receive Autonomous Alarms From ONS 1530x NEs
K.5.11 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms
K.5.12 Bandwidth-Intensive Operations Are Blocked
K.5.13 NE Displays Incorrect Configuration Management Data
K.5.14 Cannot Customize the Network Map
K.5.15 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation?
K.5.16 CTM Client Machine Displays Incorrect Colors
K.5.17 Common Topology Problems
K.5.18 Common VLAN Problems
K.5.19 CRS-1 and XR 12000 PM Collection Issues
K.5.20 PM Data Collection Failed
K.5.21 How Do I Collect Solaris Client Thread Dump?
K.5.22 How Do I Collect Windows Client Thread Dump?
K.5.23 How Do I Collect Server Thread Dumps?
K.5.24 How Do I Enable or Disable the Automatic Refresh Data Action?
K.5.25 How Do I Replace the Alarm Interface Panel?
K.6 Client Debug Messages
K.7 CTM GateWay/TL1 Problems
K.8 CTM GateWay/CORBA Problems
K.9 Cisco 7600 Problems
K.9.1 Cisco 7600 NE Not In Service and Available
K.9.2 NE Hangs in the InService State and May Later Become Unavailable
K.9.3 NE Changes to Loss of Communication State
K.9.4 Configuration Fails and Configuration Jobs Take Longer to Complete than the Preconfigured Time
K.9.5 Configuration Completes with a Warning
K.9.6 BGP Neighbor and FTP Passwords Cannot Use Some Special Embedded Characters
K.9.7 CTM Does Not Discover Connected Cisco 7609 Devices
K.9.8 The IP Address of Discovered Device Is Not the Managed IP Address
K.9.9 CTM Server Does Not Receive Traps
K.9.10 Software Download, Memory Backup, and Memory Restore Failure
K.9.11 Syslog Viewer Does Not Display Faults
K.9.12 Startup Configuration Backup Fails
K.9.13 Software Activation on the Cisco 7600 NE Fails
K.9.14 Restoring a Running Configuration Fails
K.9.15 Using the Correct CTM Debug Module
K.9.16 Collection and Location of Logs
K.9.17 Enabling Detailed CE Debugging
K.10 Cisco CRS-1 and Catalyst 6509 Problems
K.10.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service
K.10.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service
K.10.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1
K.10.4 CTM Does Not Receive Traps from the Catalyst 6509
K.10.5 CTM Does Not Receive syslog Events from the Catalyst 6509
K.10.6 CTM Does Not Collect Historical PM Data for the CRS-1
K.10.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval
K.10.8 Memory Backup or Restore Fails for the CRS-1
K.10.9 How Do I Back Up the CRS-1 Configuration File Automatically Whenever the Configuration Is Changed on the Router?
K.10.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs
K.10.11 Cannot Delete an Out-of-Service NE from CTM
K.10.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty
K.10.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer
K.10.14 How Do I Configure Telnet on the CRS-1?
K.10.15 How Do I Configure SSH on the CRS-1?
K.10.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs?
K.10.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509?
K.10.18 How Do I View the SNMP Configuration on the Catalyst 6509?
K.10.19 How Do I Load a Software Image on the Catalyst 6509?
K.11 XR 12000 Problems
K.11.1 Reporting XR 12000 Issues
K.11.2 CTM Cannot Discover an XR 12000 Node
K.11.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer
K.11.4 XR 12000 Alarm and Event Reporting Issues
K.11.5 XR 12000 PM Collection Issues
K.11.6 XR 12000 BGP and CDP Neighbor Discovery Issues
K.11.7 XR 12000 NE Explorer Applications Are Disabled
K.12 Problems with MGX Voice Gateway Devices
K.12.1 Discovery Mechanism
K.12.2 Discovery Issues at Startup
K.12.3 Discovery Issues at Runtime
K.12.4 Equipment Management Problems
K.12.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems
K.12.6 Chassis View Problems
K.12.7 Configuration Center Management
K.12.8 Connection Management Problems
K.12.9 Diagnostics Center Problems
K.12.10 Performance Management Collection and Parsing Problems
K.12.11 Statistics Report Problems
K.12.12 Service Agent Problems
K.12.13 Miscellaneous Problems
Troubleshooting
This chapter offers troubleshooting steps to help solve high-level problems while operating CTM or CTM GateWay. Follow the troubleshooting procedures described in this chapter before contacting Cisco technical support.
This chapter includes the following troubleshooting information:
•
Overview
•CTM Troubleshooting Tools
•Server Problems
•Client Connectivity Problems
•Client Operational Problems
•Client Debug Messages
•CTM GateWay/TL1 Problems
•CTM GateWay/CORBA Problems
•Cisco 7600 Problems
•Cisco CRS-1 and Catalyst 6509 Problems
•XR 12000 Problems
•Problems with MGX Voice Gateway Devices
Note For information about troubleshooting CiscoView problems, see J.7 Troubleshooting, page J-6 in Appendix J, "Using CiscoView to Configure and Monitor ONS 15501, ONS 15530, and ONS 15540 NEs."
K.1 Overview
Troubleshooting involves:
1. Identifying the source of the problem—Which devices, links, interfaces, hosts, or applications have the problem?
2. Locating the problem on the network—On what VLAN, subnet, or segment is the problem occurring?
3. Comparing current network performance against an established baseline—Is the performance better or worse?
4. Finding out when the problem started—When did you first see the problem? Is it recurring?
5. Determining the extent of the problem—How widespread is the problem? Is it getting worse?
Note This chapter assumes that the server is installed under the default /opt/CiscoTransportManagerServer directory and the client is installed under the default /opt/CiscoTransportManagerClient (Solaris) or C:\Cisco\TransportManagerClient directory (Windows). If a directory other than the default installation directory is specified, replace the default path with the installed path.
K.2 CTM Troubleshooting Tools
You can use all of the following to troubleshoot your system:
•Error and Audit Logs—Most NE communication problems that occur when adding a new network element to the CTM domain can be diagnosed by looking at the Error and Audit logs. See 8.5.1 Viewing the Audit Log, page 8-74 and 9.6.2 Viewing the Error Log, page 9-63 for more information.
•EMS alarms—Using the Dashboard, launch the Alarm Browser window to display only the EMS alarms. See 1.3.1 Dashboard, page 1-7 for more information.
•Self Monitor table—Launch the Self Monitor table to view server resource historical data. See 10.3.13 Using the Self Monitor Table, page 10-21 for more information.
•Debug options—This tool should only be used when recommended by engineering. See 9.6.5 Setting Debug Options, page 9-67 for more information.
•Getinfo.sh script—Provides UNIX and CTM configuration information. The following is an example of how to execute the getinfo.sh script:
The getinfo.sh script generates two files, logtext.tar.Z and techinfo.txt.Z. These files are created under the /opt/CiscoTransportManagerServer directory.
•System parameter information—Output of commands such as prstat and top provides system parameter information that can be used to troubleshoot performance issues.
•CTM server log files—See 9.6.2 Viewing the Error Log, page 9-63 for a list of server error logs.
In CTM R4.0 and later, the ctmop.log file is created is used to determine whether the server was stopped by the user or it shut down abnormally. If the server shuts down abnormally, check the core_pmon.log file to see if any of the critical services went down. Starting from CTM R4.0, services that are considered critical are Oracle, SMService, OSAgent, and GateWay/CORBA.
CTM server log files are located in the /opt/CiscoTransportManagerServer/log directory.
•Performance parameters—CTM periodically monitors vital performance parameters and raises EMS alarms when they cross a predefined threshold value. See 9.4.2 Setting Up and Viewing Alarm Configuration Parameters, page 9-18 for a list of performance parameters.
•UNIX commands:
–netstat—Shows the state of all sockets, all routing table entries, and all physical and logical interfaces.
–snoop—Captures and inspects network packets.
–/usr/platform/sun4u/sbin/prtdiag—Displays system diagnostic information, including the number of CPUs and the RAM.
–vmstat—Reports virtual memory statistics.
K.3 Server Problems
This section describes troubleshooting procedures for CTM server-related problems.
Note Log in as the root user on the Solaris workstation where the CTM server is installed to perform any operations on the Solaris workstation.
K.3.1 Conditions that Affect CTM Server Performance
The "System Requirements" chapter in the Cisco Transport Manager Installation Guide lists server, CPU, CPU speed, disk space, and RAM requirements for different types of CTM installations. These requirements are derived from guidelines based on scalability simulation testing.
Actual CTM server performance varies for each customer and is affected by:
•The total number of NEs actively managed by CTM
•The number of NE-related services (including northbound services) running on the server
•The rate of circuit provisioning and the methods used for provisioning
•The rate of network growth (for example, adding 50 NEs to CTM at once)
•The rate of configuration change updates received from the NEs
•The number of circuits provisioned with no signals that generate threshold crossing alerts (TCAs)
•The number of unacknowledged alarms in the CTM database
•The rate of alarm bursts received from the NEs
•The actual number of circuits provisioned in the CTM database
The following conditions might indicate that your CTM installation is near or at capacity:
•Your daily CPU utilization is consistently above 80%
•You experience loss of connectivity (LOC) to 1% to 5% of nodes at random
•Your RAM usage is 100% or your system swaps frequently
•You receive many configuration updates from the NEs
•Your system experiences high Java garbage collection time
Note The Java garbage collection time is determined by engineering after analyzing your CPU utilization data.
If you are experiencing any of these conditions, contact your account representative. Your account representative can engage Cisco's Advanced Services team to audit your CTM server and recommend a higher-capacity server and resources.
K.3.2 CTM Server Does Not Respond
Step 1 Log in as the root user on the Solaris workstation where the CTM server is installed.
Step 2 Enter the following command to view the status of the CTM server processes:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
If there is a line containing /CTMServer, the CTM server is running.
If there is no line containing /CTMServer, the CTM server is not running. Proceed to Step 3.
Note You can also check the ctmop.log file in /opt/CiscoTransportManagerServer/log to determine if the server was stopped by another user or if it stopped abnormally. If it stopped abnormally, proceed to Step 3.
Step 3 Run the getinfo.sh CTM server tool and send the data to Cisco technical support for analysis.
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 4 Start the CTM server by using the ctms-start script in the /opt/CiscoTransportManagerServer/bin directory.
a. Log in as the root user.
b. Change the directory to /opt/CiscoTransportManagerServer/bin and enter the following command:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
If the preceding procedure does not solve the problem, complete the following steps:
Step 1 Verify that the /opt/CiscoTransportManagerServer/cfg/CTMServer.cfg file is not corrupted. The file should contain the db-config-mode = auto parameter in the [database] section. If the entry is missing, the CTM server configuration file is corrupt. Reinstall the CTM server.
Step 2 Verify that the first entry in the /var/opt/oracle/oratab file looks similar to CTM6_0:/oraclesw9i/product/9.2:Y. If this entry is missing, the Oracle database might not be installed. The Oracle database is a prerequisite for installing the CTM server.
K.3.3 Cannot Connect to the CTM Server
Step 1 Ping the server's IP address from the client PC or workstation.
Step 2 If the ping fails, resolve the IP connectivity problem and try again.
Step 3 Telnet to the server and log in as the root user.
Step 4 Enter the following command to verify that CTM is running:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 5 The server should have at least the following processes running:
root 520 12.0 0.02855219536 Dec_19 14:18 CTM Server
root 489 0.0 0.117384 Dec_19 0:00 CTM Server
root 749 0.6 5.7255104112848 Dec_19 346:27 SnmpTrapService
root 541 0.1 5.8284232115256 Dec_19 15:02 SMService
root 507 0.0 0.2 5512 3496 ? Dec 19 Apache Web Server
Step 6 Manually stop the server if you see fewer than four processes running. Enter the following command:
Step 7 If you changed the server IP address, verify that the configuration files shown in Table 4-15 on page 4-44 have been updated. See 4.4.2 Updating the Configuration Files After Changing the CTM Server IP Address, page 4-44.
Step 8 Verify that the Oracle database is accepting connections:
a. Enter the following command to log in as the Oracle user:
b. Enter the following command to open an SQL*Plus session:
omu-u60-3% sqlplus ctmanager/ctm123!
Step 9 Reboot the server if you receive another error message and the SQL prompt does not appear. Wait enough time for the server to boot up and try to run the client. The SQL prompt indicates that the Oracle database is running and accepting connections.
Step 10 Enter the following commands to manually restart CTM:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 11 Wait for 5 minutes and run the client.
K.3.4 How Do I Restart the CTM Server When the Network Contains Many NEs?
If your network contains a high number of NEs, it often takes a long time for all of the NEs to synchronize after a CTM server restart. To avoid this NE synchronization delay, complete the following procedure.
Step 1 In the Domain Explorer, choose Administration > Control Panel.
Step 2 In the Control Panel window, expand the NE Service node and click any NE with a green arrow (which indicates that there are services running).
Step 3 In the Status tab, click the Deactivate button to deactivate the NE service. This ensures that when the CTM server is restarted, no service consumes too much CPU and the NE resynchronization goes smoothly.
Step 4 For CTC-based NEs, expand the Network Service node and click CTC-Based SONET or CTC-Based SDH. In the Status tab, click the Deactivate button to deactivate the network service.
Step 5 Expand the PM Service node and click any NE with a green arrow. In the Status area, click the Deactivate button to deactivate the PM service.
Step 6 Enter the showctm command. The command output should not show any active NE or PM services. The output should look similar to the following sample:
CTM Processes for Cisco Transport Manager Server Version: 7.2 Build: 191
-------------------------------------------------------------------------------------
USER PID %CPU %MEM START TIME PROCESS
-------------------------------------------------------------------------------------
root 14023 0.0 0.117360 11:00:13 0:00 CTM Server
root 14101 0.0 0.82607214968 11:00:18 0:01 CTM Server
root 14351 0.2 5.7263488113176 11:01:31 1:05 SnmpTrapService
root 14123 0.0 4.728914494384 11:00:20 0:57 SMService
-------------------------------------------------------------------------------------
Step 7 Enter the ctms-stop command.
Step 8 After the ctms-stop command is complete, re-enter the showctm command to verify that there are no CTM processes running. The output should look similar to the following sample:
CTM Processes for Cisco Transport Manager Server Version: 7.2 Build: 191
-------------------------------------------------------------------------------------
USER PID %CPU %MEM START TIME PROCESS
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Step 9 If any process is still running, identify the process ID and enter the following command to stop it:
Step 10 After all processes have been stopped, enter the ctms-start command at the command prompt. This command starts the CTM server but not the NE or PM services.
Step 11 Activate the NE service instances individually after most of the NEs in a partition have started synchronizing. That is, wait until the NE service instance displays an hourglass icon in the Control Panel tree before you activate the next NE service instance. To do this, expand the NE Service node; then, expand the appropriate NE node and in the NE Service Instance properties pane, click the Start Service Instance button.
Step 12 For CTC-based NEs, activate the network service instances individually after most of the NEs in a partition have started synchronizing. To do this, expand the Network Service node; then, expand the CTC-Based SONET or CTC-Based SDH nodes. In the Network Service Instance properties pane, click the Start Service Instance button.
Step 13 Activate the PM service instances. To do this, expand the PM Service node; then, expand the appropriate NE node and in the PM Service Instance properties pane, click the Start Service Instance button.
K.3.5 NE Connection State Is Listed as Unavailable
If the connection state of an NE is listed as Unavailable in the Domain Explorer window, a connectivity or configuration problem exists. Wait 5 to 10 minutes after adding the NE to the CTM domain; then, complete the following steps:
Step 1 To see the NE IP address, select the NE in the Domain Explorer window. The Address tab of the Network Element Properties pane lists the IP address of the selected NE.
Step 2 From the CTM server, enter the following command to verify connectivity between the CTM server and the NE:
Step 3 If the ping fails, a physical or configuration problem exists in the data communications network (DCN).
a. If connectivity existed earlier:
•If there were no configuration changes made to any routers in the DCN or to the CTM server, the problem is a physical problem.
•If the configuration was changed, the change might have introduced problems. Verify the changed configuration.
b. If connectivity was never established:
•Verify that there are no problems on the physical level.
•Verify the DCN configuration.
Step 4 If the ping succeeds, verify that the NE software version is listed in the Supported NE Table. See 4.3.8 Adding a New NE Software Version to the CTM Domain, page 4-38.
Step 5 For ONS 15501, ONS 15530, and ONS 15540 NEs, SNMP read and write community strings must be set up properly on each NE. The running configuration on each NE should contain entries similar to the following example:
snmp-server community public RO
snmp-server community private RW
The community strings configured on the server should match the strings configured on the NE. For information on configuring community strings in CTM for ONS 15501, ONS 15530, and ONS 15540 NEs, see 3.5.1.4 Prerequisites for Adding ONS 15501, ONS 15530, and ONS 15540 NEs, page 3-10.
K.3.6 Launching Tables Results in Database Errors
If the Oracle database or Oracle listener that CTM is using is down, launching tables will generate database errors. To troubleshoot table launching errors:
Step 1 Log in as the Oracle user and enter the following command:
sqlplus ctmanager/ctm123!
If the login is successful, an SQL> prompt appears, which indicates that the Oracle database has been installed and the database server is up and running. If login fails, either the Oracle database has not been installed or the database server is not running.
Step 2 To start the Oracle database, log into the Solaris workstation as the Oracle software owner user.
Step 3 Enter the following command at the shell prompt to start the Oracle database:
Step 4 Enter the following command at the shell prompt to start the Oracle listener:
If there are still problems with starting the Oracle database or the Oracle listener, refer to the Oracle documentation or contact Oracle support.
K.3.7 SNMP Traps Are Not Forwarded from NEs
SNMP traps might not be forwarded, either because the trap port is already in use, or because the NE is not properly configured.
K.3.8 Trap Port Is Unavailable
The CTM server requires exclusive access to the SNMP trap port to receive SNMP traps from the NE.
Step 1 To verify that the standard SNMP trap port (number 162) is not being used by another application running on the same Solaris workstation, enter the following command:
If the following line is seen, the SNMP trap port is being used by another application:
Step 2 If the trap port is being used by another application, stop the other application.
K.3.9 Problems with ONS 15530 and ONS 15540 Configuration
ONS 15530 and ONS 15540 NEs must be properly configured to send traps to the server.
Step 1 The running configuration on the NE should contain entries similar to the following example:
snmp-server enable traps snmp authentication warmstart
snmp-server enable traps bgp
snmp-server enable traps oscp
snmp-server enable traps config
snmp-server enable traps entity
snmp-server enable traps fru-ctrl
snmp-server enable traps topology throttle-interval 60
snmp-server enable traps optical monitor min-severity not-alarmed
snmp-server enable traps rf
snmp-server enable traps aps
snmp-server enable traps patch
snmp-server enable traps cdl all
snmp-server enable traps alarms
snmp-server enable traps threshold min-severity degrade
snmp-server host 172.20.126.214 version 2c public snmp bgp oscp
config entity fru-ctrl topology optical rf aps patch cdl alarms threshold
In this example, the host 172.20.126.214 will receive the specified traps from the NE.
Step 2 You can also run the following commands on the server and device:
a. As the root user, enter the following command on the server:
b. On the device, enter the following command to generate a configuration change:
If the device is configured correctly, the snoop command reports a configuration change trap.
K.3.10 Performance Monitoring Data Is Not Displayed
Step 1 Use the Domain Explorer NE Properties pane to see if PM collection is disabled for the selected NE.
Step 2 Choose Administration > Control Panel and expand PM Service. Select the NE type to see if the service status is Active or Not Active.
Step 3 Choose Administration > Audit Log to see if PM was ignored for the selected NE.
Step 4 Choose Administration > Control Panel and click PM Service. Look at the PM Data Storage field. Choose Optimized to save only nonzero values in the database. Choosing Optimized requires less space than saving all values, including zero values. If you choose Normal in the PM Data Storage field, the database saves all PM values, including zero values.
Note The Optimized PM data storage feature is available only for CTC-based and ONS 155xx NEs.
Step 5 Choose Administration > Error Log to see if there are any critical, major, or minor errors against the PM service.
Step 6 Verify that the NE is reachable during the duration that PM data is collected. If the NE is not reachable, there is an entry in the Audit Log table (Administration > Audit Log).
K.3.11 CRS-1 and XR 12000 Communication Problems
Step 1 Check whether the NE is configured with root-system privileges and the following login information:
•Username: hfrems
•Password: hfrems
Step 2 Verify that the XML CORBA agent service is running.
Step 3 The CORBA agent on the router only accepts connections when using the hostname. If the IP address has an entry in DNS, check the device. If DNS is not used, add an entry in the /etc/hosts file.
Step 4 Check in the Supported NE Table to determine whether the version of the software image running on the router is supported by CTM.
Note In CTM, when an NE is added, the NE type is retrieved to determine if it is a CRS-1 or XR 12000 NE. If it is not a CRS-1 or XR 12000, the NE changes to Loss of Communication state and an NE type mismatch alarm is raised.
Note The NE hostname specified in the DNS or the CTM server /etc/hosts file should match the hostname configured on the NE. If there is a mismatch, the NE changes to Loss of Communication state.
K.3.12 CTC-Based NE Is Not Discovered
Step 1 Verify that the NE is up and running.
Step 2 Verify that the NE IP address and default route are configured correctly.
Step 3 To verify that the NE is running the correct version of system software, open a CTC session to the NE and check the NE software version.
Step 4 Verify that the NE that acts as the gateway NE (GNE) (through which this node is discovered) is added to CTM and is available.
Step 5 Open CTC from the GNE and see if CTC can discover the NE.
K.3.13 CTC-Based NE Is Not Reachable
Step 1 Verify that the NE is up and running.
Step 2 Verify that the NE IP address and default route are correctly configured.
Step 3 Wait for five poll cycles while CTM reestablishes connectivity with the NE.
Step 4 To test IP connectivity to the NE from the CTM server, enter the following command from the Solaris workstation running CTM:
Step 5 To verify that the NE is running the version of system software supported by CTM, open a CTC session to the NE and check the NE software version.
Step 6 Verify that the username and password that CTM uses to reach the NE exist on the NE.
K.3.14 ONS 1530x Communication Problems
Step 1 Make sure that there are no other ONS 1530x NEs with the same name.
Step 2 Test the IP connectivity from the server to the NE.
Step 3 Telnet to the NE from the CTM server. If you are prompted for a password, the NE is running.
Step 4 Connect to the NE using Cisco Edge Craft to verify that the SNMP is enabled.
Note Only users connecting from servers that are listed in the SNMP Community table are allowed to log into the NE.
Step 5 Verify that the NE you are trying to connect to is running a software version supported by CTM.
Step 6 Check if the product name is one of the following:
•ONS 15302
•ONS 15305
•AXXEDGE
If the product name does not match any of the recognized names, enter the change-system-name command.
K.3.15 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable
Step 1 Enter the following command on the server to verify IP connectivity between the server and the NE:
Step 2 If the NE is reachable through IP, verify that the correct read community string is configured on the server.
•On the server, choose Administration > ONS 155XX > ONS 155XX SNMP Settings Table to view the community strings set in CTM.
•On the NE, enter the following command to view the read community string set on the NE:
•In the running configuration, the community strings are shown in entries similar to the following:
snmp-server community public RO
snmp-server community private RW
In this example, the read string is public, and the read/write string is private.
Step 3 If IP connectivity and community strings are verified and the device is still unavailable, check the error log for any errors reported by ONS 155xx NE Service.
K.3.16 ONS 1580x NE Is Not Discovered
If CTM cannot discover a newly added ONS 1580x NE, do the following:
Step 1 Make sure that there are no other ONS 1580x NEs with the same name.
Step 2 Connect to the NE using the local craft tool and provide the current running level. The running level must be six to enable CTM connect to the NE. If the running level is three, only the local craft is able to connect.
Step 3 Do the following to check all available connections on the TL1 socket (port 1000):
Note The TL1 Agent supports up to five simultaneous connections. CTM uses one TL1 connection for the NE service, one for the PM service, and another for the GateWay/TL1, if enabled.
a. Open a Telnet connection.
b. At the pSOS prompt, enter the netstat command.
Step 4 Make sure that the board name and slot position inside the shelf match those reported by the TL1 Agent.
K.3.17 Changing the NE Operational State from the UNIX CLI
This procedure is useful when troubleshooting NE connectivity problems, alarm, or configuration synchronization issues and you do not have access to the CTM client GUI.
Step 1 Identify the NE database ID number using the vwdata.sh script.
Step 2 Run the UNIX CLI tool and connect to the server. The tool name is ctm and is located in the /opt/CiscoTransportManagerServer/bin directory.
Step 3 Change the NE operational state using the command set nestate.
K.3.18 Circuits Are Not Displayed
If CTM fails to display some circuits or if the information displayed by CTC and CTM differs, complete the following steps:
Step 1 Wait for two minutes while the CTM server synchronizes with the NE. If the Circuit table is not in autorefresh mode, click the Refresh Data tool when the tool flashes.
Step 2 Verify that the Circuit table is opened from either the source or destination NE of the circuit to be viewed.
Step 3 Verify that the NEs that are part of the circuit (source, destination, or drop) are available in CTM.
Step 4 Open another CTC session and compare the circuit information in the newly opened CTC session with the circuit information displayed by CTM.
K.3.19 Circuit State for Monitor Circuits Reads "Duplicate ID"
CTM generates a unique number that is appended to the circuit name to make the monitor circuit name unique. On rare occasions, CTM might create two or more monitor circuits with the same name. The circuit state for these circuits reads "Duplicate ID." This is a known issue that has been tracked as DDTS number CSCdz87566.
When the circuit state reads "Duplicate ID," you cannot see the actual circuit state. In this case, you must change the duplicate name to a unique name, so that you can see the correct circuit state. Use the Modify Circuit wizard to enter a unique name for each monitor circuit.
K.3.20 NE Model Type Appears as Unknown
If an in-service NE is added to the Domain Explorer, but the model type appears as unknown, the software version of the NE might not be prepopulated in the database. In other words, CTM cannot match the NE with a recognizable version.
Use the NE Software Table to add the NE software version string to the CTM database. See 4.3.6 Viewing Software Versions and Restarting the NE with a New Software Image, page 4-35.
K.3.21 Cannot Copy CTC Binary to the CTM Server
If a CTC binary fails to be copied to the CTM server, the <CTM_install_directory>/cms/ directory might be missing or write-protected. If the directory is missing, create a cms directory with write-access permissions. Otherwise, change the permissions of the existing cms directory to allow write access.
K.3.22 Memory Backup, Memory Restore, or Software Download Fails
Step 1 In the Domain Explorer window, choose Administration > Job Monitor. The Job Monitor table shows the status of the operation. The reason for the failure is shown in the Additional Information column.
Step 2 Return to the Domain Explorer window and choose Administration > Error Log. The Error Log table shows information about the backup, memory restore, or download failure.
Step 3 See Appendix I, "Error Messages" for the correct action to take in response to the error.
K.3.23 Memory Autobackup, Software Commit, or Software Revert Fails
Step 1 In the Domain Explorer window, choose Administration > Audit Log. The Audit Log table shows the status of the operation.
Step 2 Return to the Domain Explorer window and choose Administration > Error Log. The Error Log table shows the reason for the failure.
Step 3 See I.2 CTM Server Error Messages, page I-95 for the correct action to take in response to the error.
K.3.24 The getinfo.sh Script Fails
You might experience a problem where the getinfo.sh script fails with the following error:
"Error: permission denied while running `rsh.' Check the environment and make sure the root is permitted to run .rsh on host <IP_address>."
This problem occurs because the getinfo.sh script does not run unless an entry is made in the /.rhosts file. This problem occurs even on a local server and database configuration. This is a known issue that has been tracked as DDTS number CSCin66495.
The solution is to add the hostname to the /.rhosts file. For example:
(omu-e450) #more /.rhosts
K.4 Client Connectivity Problems
The CTM client might not be able to connect to the CTM server for various reasons. Complete the following procedures in the order listed until the problem is resolved.
K.4.1 Database Is Not Available
If the CTM client cannot connect to the CTM server, verify that the database is available.
Step 1 Log into the CTM server as the Oracle user.
Step 2 Enter the following command to connect to the database:
sqlplus ctmanager/ctm123!
Step 3 If the error message "maximum processes exceeded" is received, the maximum number of database connections has been reached. Close several clients or ask the database administrator to increase the maximum number of processes for the database.
K.4.2 Database Timeout Occurs
Step 1 Reduce the scope of the query by selecting a group or an NE and not the entire domain before opening tables such as the Alarm Browser, Alarm Log, Audit Log, and PM tables.
Step 2 Increase the client database query timeout period by editing the ems-client.cfg file in the C:\Cisco\TransportManagerClient\config or /opt/CiscoTransportManagerClient/config directory.
Step 3 Add hardware resources to the Oracle database server.
Step 4 Ping the Oracle database server from the client to verify the response time. Increase the bandwidth if the round-trip response time is inadequate.
K.4.3 Are the CTM Client and the CTM Server Connected?
If the database is available, check connectivity between the client and the server.
Step 1 To see the CTM server IP address, enter the following command on the Solaris workstation that is running the CTM server:
The command output looks similar to the following example:
hme0:flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST>mtu 1500
inet 192.168.120.93 netmask ffffff00 broadcast 192.168.120.255
The IP address is the address following the inet field.
Step 2 To verify that the physical connection between the CTM client and the CTM server does not have problems, enter the following command from the CTM client:
where the IP address belongs to the Solaris workstation that runs the CTM server.
Step 3 If the ping command is not successful, fix the physical connectivity; then, log into the CTM client.
K.4.4 Cannot Log In as Provisioner or Operator
Step 1 Use the default username and password to log into the CTM client:
Username: SysAdmin
Password: Ctm123!
Step 2 If the SysAdmin login is successful, but the provisioner or operator login fails, verify that the provisioner or operator exists and is not disabled. In the Domain Explorer window, choose Administration > CTM Users to view a table of all configured CTM users.
Step 3 If the provisioner or operator is not in the CTM Users table, the user is not configured. Configure the provisioner or operator; then, log in as that user.
Step 4 If the provisioner or operator is in the CTM Users table, select the row corresponding to that user and click the Modify User Properties tool to bring up the Modify CTM User Properties wizard. Verify that the Login State field is set to Enabled. If the Login state is disabled, enable it and log in as the provisioner or operator. The user might have been disabled after attempting to log in with an incorrect password.
Step 5 If the password is not correct, set a new password for the user and log in again.
K.4.5 Cannot Authenticate User Message Appears
If the "Cannot authenticate user" error message is received when logging into the CTM client, the CTM server might be initializing. Wait for five minutes while the CTM server finishes initializing; then, try to log in again. Alternately, check your username and password and enter them again. The username and password are case sensitive.
K.5 Client Operational Problems
This section describes troubleshooting procedures for CTM client operational problems.
K.5.1 Model Index Is Unknown
If a new NE is added to the CTM domain and marked as In Service, but the model index is unknown, complete the following steps:
Step 1 Verify that the IP address and other parameters are entered correctly.
Step 2 The CTM server might not be able to establish connectivity to the GNE for the NE. Verify that the GNE for the NE is marked as In Service, and that the GNE is correctly populated in the CTM domain.
Step 3 Verify that the GNE is DCC-connected to the NE.
Step 4 The CTM server might not recognize the NE's software version. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity. If the NE is available, contact Cisco Systems and verify that the version of the software on the NE is compatible with the CTM server being used. Use the Supported NE Table to provide the new NE software version that the CTM server will recognize.
Step 5 The CTM server might be configured to attempt communication with new NEs after a certain delay. To change the health poll frequency:
a. In the Domain Explorer window, choose Administration > Control Panel and click NE Service.
b. In the NE Poller tab, look at the NE Health Poll Interval field. The default is 240 seconds. If the value is too large, there might be a communication delay between CTM and the NEs. Under normal conditions, the CTM server attempts communication with new NEs for no more than twice the number of seconds specified in the NE Health Poll Interval field.
K.5.2 Cannot Delete an NE
If an NE cannot be deleted, verify that the user logged in as SuperUser or NetworkAdmin—not as a Provisioner or Operator. Provisioners and Operators cannot delete NEs. If the user logged in as Provisioner or Operator, restart the CTM client session and log in as SuperUser.
K.5.3 CTC Fails to Start
If CTC cannot be started for a CTC-based NE, complete the following steps:
Step 1 If an error occurs while opening the NE Explorer yet the CTC progress screen and login dialog boxes appear, the CTC username or password might be incorrect.
a. Choose Administration > CTM Users.
b. In the CTM Users table, choose Edit > Modify. The Modify CTM User Properties wizard opens.
c. Click Next to move to the CTC/Craft User Properties panel and change the CTC username and password to match those configured on the device. Click Finish.
Step 2 If the CTC progress screen idles for a long time before the login dialog box appears, there might be problems with network connectivity to the device.
a. From a DOS command window or a Sun Solaris terminal, ping the device.
b. If no response is received, set up routes so that the device is available; then, restart CTC. If CTC does not start, the workstation might have resource constraints.
c. Close some open applications and try again.
d. Close some instances of CTC that are managing another group of CTC-based NEs and try again.
Step 3 If the GNE for the selected NE was not found, there might be a mismatch between the GNE specified for the device and the available GNEs. This means that the data in the database is corrupt. Contact Cisco technical support for assistance.
K.5.4 Added a New Software Version to the Wrong NE
Step 1 In the Domain Explorer window, choose Administration > Supported NE Table.
Step 2 In the Supported NE Table, select the incorrect entry and choose Edit > Delete.
Step 3 Click OK in the confirmation dialog box.
Step 4 Select the correct NE row from the Supported NE Table.
Step 5 Add the correct software version.
Step 6 In the Domain Explorer > Network Element Properties pane, set the operational state of all NEs that are behaving erroneously to Out of Service.
Step 7 Click Save.
Step 8 In the Network Element Properties pane, set the operational state of all the NEs back to In Service.
Step 9 Click Save.
K.5.5 CRS-1 and XR 12000 NE Explorer Applications Are Disabled
CRS-1 and XR 12000 NE Explorer applications are disabled when the XML version in CTM does not match the XML version in the router.
Step 1 Collect the XML files from the CTM_HOME/cfg/metadata directory.
Step 2 Run the CTM_HOME/bin/vwdata.sh script to collect the hfr_version_info database table entries.
Step 3 If any of the applications throws an error, run one of the following depending on your configuration:
•In Windows:
–CTM_HOME/bin/ctm-small-debug.exe
–CTM_HOME/bin/ctm-medium-debug.exe
–CTM_HOME/bin/ctm-large-debug.exe
–CTM_HOME/bin/ctm-highend-debug.exe
•In UNIX, run CTM_HOME/bin/ctmcdebug-start, then:
–CTM_HOME/bin/ctm-small-debug.exe
–CTM_HOME/bin/ctm-medium-debug.exe
–CTM_HOME/bin/ctm-large-debug.exe
–CTM_HOME/bin/ctm-highend-debug.exe
Step 4 Collect the console logs and terminal messages.
K.5.6 Cannot Delete a Subnetwork
If a subnetwork cannot be deleted from the Subnetwork Explorer, complete the following steps:
Step 1 Verify that the user logged in as a SuperUser—not as a Provisioner or an Operator. Provisioners and Operators cannot delete subnetworks. If the user logged in as a Provisioner or Operator, restart the CTM client session and log in as a SuperUser.
Step 2 Verify that the target subnetwork does not contain NEs. Move all NEs to another subnetwork before deleting the empty target subnetwork.
K.5.7 Cannot Move an NE Between Subnetworks
When automatic grouping of NEs in subnetworks is enabled, you cannot move NEs between subnetworks. To move NEs from one subnetwork to another:
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 In the Control Panel, click UI Properties.
Step 3 Under Subnetwork Grouping, uncheck the Automatically Group NEs in Subnetworks check box.
Step 4 Click Save.
K.5.8 NEs Change Subnetworks Rapidly
NEs that are part of a subnetwork may change subnetworks rapidly if the status of the link or of all links connecting subnetworks fluctuates rapidly; this can occur if the automatic grouping of subnetworks option is enabled in CTM. Disabling the automatic grouping of subnetworks may prevent NEs from changing subnetworks when caused by links going up and down. To disable the automatic grouping of subnetworks:
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 In the Control Panel, click UI Properties.
Step 3 Under Subnetwork Grouping, uncheck the Automatically Group NEs in Subnetworks check box.
Step 4 Click Save.
K.5.9 Cannot Schedule Jobs
The CTM client can schedule three types of administrative tasks:
•Software download
•Memory backup
•Memory restore
CTM maintains an Error Log and Audit Log to track potential problems. To view the Error Log or Audit Log:
Step 1 In the Domain Explorer window, choose Administration > Audit Log or Error Log.
Step 2 Look for errors related to software download, memory backup, or memory restore.
K.5.10 CTM Does Not Receive Autonomous Alarms From ONS 1530x NEs
CTM can collect alarms or receive autonomous notifications from an ONS 1530x NE. Alarm collection occurs when the NE is first discovered and does not rely on spontaneous events coming from the NE.
Step 1 Verify that the CTM server is present in the SNMP Community table and that the trap is set to enable. If not, you can connect to the NE but CTM will not receive traps.
Step 2 If the CTM server and the NEs are in different subnetworks, make sure that there is no firewall filtering SNMP packets between the subnetworks.
Step 3 Verify that Cisco Edge Craft is getting autonomous alarms.
Step 4 Check the following:
•SNMP traps reach the CTM server machine.
•The command snoop udp and port 162 do not show the incoming traps.
•There is no network problem.
Step 5 Check if the CTM server application is receiving traps.
Step 6 Check if the ONS 1530x NE service is receiving SNMP traps.
K.5.11 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms
Occasionally, ONS 15800, ONS 15801, and ONS 15808 NEs generate so many alarms that the CTM client cannot process all of them. Set the operational state of the NE to Under Maintenance while debugging the cause of the alarms:
Step 1 In the CTM Domain Explorer window, right-click the ONS 15800, ONS 15801, or ONS 15808 that is generating the large number of alarms and choose Mark Under Maintenance.
Step 2 Debug the ONS 15800, ONS 15801, or ONS 15808 NE to see which card is in error. Find the card that is generating the alarms and mark it as Under Maintenance.
Step 3 While the card is under maintenance for debugging, set the ONS 15800, ONS 15801, or ONS 15808 NE back to In Service.
K.5.12 Bandwidth-Intensive Operations Are Blocked
Bottlenecks in the DCN affect bandwidth-intensive operations, such as software download. Tune the timeout and retry values to match DCN performance. Log into the NE and enter the following commands:
•ip tftp min-timeout—Sets the minimum wait time, in seconds, before retrying the read/write request.
•ip tftp max-timeout—Sets the maximum wait time, in seconds, before retrying the read/write request.
•ip tftp backoff-delay—Sets the number of seconds to extend wait time if the read/write request times out.
•ip tftp write-retries—Sets the maximum number of times to retry the TFTP read/write request before declaring a failure.
K.5.13 NE Displays Incorrect Configuration Management Data
Step 1 In the Domain Explorer window, choose Administration > Service Monitor and verify that the NE Service is running. If it is not running, start the NE Service.
Step 2 Test NE connectivity.
a. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity. The message "<NE_name> (<IP_address>) is available" should be generated.
b. If the NE is unavailable, establish connectivity to the NE before retrieving configuration data.
K.5.14 Cannot Customize the Network Map
If an image file is not displayed while changing the Network Map background or while changing a node icon, complete the following steps:
Step 1 Choose another image file. The file might be corrupt.
Step 2 Check the size of the image file. The image file might be larger than 100 KB, which is too big to load. If the file is too big, use a smaller image file.
Step 3 Verify that the image file exists in the \images\mapbkgnds\shapefiles directory or the \images\mapicons directory. If the file is missing, it has been deleted. Reinstall the CTM client.
Note The client is bundled primarily with shape files (*.shp) and only a few map background GIF files.
K.5.15 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation?
To run the client on a Sun Ultra 5 workstation, the color map must be changed from 8 bit to 24 bit. If the colors do not look right, change the color map from 8 bit (the default) to 24 bit as follows:
Step 1 To see the current color settings, enter the following command:
/usr/sbin/m64config -prconf
Command output looks similar to the following example:
--- Hardware Configuration for /dev/fbs/m640 ---
Card possible resolutions: 720x400x88, 640x480x60, 640x480x72, 640x480x75, 800x600x56,
800x600x60, 800x600x72, 800x600x75, 1024x768x87, 1024x768x60, 1024x768x70, 1024x768x75,
1280x1024x75, 1280x1024x60, 1152x900x66, 1152x900x76, 1280x1024x67, 1280x800x76,
1280x1024x85
1280x1024x76, 1152x864x75, 1024x768x77, 1024x800x84, vga, svga, 1152, 1280, 800x600,
1024x768, 1280x1024, 1152x900
Monitor possible resolutions: 720x400x70, 720x400x88, 640x480x60, 640x480x67, 640x480x72,
640x480x75, 800x600x56, 800x600x60, 800x600x72, 800x600x75, 832x624x75, 1024x768x60,
1024x768x70, 1024x768x75, 1280x1024x75, 1152x870x75, 1152x900x66, 1152x900x76,
1280x1024x67, 1280x1024x76, vga, svga, 1152, 1280, 800x600, 1024x768, 1280x1024, 1152x900
Current resolution setting: 1280x1024x76
Step 2 To change the color setting to 24 bit, log in as the root user and enter the following commands:
/usr/sbin/m64config -depth 24 -res 1152x900x76
Step 3 Reboot the workstation for the new color settings to take effect.
Note The resolution in 24-bit depth is a little lower than is possible with 8-bit depth (1152 x 900 x 76 versus 1280 x 1024 x 76), but the difference is hardly noticeable.
K.5.16 CTM Client Machine Displays Incorrect Colors
When using a UNIX machine as the CTM client host, the machine's display may show incorrect colors. This problem occurs when UNIX systems use a 256-color palette and the graphics require colors that are not available in the 256-color scheme. Although some UNIX systems can work around the missing colors, there are still some that have problems making adjustments. For these systems, the only solution is to disable any other graphical application (for example, Netscape) that is running and/or upgrade the systems' graphics capabilities to a 24-bit color display (for example, upgrade the video card or monitor.)
K.5.17 Common Topology Problems
Note After marking NEs as Out of Service, wait until the circuits traversing over these NEs (and not involving any other NE) clear from the CTM Circuit table before marking the NEs as In Service again.
K.5.17.1 Topology Created Goes into Incomplete State
The topology goes into an Incomplete state when one or more Layer 1 circuits related to the topology are missing.
Step 1 Launch the Circuit table and check if all the circuits related to the topology are discovered properly.
Step 2 If one or more Layer 1 circuits are missing, recreate the circuits using the Circuit Creation wizard. Once the Layer 1 circuits are created and CTM discovers the new circuits, the topology goes into Complete state.
Step 3 Check the Circuit table to verify that the newly created Layer 1 circuits are discovered properly.
Step 4 If the topology still fails to go into Complete state, the topology discovery logic has failed. Do the following to restart the discovery process:
a. Mark the NE service related to the NE as Out of Service and then mark it as In Service.
b. Mark the NE related to the topology as Out of Service and then mark it as In Service.
Step 5 If the problem persists, contact the Cisco Technical Assistance Center (TAC).
K.5.17.2 Topology Created Goes into L2ServiceNotReady State
The topology goes into L2ServiceNotReady state because there is a mismatch in data present in the CTM database and in data present in the ML-series cards.
Step 1 Launch the Circuit table and check if all the circuits related to the topology are discovered properly.
Step 2 Launch the Equipment Inventory table and check whether the NE inventory is collected.
Step 3 Do one of the following:
•Launch the L2 Topology sable and enable L2 service. The CTM database and ML-series card synchronizes. During the synchronization process, the topology goes into the InProgress state, and then to the Complete state.
•Do a force resync of the database by launching the NE Explorer for the NEs in the topology and do an NE refresh. This restarts the synchronization process. Once the NEs are up, the topology goes into Complete state.
Step 4 If the topology still fails to go into Complete state, do one of the following:
a. Telnet into the cards and check whether the configuration is corrupt.
b. Mark the NE service related to the NE as Out of Service and then mark it as In Service.
c. Mark the NE related to the topology as Out of Service and then mark it as In Service.
d. Delete the topology, reload the config file on the ML-series cards, and reset the cards. To make sure that the barebone config file is reloaded, execute the write memory command.
Step 5 If the problem persists, contact the TAC.
K.5.17.3 Topology Created Goes into SyncFailed State
The topology can go into SyncFailed state for any of the following reasons:
•A preprovisioned card was used to create a topology. There are two ways you can determine whether or not a card is preprovisioned:
–Launch the NE Explorer and check if the card is present.
–Telnet to a card to see if the card is present.
•The barebone configuration file is not present in all the cards.
•You have not specified the correct CTM-to-ML-series card login details.
•The database and the ML-series card are not synchronized.
Step 1 Launch the Circuit table and check if all the circuits related to the topology are discovered properly.
Step 2 Launch the Equipment Inventory table and check whether the NE inventory is collected.
Step 3 Wait for the next synchronization cycle. The database should synchronize and the topology should go into InProgress state. If the topology goes into L2ServiceNotReady state, see Topology Created Goes into L2ServiceNotReady State.
Step 4 If the topology still fails to go into Complete state, do one of the following:
a. Telnet into the cards and check whether the configuration is corrupt.
b. Mark the NE service related to the NE as Out of Service and then mark it as In Service.
c. Mark the NE related to the topology as Out of Service and then mark it as In Service.
d. Delete the topology, reload the config file on the ML-series cards, and reset the cards. To make sure that the barebone config file is reloaded, execute the write memory command.
Step 5 If the problem persists, contact the TAC.
K.5.17.4 Topology Created Goes into Partially Complete State
The topology goes into Partially Complete state because all the cards are not in synch with the CTM database.
Step 1 Launch the Circuit table and check if all the circuits related to the topology are discovered properly.
Step 2 Launch the Equipment Inventory table and check whether the NE inventory is collected.
Step 3 Wait for the next synchronization cycle. The database should synchronize and the topology should go into Complete state. If the topology goes into L2ServiceNotReady state, see Topology Created Goes into L2ServiceNotReady State.
Step 4 If the topology still fails to go into Complete state, do one of the following:
a. Telnet into the cards and check whether the configuration is corrupt.
b. Mark the NE service related to the NE as Out of Service and then mark it as In Service.
c. Mark the NE related to the topology as Out of Service and then mark it as In Service.
d. Launch the NE Explorer of the related NE and do a force resync by clicking the Refresh button.
e. Delete the topology, reload the config file on the cards, and reset the cards. To make sure that the barebone config file is reloaded, execute the write memory command.
f. Check if the IOS Users table has all the entries for all cards included in a ring.
Step 5 If the problem persists, contact the TAC.
K.5.17.5 CTM Cannot Discover the Created Topology
Step 1 Restart the discovery process by doing the following:
a. Launch the Circuit table and check if all the circuits related to the topology are discovered properly.
b. Mark the NE service related to the NE as Out of Service and then mark it as In Service.
c. Mark the NE related to the topology as Out of Service and then mark it as In Service.
Step 2 If the topology is still not discovered, do the following to enable the trace level in the DataService logs:
a. In the Domain Explorer window, choose Administration > Control Panel.
b. In the Control Panel window, select the NE Service to open the NE service pane.
c. Click the NE Manual Backup tab.
d. Select Trace as the error level.
e. Click Save.
Step 3 On the CTM server, Telnet to the localhost. This starts the data log collection in the server. Check if the DataService logs are populated.
Step 4 Check the DataService logs to determine whether or not the circuit addition events are received properly. You can do this by searching for the string CIRCUIT_ADDED. If the circuit addition events are not received properly, it means there is a problem with Layer 1 circuit provisioning. If they are receiving properly, it means that circuit deletion logic failed and you need to log a bug.
K.5.17.6 CTM Cannot Delete the Created Topology
Step 1 Launch the L2 Topology table and check if the topology is in Deleting state. If the topology is not in Deleting state, there is a problem with topology the deletion logic in CTM.
Step 2 Check the state of related Layer 1 circuits. They should also be in deleting state. If they are in deleting state but are not eventually deleted, there is a problem related to Layer 1 circuit provisioning.
Step 3 If not all the circuits are deleted and they are not in deleting state, use CTC to delete the circuits.
Step 4 After the circuits are deleted, if the topology is still not deleted, do the following to enable the trace logs for data service and check if the circuit deletion events are received properly:
a. In the Domain Explorer window, choose Administration > Control Panel.
b. In the Control Panel window, click the NE Service to open the NE service pane.
c. Click the NE Manual Backup tab.
d. Select Trace as the error level.
e. Click Save.
f. On the CTM server, telnet the localhost. This starts the data log collection in the server. Check if the circuit deletion events are received properly. If they are not received properly, there is a problem with Layer 1 circuit provisioning. If they are received properly, there is a problem with the topology deletion logic in CTM. Explicitly delete the topology from the L2 Topology table.
Step 5 If the problem persists, contact the TAC.
K.5.18 Common VLAN Problems
K.5.18.1 VLAN Is Not Discovered
This error occurs when the VLAN discovery process fails. Do the following:
Step 1 Telnet into the cards and check if the VLAN configuration parameter set has any discrepancy with the database or if the configuration is corrupt.
Step 2 Mark the NE service related to the NE as Out of Service and then mark it as In Service.
Step 3 Mark the NE related to the topology as Out of Service and then mark it as In Service.
Step 4 Launch the NE Explorer of the related NE and do a force resync by clicking the Refresh button.
Step 5 If the problem persists, contact the TAC.
K.5.18.2 VLAN Is Not Created or Deleted
Do the following, depending on the cause of the problem:
•If the CTM server is resynchronizing, wait for some time before creating or deleting the VLAN again.
•If you receive an error message that says you cannot create a bridge group, you might have entered invalid AV pair commands in the CLI. Check the IosTransportModule.log for invalid commands. Also check the server logs for exceptions.
K.5.19 CRS-1 and XR 12000 PM Collection Issues
The CRS-1 and XR 12000 NEs collect and dump data to a binary file on a TFTP server at intervals configured by the user of the IOS-XR CLI. After dumping the performance data, the CRS-1 or XR 12000 NE sends a notification indicating the location of the file. This notification is logged in the Alarm Log every 15 minutes. On receiving the notification, CTM collects the performance data from the CRS-1 or XR 12000 NE. You should configure the CRS-1 and XR 12000 NEs to collect data every 15 minutes.
If CRS-1 or XR 12000 PM collection issues occur:
Step 1 Check if the TFTP server and directory have been configured in the running configuration.
Step 2 Check whether the proper PM configuration is in the running configuration on the router.
Step 3 In the Alarm Log, check the notifications that the CRS-1 sends.
K.5.20 PM Data Collection Failed
If PM data collection failed, collect PM data directly from the NE using a browser. Enter one of the following:
•Current PM data:
–For 15-minute collection, use http://<NE IP address>/pm/15min/1
–For 1-day collection, use http://<NE IP address>/pm/day/1
•Historical PM data:
–For 15-minute collection, use http://<NE IP address>/pm/15min/bucket#
–For 1-day collection, use http://<NE IP address>/pm/day/bucket#
where bucket# = (current time - missed historical time) / 900000 + 1
K.5.21 How Do I Collect Solaris Client Thread Dump?
Step 1 Open a terminal session.
Step 2 Run the client using the ctmc.sh script. The ctmc.sh script is located in the /opt/CiscoTransportManagerClient directory.
Step 3 Enter the following command to obtain the client java process ID:
Step 4 Enter the following command:
PID is the client java process ID.
K.5.22 How Do I Collect Windows Client Thread Dump?
To collect thread dumps for CTM R5.0.2 and earlier, do the following:
Step 1 Open a DOS window.
Step 2 Run the ctmc.bat file located under the C:\Cisco\TransportManagerClient directory.
Step 3 Generate the thread dump by holding down the Ctrl key and pressing the Break (Pause) key.
Step 4 Copy and paste the output to a text file.
To collect thread dumps for CTM R6.0 and later, do the following:
Step 1 Open a DOS window.
Step 2 Depending on your network size, enter one of the following commands:
•ctm-small-debug.exe
•ctm-medium-debug.exe
•ctm-large-debug.exe
•ctm-highend-debug.exe
Step 3 Generate the thread dump by holding down the Ctrl key and pressing the Break (Pause) key.
Step 4 Copy and paste the output to a text file.
K.5.23 How Do I Collect Server Thread Dumps?
Thread dumps are helpful references when debugging the CTM process. To collect thread dumps:
Step 1 Log into the server workstation as the root user.
Step 2 On the command line, enter the following command:
thread_dumper [\<Group/Service>\]
where:
•Group is the group name for which to collect thread dumps. It can be:
–SM
–SNMPTRAP
–NE
–PM
–GW
•Service is the service name for which the thread dump is required. It can be:
–SMService
–SNMPTrapService
–CORBAGWService
–ONS15216NEService
–ONS15302NEService
–ONS15305NEService
–ONS15310NEService
–ONS15327NEService
–ONS15454NEService
–ONS15454SDHNEService
–ONS15501NEService
–ONS15530NEService
–ONS15540NEService
–ONS15600NEService
–ONS15600SDHNEService
–ONS15800NEService
–ONS15801NEService
–ONS15808NEService
–UnmanagedNEService
–ONS15216PMService
–ONS15302PMService
–ONS15305PMService
–ONS15310PMService
–ONS15327PMService
–ONS15454PMService
–ONS15454SDHPMService
–ONS15501PMService
–ONS15530PMService
–ONS15540PMService
–ONS15600PMService
–ONS15600SDHPMService
–ONS15800PMService
–ONS15801PMService
–ONS15808PMService
K.5.24 How Do I Enable or Disable the Automatic Refresh Data Action?
The automatic Refresh Data feature automatically refreshes all data being displayed by CTM. When automatic refresh is enabled, you receive the following prompt:
Refresh Data action suggested. This action will result in closing all windows and might
take some time. Do you want to continue? {Yes | No}
In an unstable environment where NEs are synchronizing or NEs change their operational state frequently, you might receive the preceding prompt continuously. To disable the prompt:
Step 1 In the Domain Explorer window, choose Edit > User Preferences. The User Preferences dialog box opens.
Step 2 Click the Miscellaneous tab.
Step 3 To disable the prompt, uncheck the Enable Refresh Data Timer check box. (To enable the prompt, leave the check box checked.)
Step 4 Click OK.
K.5.25 How Do I Replace the Alarm Interface Panel?
The Alarm Interface Panel (AIP) provides surge protection for CTC-based NEs. This panel has a nonvolatile memory chip that stores the unique node address known as the MAC address. The MAC address identifies the nodes that support circuits. It allows CTM to determine circuit sources, destinations, and spans.
If an AIP fails, an alarm will be generated and the LCD display on the fan-try assemblies of the NEs will go blank. To perform an in-service replacement of the AIP, you must contact Cisco technical support. For contact information, go to the technical support website at http://www.cisco.com/tac.
You can replace the AIP on an in-service system without affecting traffic by using the circuit repair feature. See 7.2.11 Repairing a Circuit, page 7-115.
K.6 Client Debug Messages
On Windows, you can start the CTM client with a console window that displays exceptions and client debug messages. You can use these messages to help troubleshoot any client problems.
Step 1 Depending on your network size, enter one of the following commands to start the client with a console window:
/bin/ctm-medium-debug.exe
/bin/ctm-highend-debug.exe
Step 2 When the CTM client exits, the console window closes. If you want to capture and save the messages when the client exits, you must redirect the output to a file. In this case, the console will not be displayed. To save exceptions to a file, open one of the following files that corresponds to your network size:
•/bin/ctm-small-debug.lax
•/bin/ctm-medium-debug.lax
•/bin/ctm-large-debug.lax
•/bin/ctm-highend-debug.lax
Note It is strongly recommended that you save a copy of the original .lax file before modifying it.
Step 3 Using a text editor, open the appropriate .lax file and change the following lines by replacing "console" with a filename:
lax.stderr.redirect=console
lax.stdout.redirect=console
When specifying filenames, make sure to use escaped backslashes (such as c:\\myfolder\\client.log). For example, to save messages to the C:\myfolder\client.log file, change the lines to read:
lax.stderr.redirect=c:\\myfolder\\client.log
lax.stdout.redirect=c:\\myfolder\\client.log
Step 4 After modifying the .lax file, enter one of the following commands to launch the CTM client:
/bin/ctm-medium-debug.exe
/bin/ctm-highend-debug.exe
On Solaris, you can start the CTM client with a console window that displays exceptions and client debug messages. You can use these messages to help troubleshoot any client problems.
Depending on your network size, enter one of the following commands to start the client with a console window:
K.7 CTM GateWay/TL1 Problems
If the OSS cannot connect to CTM GateWay/TL1, or if the OSS does not receive a response, check the cables, address bindings, and configuration.
Step 1 If the connection times out, check the cables and connections.
Step 2 Verify that the address bindings between the OSS and the Solaris workstation running the CTM GateWay/TL1 service are correct.
Step 3 Using the ping command, verify connectivity between the OSS and the Solaris workstation running the CTM GateWay/TL1 service.
Step 4 Verify that the TCP/IP socket connection is still open for the OSS-to-CTM GateWay/TL1 port.
Step 5 Choose Administration > Control Panel. Click GateWay/TL1 Service and confirm that the service status is Active.
Step 6 Verify that the username and password under Control Panel > Security Properties match the user configuration of the NEs that CTM GateWay/TL1 will manage.
Step 7 Verify that the username and password in the Domain Explorer > Network Element Properties > NE Authentication tab match the user configuration of the NEs that CTM GateWay/TL1 will manage.
K.8 CTM GateWay/CORBA Problems
If the CTM server is installed without the CTM GateWay/CORBA option, and the CTM GateWay/CORBA option is installed later, the Control Panel does not refresh automatically after the CORBA installation to show that CTM GateWay/CORBA is installed.
This problem occurs because the CTM GateWay/CORBA installation is handled by a script that notifies the database that the component is installed, but does not notify the CTM server.
To work around this problem, click the Refresh Data button in the Control Panel. It might take some time for the Control Panel to show that CTM GateWay/CORBA is installed. You might need to close and reopen the Control Panel to see the change.
K.9 Cisco 7600 Problems
This section describes the troubleshooting procedures for problems related to the Cisco 7600.
K.9.1 Cisco 7600 NE Not In Service and Available
If you have added a new Cisco 7600 NE, the NE may not be marked as In Service and Available. To work around this problem, do the following:
Step 1 Verify that CTM server hostname-to-IP mapping is configured on the NE; on the NE, run the following command:
show running | include ip
If the mapping does not exist, run the following command:
ip host <CTM server hostname> <CTM server IP address>
Step 2 Verify that the SNMP community string is configured on the NE; on the NE, run the following command:
show running | include snmp
If the community is not configured, run the following command:
snmp-server community <community name> RW
where <community name> is the same value as the community string specified in the Network Elements Properties pane (see 1.3.2.3 Network Element Properties, page 1-13).
Step 3 Verify that NE login and enable privilege-mode authentication information is configured on the CTM server.
Note You can configure the default NE login and enable password in the Security Properties pane; from the Domain Explorer, select Administration > Control Panel; then, click Security Properties and select the IOS 7600 tab. See 8.3.4.14 Configuring CTM Security Parameters, page 8-35 for more information.
a. If the NE password in the Security Properties pane does not match the password configured on the NE, configure the password on the NE Authentication tab in Network Element Properties pane (see 1.3.2.3.4 NE Authentication Tab, page 1-17 for more information).
b. After you configure the password in the NE Authentication tab, set the operational state of the NE to Out of Service, then set it as In Service. Select Administration > Control Panel, then click NE Service and select the correct IOS 7600 service instance (see 4.5.5 Viewing and Modifying Service Instance Properties, page 4-80 for more information). In the NE Service Instance pane, click the Stop Service Instance button to stop the NE service. Wait for several seconds, then click the Start Service Instance button to start the NE service.
Note Information configured through the NE Properties pane overwrites information configured in the Control Panel.
K.9.2 NE Hangs in the InService State and May Later Become Unavailable
An NE may hang in the In Service state and become unavailable because of sync-configuration error. Do the following:
Step 1 Check if the health poll has failed. You can do this by verifying IP connectivity; enter the ping <NE IP address> command. There should be an IP connection between the CTM server and NE.
Step 2 Check if one or more configuration upload requests timed out. You can do this by checking for a timeout exception (by searching, for example, for the Timeout Expired string) in the /opt/CiscoTransportManagerServer/log/IOS7600NEService-<id>-<time>.log file.
If there are many timeout exceptions, and the NE is IP-reachable, the network may be experiencing heavy traffic and may need more time for a CLI command to respond.
Step 3 Increase various timeout values for various parameters in the /opt/CiscoTransportManagerServer/cfg/csm.properties file; however, increasing the values decreases the overall system performance.
You can modify the following parameters:
•# config reply timeout in second for each retry
Note Increasing this value ensures that configuration update requests wait longer before timing out.
•# image reply timeout in second for each retry
Note Increasing this value ensures that image update request wait longer before timing out.
•# number of retries
REPLY_RETRIES=20
•# number of retries for get running config
GET_RUN_RETRIES=2
Note Increase this value if the NE is stuck in sync-configuration mode because a running configuration is uploading.
•# http reply timeout in second
HTTP_TIMEOUT=1800000
•# telnet timeout in second
TELNET_TIMEOUT=120
•# time in seconds CTM waits after creating a device
WAIT_AFTER_CREATE=20
K.9.3 NE Changes to Loss of Communication State
An NE may change or continuously change to Loss of Communication (LOC) state after an NE configuration change. The problem may be due to one of the following:
•The health poll failed. See NE Hangs in the InService State and May Later Become Unavailable for more information.
•One or more configuration upload requests timed out. See NE Hangs in the InService State and May Later Become Unavailable for more information.
•The CTM server may not have the adequate memory to run the server. Verify CTM server requirements; see the Installation Guide for Cisco Transport Manager Release 7.2 for more information.
K.9.4 Configuration Fails and Configuration Jobs Take Longer to Complete than the Preconfigured Time
Step 1 Check if the event pipe is broken for CNS Agent- enabled devices. To verify the CNS event connectivity, enter the following command on the NE:
show cns event connection
The event hostname should be the CTM server hostname, and the status should be "Connection Established."
Step 2 Verify that the CTM server hostname is configured properly.
The CTM server hostname is configured in the /etc/hosts file. For example, a valid entry would be:
<CTM_server_IP_address><hostname>
If the /etc/hosts file also contains a .cisco.com entry, verify that the preceding example matches the output of the UNIX hostname command that is executed on the CTM server.
The specified hostname should also be configured in the DNS server.
Check your DNS setup for the server. If you do a lookup on the hostname on DNS, the hostname and IP address displayed should resolve to the same CTM server.
Step 3 Increase the configuration request timeout value.
If the configuration request times out and the NE is IP-reachable, the network may be experiencing heavy traffic and may require more time for a CLI command to respond. In the the/opt/CiscoTransportManagerServer/cfg/csm.properties file, increase the timeout value so that configuration update requests wait longer before timing out. You can do this by modifying the following parameter:
# config reply timeout in second for each retry
K.9.5 Configuration Completes with a Warning
If configuration of a Cisco 7600 NE completes, but there are warnings, the NE may be partially configured. You should synchronize the configuration.
K.9.6 BGP Neighbor and FTP Passwords Cannot Use Some Special Embedded Characters
You can use only the following characters: & < > ! ? . @ # $ % ^ ~ \n \r + - = | * ( ) { } [ ]
K.9.7 CTM Does Not Discover Connected Cisco 7609 Devices
Step 1 Run the following commands:
CTM uses BGP and CDP to discover 7600 NEs. Verify that the output of the show cdp neighbor and show bgp neighbor commands show the connected devices. Also, verify that the IP address returned by the commands is reachable from the CTM server machine.
Step 2 Verify that there is SNMP read access to the connected devices for the community strings that are configured on the devices.
K.9.8 The IP Address of Discovered Device Is Not the Managed IP Address
If CTM discovers a device, but its IP address is not the managed IP address, it could be because CTM first retrieves the hostname of the connected devices by querying the CDP/BGP protocol information from the device. CTM then uses this hostname and performs a DNS lookup to obtain an IP address; if DNS does not return an IP address, CTM uses the IP address specified in the protocol information that is returned.
To work around the problem, verify that there is a DNS entry that contains the hostname of the connected device and its management IP address.
K.9.9 CTM Server Does Not Receive Traps
Do the following:
•If the CTM server does not receive traps for a Cisco 7600 NE, verify that SNMP is configured on the NE by running the following command:
sh run | i snmp-server
If the snmp-server is not configured on the NE, run the following commands:
config t
snmp-server host <CTM server IP address> version 2c <community string used by CTM>
•If the CTM server does not receive traps for all Cisco 7600 NEs, verify that SNMP is configured on all NEs by running the following commands:
sh run | i snmp
snmp-server enable traps snmp linkdown linkup
snmp-server enable traps module
snmp-server enable traps syslog
K.9.10 Software Download, Memory Backup, and Memory Restore Failure
Step 1 From the Domain Explorer, select Administration > Control Panel.
Step 2 In the Control Panel, click Security Properties, then select the IOS 7600 tab.
Step 3 In the CTM Server - FTP Connection Username field, enter the username that the CTM server uses on the FTP connection.
Step 4 In the Password field, enter the password that the CTM server uses on the FTP connection.
Step 5 Restart the NE service. In the Control Panel, select NE Service and select the correct IOS 7600 service instance. In the NE Service Instance pane, click the Stop Service Instance button to stop the NE service. Wait for several seconds, then click the Start Service Instance button to start the NE service.
K.9.11 Syslog Viewer Does Not Display Faults
Step 1 Run the following command:
Output similar to the following should be displayed:
logging <CTM server IP address>
If similar output is not displayed, configure this parameter.
Step 2 Verify that the following line appears in the /etc/syslog.conf file:
Step 3 Restart the syslog daemon; on the CTM server, enter the following command:
K.9.12 Startup Configuration Backup Fails
If backing up the startup configuration fails, and you receive %Device or resource busy errors, do the following:
Step 1 Verify the following:
•Only one CTM server exists within a subnet.
•If there are multiple CTM servers, one CTM server manages the NE through CNS Agent-enabled mode, and another CTM server manages the NE through the IMGW mode.
Step 2 Verify that the CTM server hostname is configured in the /etc/hosts file. A valid entry should look similar to the following:
<CTM server IP address><hosttname>
K.9.13 Software Activation on the Cisco 7600 NE Fails
Step 1 Verify that the Config Register value has a nonzero value on the NE; on the NE, run the following command:
The last line of the command output should display a nonzero value for the Config Register; for example:
Configuration register is 0x2
Step 2 If the value of the Config Register is 0x0, change it by entering the following commands on the NE:
K.9.14 Restoring a Running Configuration Fails
When you perform a configuration restore of a running configuration, you are attempting to add configuration commands that already exist, and the operation fails. You can only add new commands to the existing configuration.
K.9.15 Using the Correct CTM Debug Module
There are several CTM debug modules you can use to diagnose a problem. To select the correct module, do the following:
Step 1 From the Domain Explorer, select Administration > Control Panel.
Step 2 Click NE Service, then select IOS 7600.
Step 3 Click the Debug tab. See Table 4-26 for more information on the NE Service pane.
Step 4 In the Overall Logging area, select the Enable radio button to enable debugging. You can then select from a list of available debug modules in the Debug Modules area.
Step 5 You can select the modules to debug when a problem occurs. For example:
•7600 CSM—This module provides transportation layer debugging information for the Cisco 7600 NE Service.
•Initial Poll—This module provides logs when an NE moves to the In Service state from other states during an initial poll.
K.9.16 Collection and Location of Logs
CTM logs are located in the /opt/CiscoTransportManagerServer/log directory.
Other logs include:
•IMGW Requests Log located in the /var/log/CNSCE/imgw directory
•CNS Agent Log, Exec Requests Log, and Event Log located in the /var/log/CNSCE/cfgsrv directory
•Image Download Log, located in the /var/log/CNSCE/imgsrv directory
K.9.17 Enabling Detailed CE Debugging
Do one of the following:
•In your browser, type http://<CTM Server IP address>, then enter the CE username and password configured during CTM server installation.
Then, from the GUI, select Tools > Log Manager > Change log level > Debug > Submit.
Note Making these changes in the GUI does not require a restart of the CE; however, after a CTM restart, any changes you made to the CE settings are lost.
•To set the CE debugging level and restart the CE, enter the following command:
/opt/CiscoTransportManagerServer/ConfigEngine/CSCOcnsie/bin/setup
This command causes the NEs managed on the CTM server to become temporarily out of service. By default, the logging level is error, and the most detailed level of debugging is verbose. Configure the logging level information in the IMGW setup portion of the procedure.
K.10 Cisco CRS-1 and Catalyst 6509 Problems
This section describes troubleshooting procedures for problems related to the Cisco CRS-1 and the Catalyst 6509.
K.10.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service
Step 1 Verify that the CRS-1 is configured with "hfrems" as the username and "hfrems" as the password with "root-system" privileges. By default, CTM expects "hfrems" as both the username and password. You can change the username and password in the Control Panel > Security Properties > CRS-1 tab.
Step 2 Enter the following commands to verify that the XML CORBA agent service is running:
Router(config)# xml agent corba [ssl]
Note SSL is optional and enables SSL CORBA services.
Step 3 If SSL CORBA services are running, enter the following commands to verify that HTTP over SSL is running:
Router(config)# http server ssl
Step 4 The CORBA agent on the router accepts connectivity only when the connection is done through hostname. Verify that the device IP address has an entry in DNS. If not, add an entry to the /etc/hosts file.
Step 5 Verify that CTM supports the software image that is running on the router. In the Domain Explorer, choose Administration > Supported NE Table and verify that the software version is supported.
Step 6 When the CRS-1 is added to CTM, the NE type is retrieved from the NE to verify that the NE is a CRS-1. If the device does not report CRS-1 as the NE type, CTM reports a loss of communication (LOC) to the NE and raises an NE type mismatch alarm.
Step 7 Verify that the NE hostname specified in DNS or in the CTM server's /etc/hosts file matches the hostname configured on the NE. If the hostnames do not match, CTM reports an LOC to the NE.
K.10.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service
Step 1 Verify that the IP address entered in CTM is configured on the Catalyst 6509. Enter the following command to configure the IP address:
cat6509 (enable)# set interface <logical_interface_name_like_sc0/sc1>
<IP_address_of_the_Catalyst_6509> <mask>
Step 2 Enter the following command to verify that the device is configured correctly with the default route:
cat6509 (enable)# set ip route default <gateway_IP_address>
Step 3 Verify that CTM supports the software image that is running on the Catalyst 6509. In the Domain Explorer, choose Administration > Supported NE Table and verify that the software version is supported.
Note For the Catalyst 6509, the supported software version is Release 6.3(7) or later.
Step 4 Verify that the community string shown in the Domain Explorer > Address tab is same as the community string on the device. If there is a mismatch, enter the following command to change the SNMP community string on the device:
cat6509 (enable)# set snmp community read-only <community_string>
For example:
cat6509 (enable)# set snmp community read-only public
K.10.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1
Step 1 Enter the following commands to verify that the CTM server hostname and "logging-events-level" are configured in the running configuration:
Router# sh run | include domain ipv4 host
Router# sh run | include logging events level informational
Step 2 If the CTM server hostname and "logging-events-level" are not configured in the running configuration, enter the following commands:
Router(config)# domain ipv4 host <CTM_server_hostname> <IP_address>
Router(config)# logging events level informational
Step 3 If xml agent corba is configured on the router, make sure that the router and the CTM server machine are reachable by a fully qualified hostname, using the ping command.
Step 4 If the CTM server machine is a dual-home machine, in order for the router to receive configuration change notifications, it has to be configured with the IP address used by the CTM server machine to communicate with the router. For example, the CTM server machine is a dual-home machine with the following hostname and IP address:
ctmserver1-other <10.x.x.x> (IP address used to communicate with the router
The router needs to have the following configuration:
domain ipv4 host ctmserver1 <10.x.x.x>
K.10.4 CTM Does Not Receive Traps from the Catalyst 6509
Step 1 Enter the following command to configure the CTM server as the trap receiver on the Catalyst 6509:
cat6509 (enable)# set snmp trap <CTM_server_IP_address> <community_string>
Step 2 Enter the following command to configure the Catalyst 6509 to enable only specific traps:
cat6509 (enable)# set snmp trap enable <specific_trap>
where <specific_trap> is auth, bridge, chassis, config, entity, ippermit, module, stpx, syslog, systemp, vmps, or vtp.
Step 3 Enter the following command to configure the Catalyst 6509 to enable all traps:
cat6509 (enable)# set snmp trap enable all
Note The set snmp trap enable all command enables traps only on physical ports. For logical ports such as sc0 or sc1, you must enable traps separately. Go to Step 4.
Step 4 Enter the following command to enable traps on logical ports:
cat6509 (enable)# set interface trap <logical_port_name> enable
For example, to enable traps on the logical interface sc0, enter:
cat6509 (enable)# set interface trap sc0 enable
K.10.5 CTM Does Not Receive syslog Events from the Catalyst 6509
The Catalyst 6509 sends syslog traps only if the trap severity is both:
•Less than or equal to facility severity (which is set with the set logging level command)
•Less than or equal to history severity (which is set with the set logging history severity command)
It is recommended that you set the history severity to debugging (7).
The show logging command shows both the history severity and the severity for all facilities. For example, in the following output, the history severity is 7 and facility severity is 5. Syslog events of severity X will be triggered from the device if X is less than or equal to 5 and less than or equal to 7.
Cat6509 (enable)# show logging
timestamp option: enabled
{10.77.56.151, 172.19.75.86}
server severity: debugging(7)
Current Logging Session: enabled
Callhome Functionality: disabled
Callhome Severity: LOG_CRIT(2)
SMTP servers are not configured for callhome messages.
Destination addresses are not configured for callhome messages.
From Address is not configured for callhome messages.
Reply-To Address is not configured for callhome messages.
Facility Default Severity Current Session Severity
------------- ----------------------- ------------------------
0(emergencies) 1(alerts) 2(critical)
3(errors) 4(warnings) 5(notifications)
6(information) 7(debugging)
K.10.6 CTM Does Not Collect Historical PM Data for the CRS-1
Step 1 Verify that PM collection is enabled for the CRS-1. To do this, go to the Network Element Properties > Status tab and in the PM Collection area, check the 15 Min check box. Click Save. Also, verify that the CRS-1 PM Service Status is Active in the Control Panel window.
Step 2 Enter the following commands to verify that the correct PM configuration is in the running configuration on the CRS-1:
Router(config)# show run perf
performance-mgmt resources tftp-server <IP_address_of_server> directory
<directory_on_TFTP_server>
performance-mgmt statistics bgp template default
performance-mgmt statistics mpls ldp template default
performance-mgmt statistics node cpu template default
performance-mgmt statistics interface generic-counters template default
performance-mgmt statistics interface data-rates template default
performance-mgmt statistics mpls TE-link template default
performance-mgmt statistics node memory template default
performance-mgmt statistics node process template default
performance-mgmt statistics mpls TE-tunnel template default
performance-mgmt statistics mpls interface template default
Step 3 If the files are not being dumped to the target directory on the TFTP server, verify that the TFTP server is using the correct tftpd (tftp daemon binary). The tftpd should be a version that allows the CRS-1 to create new files in the specified directory. The default tftpd only allows pre-existing TFTP files to be written. The default tftpd does not create new TFTP files.
Step 4 If the files appear on the TFTP server but do not appear in the PM table, verify that the CTM server is being notified of PM TFTP file dumping events. To do this, choose Fault > Alarm Log and check for events that arrive every 15 minutes and resemble the following output:
pm_collector[301]: %PM_COLLECTOR-6-TFTP_SEND : Successfully dumped
file:tftp://<IP_address><TFTP_directory><NE_name>_node_process_1100886887.212000
Step 5 If events do not appear after 15 minutes, verify that configuration change notifications are enabled. See CTM Does Not Receive Configuration Changed Notifications from the CRS-1.
Step 6 If events are present, verify that the TFTP server is reachable from the CTM server. To do this, ping the IP address that is present in the event notification. If the IP address is not reachable, configure the routes so that it becomes reachable. Alternately, include a mapping file in the /opt/CiscoTransportManagerServer/cfg/hfr/IPAddressMapping.cfg directory that includes the following mapping:
<IP_address_in_event_notification> <reachable_IP_address_of_the_same_TFTP_server>
For example:
172.19.64.51 223.255.254.254
K.10.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval
You might experience a problem where the PM data collection interval is set to 15 minutes, but the CRS-1 PM tables are collecting data more frequently (or less frequently).
The CRS-1 collects and dumps data to a binary file on a TFTP server at user-configured intervals based on the IOS-XR CLI. After dumping the PM data, the CRS-1 notifies CTM with the location of the file. CTM then collects PM data from the CRS-1. Therefore, you should configure the CRS-1 to collect data every 15 minutes.
K.10.8 Memory Backup or Restore Fails for the CRS-1
Step 1 Verify that the NE is in service and the communication state is Available.
Step 2 Verify that you can ping the CTM server from the NE. The NE is used as a TFTP client for downloading the configuration file.
Step 3 In the /etc/inetd.conf file, verify that the TFTP server is enabled on the CTM server. Also, verify that the specified tftpboot directory has the correct permissions.
K.10.9 How Do I Back Up the CRS-1 Configuration File Automatically Whenever the Configuration Is Changed on the Router?
Step 1 In the Domain Explorer, choose Administration > Control Panel.
Step 2 Click NE Service.
Step 3 In the NE AutoBackup tab, choose Cisco CRS-1 as the NE model. Alternately, choose All NE Models.
Step 4 Check the Enable Auto Backup check box.
Step 5 Specify the number of backup copies and the backup time.
Step 6 Click Save.
K.10.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs
Step 1 Verify that there is a physical link between the two NEs.
Step 2 Complete one of the following options, depending on your configuration:
•If the link is an Ethernet link, bring up the interfaces on both NEs. For example, enter the following commands on the first router:
Router-1(config)# int GigabitEthernet0/2/0/0
Router-1(config-if) ipv4 address <IP_address> <mask>
Router-1(config-if) no shutdown
Router-1(config-if) commit
Enter the following commands on the second router:
Router-2(config)# int GigabitEthernet0/1/0/0
Router-2(config-if) ipv4 address <IP_address> <mask>
Router-2(config-if) no shutdown
Router-2(config-if) commit
•If the link is a POS link, enter the following commands on the first router:
Router-1(config)# int POS 0/2/0/0
Router-1(config-if) ipv4 address <IP_address> <mask>
Router-1(config-if) no shutdown
Router-1(config-if) commit
Enter the following commands on the second router:
Router-2(config)# int POS 0/1/0/0
Router-2(config-if) ipv4 address <IP_address> <mask>
Router-2(config-if) no shutdown
Router-2(config-if) commit
Verify that the SONET controllers are up. For example, enter the following commands:
Router-1(config)# controller SONET 0/2/0/0
Router-1(config-sonet)# no shutdown
Router-1(config-sonet)# commit
Router-2(config)# controller SONET 0/1/0/0
Router-2(config-sonet)# no shutdown
Router-2(config-sonet)# commit
Step 3 Enter the following commands to enable CDP on both NEs:
K.10.11 Cannot Delete an Out-of-Service NE from CTM
If you cannot delete an out-of-service NE from CTM, follow the procedure in 3.5.7.3 Using the prune_ne.sh Script to Remove an Out-of-Service NE from the Database, page 3-30.
K.10.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty
If you cannot see the software version in the Domain Explorer > Identification tab and NE Software table is empty, complete the following steps:
Step 1 Verify that the NE is in service and the communication state is Available.
Step 2 Verify that the device is configured with either Telnet or SSH.
K.10.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer
If you cannot use Configuration > Telnet/SSH from the CRS-1 NE Explorer, try pinging the NE from the CTM client. You might be able to reach the NE from the CTM server but not from the client if they are in different subnets.
K.10.14 How Do I Configure Telnet on the CRS-1?
To configure Telnet on the CRS-1, log into the router through the console and enter the following commands:
Router(config)# telnet ipv4 server
K.10.15 How Do I Configure SSH on the CRS-1?
Step 1 Enter the following command to verify that the security PIE(k9) is installed and activate on the router. The PIE(k9) is required for encryption, decryption, IPSec, SSH, SSL, and PKI:
Router# sh install active
Step 2 Enter the following command to verify that DSA keys have been generated on the router:
Router# crypto key generate dsa
Step 3 Enter the following command to verify that RSA keys have been generated on the router:
Router# crypto key generate rsa
Step 4 Enter the following command to verify that SSH is enabled on the router:
Router# crypto key generate rsa
Step 5 Enter the following command to enable the SSH server on the router:
Router(config)# ssh server enable
K.10.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs?
See 5.6.28 Configuring Secure Socket Layer for the CRS-1 and XR 12000, page 5-299.
K.10.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509?
Complete one of the following options:
•To trigger a linkDown trap from the Catalyst 6509, enter the following command:
cat6509 (enable)# set port disable <module_number/port_number>
For example:
cat6509 (enable)# set port disable 5/1
•To trigger a linkUp trap from the Catalyst 6509, enter the following command:
cat6509 (enable)# set port enable <module_number/port_number>
For example:
cat6509 (enable)# set port enable 5/1
K.10.18 How Do I View the SNMP Configuration on the Catalyst 6509?
To view the SNMP configuration on the Catalyst 6509, enter the following command:
cat6509 (enable)# show snmp
K.10.19 How Do I Load a Software Image on the Catalyst 6509?
Complete the following steps:
Step 1 Enter the following command to view all the software images on the Catalyst 6509:
cat6509 (enable)# dir bootflash:
The output is similar to the following example:
-#- -length- -----date/time------ name
2 5849256 Aug 09 2004 14:41:25 cat6000-sup2.6-3-7.bin
3 8801412 Aug 09 2004 15:30:57 cat6000-sup2k8.8-3-2.bin
4 5713736 Aug 11 2004 13:08:02 cat6000-sup2.6-3-2a.bin
5 8182352 Aug 12 2004 09:15:00 cat6000-sup2k8.8-2-1.bin
17 1786 Aug 23 2004 14:16:30 6509svt-171-180
19 1688 Sep 24 2004 15:26:40 6509svt-171-180.cfg
3346628 bytes available (28634940 bytes used)
Step 2 Enter the following command to retrieve the image name that is currently on the device:
cat6509 (enable)# whichboot
The output is similar to the following example:
Boot image name is 'bootflash:cat6000-sup2.6-3-7.bin'.
Step 3 Enter the following command to delete the existing image:
cat6509 (enable)# del <image_name>
Step 4 Enter the following command to remove the deleted image:
cat6509 (enable)# squeeze bootflash:
Step 5 Enter the following commands to copy the image to flash memory:
cat6509 (enable)# copy tftp flash
IP address or name of remotehost[]? <TFTP_server_IP_address>
Name of file to copy from []? <Full_path_name_and_filename_of_the_image_location>
For example:
tftpboot/cat6k/7-2/<image_name>
Step 6 Enter the following command to verify that the downloaded image exists on the device:
cat6509 (enable)# dir bootflash:
Step 7 Enter the following command to verify that the download was successful:
cat6509 (enable)# verify <image_name>
Step 8 Enter the following command to clear the flash system:
cat6509 (enable)# clear boot system flash bootflash:<previous_image_name>
Step 9 Enter the following command to set the boot system flash with the new image name:
cat6509 (enable)# set boot system flash bootflash:<new_image_name> prepend
Step 10 Enter the following command to reset the device:
K.11 XR 12000 Problems
This section describes troubleshooting procedures for problems related to the XR 12000.
K.11.1 Reporting XR 12000 Issues
When reporting an XR 12000 problem, include detailed steps to reproduce the problem and the software version(s) of the NE(s) to which the problem applies. Also, enter the following router CLI commands on the XR 12000 and collect the data:
show ipv4 interface brief
show log events buffer all-in-buffer
Turn on trace level debugging in the Control Panel for the service in question, wait until the problem occurs, and collect the GSRIOXNEService logs from /opt/CiscoTransportManagerServer/log.
Note For the CRS-1 and XR 12000 NEs, enter admin at the prompt before entering the show platform and show inventory commands to view the results in admin mode.
K.11.2 CTM Cannot Discover an XR 12000 Node
Complete the following steps if CTM cannot discover an XR 12000 node:
Step 1 Verify that the NE is configured with "hfrems" as the username and "hfrems" as the password with "root-system" privileges. By default, CTM expects the username and password to be hfrems/hfrems. You can change the security properties from the Control Panel > Security Properties pane > XR 12000 tab. CTM will use the new username and password to authenticate all XR 12000 devices in the domain.
Step 2 Check whether the XML CORBA agent service is running. To configure the XML CORBA agent, enter:
Router(config)# xml agent corba [ssl]
Note ssl is optional and enables SSL CORBA services.
If SSL CORBA services are activated, enter the following commands to check whether HTTP over SSL is running:
Router(config)# http server ssl
Step 3 The CORBA agent on the router accepts connections only when using the hostname. Check to see if the device IP address has an entry in DNS. If DNS is not used, add an entry in the /etc/hosts file.
Step 4 Verify that CTM supports the software version that is running on the router. The CTM release notes list the CTM-supported NE software versions. Go to http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/optnet/ctm/ctmreln/index.htm.
Step 5 When an NE is added to CTM, the NE type is retrieved to verify whether the NE is an XR 12000. If the device is not of NE type XR 12000, the NE is moved to an LOC state in CTM and an NE type mismatch alarm is raised.
Step 6 The NE hostname specified in DNS or in the CTM server /etc/hosts file must match the hostname configured on the NE. If the hostnames do not match, the NE is moved to an LOC state in CTM.
K.11.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer
If the LAN cable is removed, there is a delay before the status of the XR 12000 is updated in the Domain Explorer. It might take several minutes or more to reflect the status as unreachable if the router is disconnected.
If you want the status to be updated sooner, you must modify the jacorb.connection.client.pending_reply_timeout parameter in the /opt/CiscoTransportManagerServer/openfusion/classes/jacorb.properties file. The default timeout is 5 minutes (300000 ms). Updates take approximately the health poll interval plus two times the jacorb.connection.client.pending_reply_timeout value.
K.11.4 XR 12000 Alarm and Event Reporting Issues
If CTM does not receive XR 12000 alarm and event notifications, complete the following steps:
Step 1 Enter the following commands to verify that the CTM server hostname and "logging-events-level" are configured in the running configuration:
Router# sh run | include domain ipv4 host
Router# sh run | include logging events level informational
Step 2 If the CTM server hostname and "logging-events-level" are not configured in the running configuration, enter the following commands:
Router(config)# domain ipv4 host <CTM_server_hostname> <IP_address>
Router(config)# logging events level informational
Step 3 If xml agent corba is configured on the router, make sure that the router and the CTM server machine are reachable by fully qualified hostname, using the ping command.
Step 4 If the CTM server machine is a dual-home machine, in order for the router to receive configuration change notifications, it has to be configured with the IP address used by the CTM server machine to communicate with the router. For example, the CTM server machine is a dual-home machine with the following hostname and IP address:
ctmserver1-other <10.x.x.x> (IP address used to communicate with the router
The router needs to have the following configuration:
domain ipv4 host ctmserver1 <10.x.x.x>
K.11.5 XR 12000 PM Collection Issues
The XR 12000 collects and dumps data to a binary file on a TFTP server at intervals configured by the user through the CLI. After dumping the performance data, the XR 12000 sends a notification indicating the location of the file. Upon receiving this notification, CTM collects performance data from the XR 12000. Therefore, the network operator should configure the XR 12000 to collect data every 15 minutes.
Step 1 Enter the following command to verify whether the TFTP server and directory are configured in the running configuration:
performance-mgmt resources tftp-server <server_IP_address> directory
<directory_on_TFTP_server>
Step 2 Enter the following commands to verify that the router's running configuration contains the correct PM configuration:
performance-mgmt statistics bgp template default
performance-mgmt statistics mpls ldp template default
performance-mgmt statistics node cpu template default
Step 3 After dumping the performance data, the XR 12000 sends a notification indicating the location of the file. This notification is logged in the Alarm Log every 15 minutes and triggers the CTM server connecting to the TFTP server. Check for notifications in the Alarm Log that look similar to the following:
pm_collector[301]: %PM_COLLECTOR-6-TFTP_SEND : Successfully dumped
file:tftp://<IP_address><TFTP_directory><NE_name>_node_process_1100886887.212000
K.11.6 XR 12000 BGP and CDP Neighbor Discovery Issues
CTM discovers all the CDP and BGP neighbor routers and adds them to the tree view if the CDP or BGP is enabled on the router. CDP and BGP neighbor discovery occurs after the NE is synchronized. If the neighbors are not discovered, enter the following CLI commands to check whether CDP or BGP is enabled on the router and interfaces:
K.11.7 XR 12000 NE Explorer Applications Are Disabled
The CTM NE Explorer enables only those applications for which the XML version matches on CTM and the router. If an NE Explorer application is disabled, there might be a version mismatch. Do the following:
•Collect all the XML files from the <CTM_home>/cfg/metadata directory.
•Run the <CTM_home>/bin/vwdata.sh script to collect the hfr_version_info database table entries.
On Windows, if an NE Explorer or GUI application returns an error, repeat the same operation after invoking the GUI by entering one of the following commands, depending on your network size:
<CTM_home>/bin/ctm-small-debug.exe
<CTM_home>/bin/ctm-medium-debug.exe
<CTM_home>/bin/ctm-large-debug.exe
This opens a console that logs all the exceptions. Capture the console log for debugging.
On UNIX, enter the <CTM_home>/bin/ctmcdebug-start command to start the client and repeat the same operation. This generates the error and exception messages on the terminal. Capture the terminal messages for debugging.
K.12 Problems with MGX Voice Gateway Devices
This section describes troubleshooting procedures for problems related to the MGX Voice Gateway devices.
K.12.1 Discovery Mechanism
CTM manages the Private Network-Network Interface (PNNI). The ILMITopoc process uses SNMP protocol to discover the physical PNNI network. All the discovered nodes are displayed in the MGX NE GUIs (Configuration Center, Diagnostics Center, Statistics Report, and Chassis View). CTM is notified of all subsequent changes in the network through traps for MGX nodes.
Once the ILMITopoc process discovers all the nodes, it sends all the nodes to the topod process. Topod then sends these nodes and trunks to other processes in CTM such as ooemc, NMServer, nts, and so on.
K.12.2 Discovery Issues at Startup
This section includes the following information:
•No Nodes Are Discovered
•Node Name Is Incorrect in the Database
•Node IP Is Incorrect in the Database
•Node Alarm Is Shown Incorrectly in the Database
•Reachable Node Is Shown as Unreachable
•CTM State Is Incorrect
K.12.2.1 No Nodes Are Discovered
If no nodes are in the CTM database, complete the following steps:
Step 1 Verify that the nodes are IP-reachable. Complete the following substeps:
a. Enter the following CLI commands to retrieve the primary IP address of the node:
b. Enter the following command to check whether CTM has IP connectivity to the node:
Step 2 If CTM has IP connectivity to the nodes, verify that the community strings of the gateway nodes are correct. Complete the following substeps:
a. Enter the following command on the switch CLI of the node that cannot be discovered:
b. Compare this community string with the community string in the node_info table. The community string in the node_info table is encrypted. You will need to decrypt this string and verify it.
Step 3 For MGX nodes, if persistent topology is enabled on the switch gateway and some nodes in a peer group are not discovered, complete the following substeps:
a. Enter the following CLI command to verify that persistent topology is enabled on the gateway in the peer group:
b. If the gateway flag is enabled, enter the following CLI command to verify that the node is part of the list of nodes on the gateway:
Defect information—Collect the following information for further analysis:
•Topod.log and ILMITopoc.log.
•Data in the node_info table.
•Output of dspndparms, dspsnmp, and dspipif on the MGX node.
K.12.2.2 Node Name Is Incorrect in the Database
If the node name of the standalone MGX node is incorrect, complete the following steps:
Step 1 If the node table contains the correct name, complete the following substeps:
a. Enter the following command to dump the cache of topod and ILMITopoc:
b. From the data collected, verify which process contains the incorrect information. If the information is correct in all processes, open a new instance of the GUI and verify whether the problem still exists.
Step 2 If the node table contains the incorrect name, check the ILMITopoc.log file to see from which node it collected the wrong information. Complete the following substeps:
a. Enter the following command:
snmpget -c <community> <IP_address> 1.3.6.1.2.1.1.5
b. Verify that the name received as a response is correct. If the name received is incorrect, verify the name on the switch.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.2.3 Node IP Is Incorrect in the Database
Step 1 If the node table contains the correct IP address, complete the following substeps:
a. Enter the following command to dump the cache of topod and ILMITopoc:
b. From the data collected, verify which process has the incorrect information. If the information is correct in all processes, open a new instance of the GUI and verify whether the problem still exists.
Step 2 If the node table does not have the IP address set correctly, verify whether the node is managed by the LAN IP address. In CTM, since the node does not have any PNNI trunks and the persistent topology feature is not enabled, the node is managed by the LAN IP address by default.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.2.4 Node Alarm Is Shown Incorrectly in the Database
Step 1 Enter the following command to dump the cache of topod and ILMITopoc:
Step 2 From the data collected, verify which process contains the incorrect information.
Step 3 If the node table does not have the node alarm status set correctly, check the ILMITopoc.log file and enter the following command:
snmpget -c <community string> <IP_address> 1.3.6.1.4.1.351.110.1.1.14
The snmpget command returns the alarm state of the node, where 1 = Clear, 2 = Minor, 3 = Major, and 4 = Critical.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.2.5 Reachable Node Is Shown as Unreachable
If a node that is reachable from CTM is shown as unreachable in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table reports the alarm state of this node as minor, major, critical, or clear, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /opt/svplus/log directory and verify whether the node alarm status is correct. If the alarm status is correct in the cache dump, open a new instance of the GUI and check whether the node shows the correct alarm status in that window.
Step 2 If the node table shows the alarm state of an MGX node as unreachable, complete the following substeps:
a. Verify that the node is IP-reachable from CTM.
b. Ping the active IP address of the node. The active IP address is the IP address of the node that is populated in the node table.
c. Enter the following CLI command to retrieve the community string of the node:
d. Verify that the community string in the node_info table is the same as the community string on the node.
e. Check the nts logs to see if nts has declared the node as reachable. See NTS. If nts has declared the node as reachable, check the topod.log for "nonRoutingNodeMsg" for this node. Verify that the alarm status in this message is the correct alarm status.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.*.log, nts.*.log, and ooemc*log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.2.6 CTM State Is Incorrect
If the management state of the node in the node table (mgmt_state column) shows as DOWN (2) or UNKNOWN (0) even when the node is reachable by IP, complete the following steps:
Step 1 Check the trap manager registration of the CTM station on the node. Log into the node and enter the following command to check whether the CTM station has registered to the node for traps. This is required for the node to be declared as manageable (mgmt_state = UP).
Step 2 Enter the community strings (SNMP-GET and SNMP-SET) of the node through the Domain Explorer > Network Element Properties pane > NE Authentication tab. The SNMP strings might be incorrect in the node_info table. The strings can be verified if the decrypt tool for the encrypted strings is available in the database.
Step 3 Check the nts.log to see if trap registration succeeded for the node. See NTS.
Step 4 Save ILMITopoc.log, topod.log, and ooemc*.log for analysis.
K.12.3 Discovery Issues at Runtime
This section includes the following information:
•Node Is Not Shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Node Name Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Node IP Address Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
K.12.3.1 Node Is Not Shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If a dynamically added routing node is not shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /usr/user/svplus/log directory and verify whether the node is present in the cache dumps. If the node is present, open a new instance of the GUI and check whether the node is visible in that GUI.
Step 2 If the node table does not have an entry for the node, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 70005 and 70201 trap (if the persistent topology feature is enabled).
b. If you see the trap in the log, verify that ILMITopoc's SNMP requests issued after receiving the trap are successful. If an SNMP request does not go through, verify whether CTM has IP and SNMP connectivity to the newly added node.
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Step 3 Complete the following substeps to collect defect information for analysis:
a. Save ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
b. Enter the following command to collect the dump outputs of ILMITopoc, topod, and NMServer:
kill -USR1 signal <process>
c. Collect the output of the switch CLI, selnd, and dbnds.
Step 4 If the preceding steps do not solve the problem, delete the node from the network and add it again.
K.12.3.2 Node Name Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If changes to the node name are not reflected on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node with the correct name, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /usr/user/svplus/log directory and verify whether the node name is correct in the cache dumps. If the node name is correct, open a new instance of the GUI and check whether the node name is correct in that GUI.
Step 2 If the node table does not have an entry for the node with the correct name, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 60006 and 70202 trap.
b. If you see the trap in the log, verify that ILMITopoc updates its cache based on the trap. If it fails to update the node name, ILMITopoc.log will contain the message "%ILMITopoc-3-updateFailed: <Node_c::updateName> Failed to update node name."
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.3.3 Node IP Address Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If the node IP address is not updated dynamically in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node with the correct IP address, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /opt/svplus/log directory and verify whether the IP address is correct in the cache dumps. If the IP address is correct, open a new instance of the GUI and check whether the IP address is correct in that GUI.
Step 2 If the node table does not have an entry for the node with the correct IP address, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 60007 and 70202 trap.
b. If you see the trap in the log, verify that ILMITopoc updates its cache based on the trap.
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
K.12.4 Equipment Management Problems
This section describes troubleshooting procedures for equipment management problems on the Cisco MGX devices.
K.12.4.1 Equipment Management Declares Successful Node Resync when Node Mode Is 3
MGX nodes are managed by the ooemc component in CTM. Equipment management (EM) uses coldstart, warmstart, and periodic resync to synchronize network data. Each of these resync methods involves configuration file upload, parsing, and database population. The node mode in the node table indicates the state of the synchronization process. The possible node modes are 1, 2, 3, and 5. The expected mode for a successful node resync is 3. If the node mode stays in other modes for a long time, it indicates a resync problem. Any synchronization problem could be caused by the switch, SNMP, FTP, or ooemc. You must investigate different areas to identify the root cause of any synchronization problem.
The following is the flow of events in the ooemc coldstart synchronization process:
1. The topo process discovers a node in the network. The process changes the node mode in the node database table to -1. It then notifies the ooemc component for the discovery. The ooemc changes the node mode from -1 to 1 after the component begins managing the node.
2. When the nts component is able to register with the node, it sends a link up message to the ooemc that manages the node. The ooemc component changes the node mode from 1 to 2 to signify the start of the node synchronization.
3. The ooemc component sends an SNMP bulk file creation request to the switch. If the request is allowed, the switch sends a trap 60901 to CTM to signify the start of bulk file creation. When bulk file creation is complete on the switch, the switch sends a trap 60902 to CTM. After CTM receives the trap, it begins to FTP the config upload files.
4. If the upload and parsing of all config files completes without errors, the ooemc changes the node mode from 2 to 3. If the upload or parsing encounters errors for any service module on the switch, the node synchronizes partially and the ooemc changes the node mode from 2 to 4 at the end of the node synchronization process. If the error occurs on the shelf generic card file (CARD_01_CC.CF) or pnni file (PNNI_01_CC_CF), the ooemc changes the node mode from 2 to 5 to signify node sync-up failure.
K.12.4.2 Network Setup and Configuration Required for OOEMC Sync-Up Process
The EM requires network setup and CTM configuration prerequisites before it can start the sync-up process. On the switch, perform the following:
•Use the dsptrapmgr CLI command to verify that the CTM IP address has been added to the trap manager. If your CTM IP address is not registered with the switch, add the CTM IP address to the trap manager by entering the addtrapmgr <cwm_ip> 2500 CLI command, where 2500 is the port number and cwm_ip is the IP address of your CTM machine. You might need to delete other IP addresses from the trap manager list if the list is full. If the list is not full, the registration takes place automatically if CTM is configured to manage the node.
•The trap IP should have been configured on the switch properly. Whenever you configure the switch to use any IP (atm0 IP or lnPci0 IP) of the switch as primary IP, the topology component of CTM should use that IP for node discovery. You can display the primary and secondary IP information of the switch by typing the CLI command dspndparms. To configure the primary and secondary IP addresses, you can enter cnfndparms and the CLI will prompt you for options. After you have set up the primary and secondary IP addresses, you should also configure the primary IP as trap IP on the switch by entering the CLI command cnftrapip configured_IP, where configured_IP is the IP that you have chosen to use as the primary IP in the CLI command cnfndparms.
•On CTM, the following is a list of configurable parameters in the file /opt/svplus/config/emd.conf of your CTM machine that you should set up for proper problem logging:
–"OODebug": For debugging purposes, this parameter should be set to level 6 or above; for example, the log level statement in emd.conf should be "OODebug Level 6."
–"OOKeep": For debugging purposes, this parameter should be set to a value depending on how many log files you want to keep; for example, the statement in emd.conf should be "OOKeep 100 ooemc log files per oochild."
Note Sometimes, a different network environment requires other parameters in the same files to be tuned for correct functioning of CTM.
K.12.4.3 Node Mode Remains in 1
The node has been discovered by the network topology process in CTM, its node mode has been changed from -1 to 1 in node table entry, and it stays in mode 1 for a long period of time.
Step 1 If CTM stays in mode 1 for a long time after CTM core has been started, you need to check the trap manager on the switch. See Network Setup and Configuration Required for OOEMC Sync-Up Process for trap manager and trap IP setup.
Step 2 If step 1 is not a problem, then check for rtm link up message that nts sends to EMD. The ooemc will start node resync once EMD notifies the ooemc that the node is active. Therefore, the second step for debugging this mode 1 problem is to see whether rtm link up message has been received by EMD and whether it has been forwarded to the ooemc.
For example, to find the rtm link up messages received by EMD for node with ID 9, issue the following command:
grep RTM_LINK_UP emd* | grep "Node id 9"
Once the location of the rtm link up message is found in the log, view the log file for more information on notification to ooemc.
Step 3 If the rtm link up message for the node is found and the log indicates that notification to ooemc has been sent, then collect the log files and report the problem.
Step 4 If the rtm link up message cannot be found, then search for link down message. Grep RTM_LINK_DOWN from the emd log files. If the rtm link down message is found, then the node is not reachable by the nts process. Check with the network administrator and see NTS if RTM_LINK_UP and RTM_LINK_DOWN messages are not found.
Defect information—Collect the following information for further analysis:
•Ooemc and nts log files from the /opt/svplus/log directory.
Possible alternative workaround—None.
K.12.4.4 Node Mode Remains in 2
After EMD receives the rtm link up message and the node active message has been sent to ooemc, ooemc will change the node mode from 1 to 2 and trigger the node resync process. The node resync time varies depending on the switch configuration and network activities. If the node mode stays in mode 2 for longer than the normal time for node resynchronization, there can be a problem with the node resync process.
Step 1 The debugging process for this problem is mainly focused on log file inspection. Make sure that log level of all related CTM processes is set at the right level. See Network Setup and Configuration Required for OOEMC Sync-Up Process for more information.
Step 2 Go to the /opt/svplus/tmp directory to check for configuration upload files.
Step 3 Determine which files have been uploaded for a node. For example, issue the following command on a node with node_id=9:
If files have been uploaded for this node, then the node resync process is ongoing. Make sure that the time stamps for these files refer to the current time.
Note Refer to step 6 for a list of configuration upload files.
Step 4 If there are no files uploaded for a node, it is possible that the trap IP address on the switch does not match the IP address used for node discovery. It is also possible that an SNMP request failure occurred. To check the IP addresses, perform the following:
a. Use the CLI command dbnds to display the IP address used for node discovery.
b. Use the CLI command dsptrapip to display the trap IP address.
c. If there is a mismatch, use the CLI command cnftrapip to reconfigure the trap IP address. Use the IP address used by CTM for node discovery. See Network Setup and Configuration Required for OOEMC Sync-Up Process for more information.
Step 5 If the trap IP address is not the issue, check whether there is an SNMP request failure. Normally, after the node resync process is triggered, ooemc will send SNMP requests to the switch to start the configuration upload file creation process. If the request fails, then ooemc will continue resending the SNMP requests until it exceeds the maximum number of retries and declares node resync failure with node mode equal to 5.
Note It should not take long to declare node resync failure because of the SNMP request failure. To verify that there is an SNMP failure, check the ooemc and snmpcomm log files.
Note Using ooemc10.6568.log as an example of an ooemc log filename, 10 is the child ID, and 6568 is the ooemc process ID. The child ID is calculated from the following formula: Remainder (NEDBACCSSID / number of ooemc child) + 1. To retrieve the process ID, use the CLI command psg em.
Step 6 Grep RESYNC from the log file. The grep result should include the node ID and node mode. Review the log messages in the log file to determine if an SNMP failure has occurred.
Step 7 Do the following:
a. Check the log file for trap 60903. If you find it, contact the platform team to investigate the bulk file creation failure on the switch.
Very often, the node mode 2 problem is caused by a bulk file creation abort process on the switch. If there are no SNMP request failures, the switch should send traps 60901 and 60902 to ooemc. Trap 60901 indicates that the switch will start bulk file creation, while trap 60902 indicates bulk file creation done. When the ooemc receives a trap 60902, it will start to FTP the bulk file, which is a shelf generic configuration file. This is also the first file to be uploaded in every resync mode (coldstart, warmstart, or periodic resync.) The commonly seen mode 2 problem is that the switch will abort the bulk file creation and send trap 60903 to CTM. CTM will reschedule the next SNMP request unless it exceeds maximum retries.
b. Check the log file for trap 60902.
CARD_01_CC.CF.10 is a shelf generic file and is the first file to be uploaded after CTM receives trap 60902 from the switch. The 10 is the node ID. If you find trap 60902 in the log file but no config file has been uploaded after a reasonable amount of time, it is possible that the FTP failed.
To find out if FTP failure has occurred, do the following:
a. Grep "to ftp file" and the config filename, or just "FTP" from the log files.
b. Trace the log messages in the log file.
For more detailed information on SNMP request failure and FTP failure, check the snmpcomm and cwmftpd log files and the cwmftp.request_log. Search for the filename at the time the error happened in both files. The cwmftp.request_log gives the summary or final result of the FTP operation, and any error is reported. The cwmftp.log shows the FTP operation details.
CTM will upload and parse a set of config files from the switch. The following files are uploaded from the MGX NE, which contains VISM, AXSM, VXSM, SRM, and RPM/RPM-PR cards:
1. CARD_01_CC.CF
2. SM_1_slot#.CF
3. SM_1_slot#.CS
4. SM_CARD_01_slot#.CF
5. SM_CONN_01_slot#.CF
6. SM_ALARM_01_slot#.CF
7. SM_CON_UPDATE_01_slot#.CF
8. SM_CARD_01_SRM.CF
9. SM_CARD_01_RPM.CF
10. PNNI_01_CC.CF
Files 1 and 2 are uploaded for each NBSM. Files 4 to 7 are uploaded for each AXSM. File 9 is uploaded for all RPM/RPM-PR cards on the switch.
The following files are uploaded from the MGX NE, which includes VISM, AXSM, VXSM, SRM, RPM/RPM-PR, and RPM-XF cards:
1. CARD_01_CC.CF
2. SM_1_slot#.CF
3. SM_1_slot#.CS
4. SM_CARD_01_slot#.CF
5. SM_CONN_01_slot#.CF
6. SM_ALARM_01_slot#.CF
7. SM_CON_UPDATE_01_slot#.CF
8. SM_SC_slot#_transactionID_date.CF
9. SM_IC_slot#_transactionID_date.CF
10. SM_CARD_01_SRM.CF
11. SM_CARD_01_RPM.CF
12. PNNI_01_CC.CF
The difference between the items in these lists is in the files uploaded for AXSM cards and RPM-XF cards. For switch software release 4.0 or later, the static file (SM_SC_slot#_transactionID_date.CF) and incremental file (SM_IC_slot#_transactionID_date.CF) are uploaded. For switch software release 3.0 and earlier, conn, alarm, and conn update files are uploaded for each AXSM. Files 4 to 7 in the second list are uploaded for each RPM-XF card, and file 11 is uploaded for all RPM/RPM-PR cards on the switch.
Step 8 In another scenario in which the node mode stays in 2 for a very long time, the parsing of one particular config upload file has taken a very long time and has not completed. To view the parsing process, enter the following command:
Step 9 Collect the ooemc, nts, snmpcomm, and cwmftpd log files from the /opt/svplus/log directory.
Possible alternative workaround—None.
K.12.4.5 Node Resync Mode Is 5
Node mode 5 indicates that node resync has failed. This could be caused by the config upload or parsing failure of the shelf generic file (CARD_01_CC.CF.13, for example) or pnni file (PNNI_01_CC.CF.13, for example). In general, mode 5 signifies that a problem has happened in one or more stages of the entire resync process. The resync process is composed of the following stages and each stage alone can lead to mode 5 problems:
1. The ooemc triggers node resync.
2. The ooemc sends an SNMP request to the switch for bulk file creation.
3. The ooemc receives bulk file creation-related traps: 60901, 60902, and 60903.
4. The ooemc FTPs the config upload files from the switch after it has received 60901 and 60902 from the switch.
5. The ooemc parses the config upload files.
6. The ooemc declares sync-up done.
Step 1 Determine if the problem is caused by the SNMP request:
a. Grep RESYNC from the ooemc log files. The ooemc will change node mode to 2 when it starts the node resync process. You should see something like the following from the log:
NOTICE: N17 <EMC_Node_c::InSync> SENDING RESYNC STATUS 2 FOR NODE 17 TO GUI - Node is
synchronizing.
This message tells you that ooemc will start the node resync process for nodes with ID equal to 17 (N17). If you look further in the log, you should see log messages related to SNMP requests and SNMP responses. If the SNMP request is successful, the switch will respond to the request. The ooemc will then process the SNMP response by invoking response function, which may do nothing:
<EMC_SnmpFunc_c::ProcFunc_GenNodeBulkFile_1> entering
b. If the log messages indicate SNMP error, refer to the snmpcomm log files for more failure information.
Step 2 Determine if the problem is caused by bulk creation traps. Grep 60901, 60902, and 60903 from the log files. (See Node Mode Remains in 1 for more information.) You should see something like the following:
INFO: <EMC_TrapClientImpl::onIncomingTrap> NTS NodeId 17 genericTrap 6 specificTrap 60901
INFO: <EMC_TrapClientImpl::onIncomingTrap> NTS NodeId 17 genericTrap 6 specificTrap 60902
Step 3 Determine if the problem is caused by the config file FTP. Check the FTP or the FTP plus config upload filename. You should see something like the following:
NOTICE: N17 <EMC_NodeFsmHandler_c::LoadShelf> to ftp /opt/svplus/tmp/CARD_01_CC.CF.17
INFO: <ParseFile_c::CheckFile> OOEMC9 CHECKSUM OK FOR FTP FILE
/opt/svplus/tmp/CARD_01_CC.CF.17
These messages indicate that FTP is successful and there is no checksum error in the file. From the log file, you should able to find out if there is an FTP problem. Refer to the cwmftpd log files for more failure information.
Step 4 Determine if the problem is caused by shelf generic file or pnni file parsing. After you have located where to do a checksum check on the shelf generic file (CARD_01_CC.CF.17, for example) or pnni file (PNNI_01_CC.CF.17, for example), continue to trace the log messages to see whether or not a parsing error has occurred. If there are no errors, you should see the following:
INFO: N17 <EMC_NodeFsmHandler_c::FinishShelf> Parse /opt/svplus/tmp/CARD_01_CC.CF.17
successfully.
Defect information:
•Collect the ooemc, nts, snmpcomm, and cwmftpd log files from the /opt/svplus/log directory.
•Collect the config upload files from the /opt/svplus/tmp directory.
Possible alternative workaround—None.
K.12.4.6 CTM Database Is Inconsistent with Switch Data After Successful Coldstart (CTM Server Stop/Start) or Periodic Resync
After a successful node resynchronization triggered by the periodic resync, the CTM database is found to be inconsistent with the switch data.
Step 1 Collect the ooemc log files and all config upload files for the node. The ooemc implements a node-based cache. You can dump and save the cache.
Step 2 Get the process ID of the ooemc process that manages the node.
Step 3 Save the config upload files and ooemc log files.
Defect information—Collect the ooemc and config upload files from the /opt/svplus/log directory.
Possible alternative workaround: Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
K.12.4.7 CTM CM GUI Shows Incomplete Connections After Successful Node Resync and Dbroker Sync-Up
CTM shows that some of connections are incomplete after the node resync process and dbroker synchronization process.
Step 1 Determine which ooemc process manages the node. The ooemc will manage the connection segment in the CTM database, which terminates on the MGX. Find the child ID of the ooemc process that manages your node. The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection. The debugging process for this issue mainly focuses on log file inspection.
Step 2 After you have identified the ooemc log files, grep NotifyDataBroker from the files to see some log message examples before you modify your grep format. This will help you more efficiently grep the pair of log messages that correspond to the local and remote ends of one endpoint (primary endpoint or secondary endpoint) of a connection segment. In other words, each end (either primary end or secondary end) of a connection segment managed by ooemc should be logged with one pair of messages, one for local end and one for remote end. This pair of messages corresponds to one atm_connection segment entry in the database. Verify the information that ooemc sends to dbroker. Dbroker will refer to these messages from ooemc and construct a user_connection database entry. The CTM Connection Management (CM) GUI displays connection information based on the user_connection database entry. Whether or not the connection is displayed as complete or incomplete depends on the information in the user_connection database entry. It is important to verify that ooemc sent the correct information to dbroker. Also verify that the segment database entries are populated correctly.
Step 3 If the correct number of messages have been forwarded to dbroker and the data in the messages is correct, but the user_connection database entry still contains invalid data, see Data Inconsistency for more debugging information, or contact the dbroker development engineer (DE) for further investigation. If the problem is due to wrong data populated in segment database entries, then the EM DE should look at the problem.
Defect information:
•Collect ooemc and dbroker log files from the /opt/svplus/log directory.
•Collect config upload files from the /opt/svplus/tmp directory.
Possible alternative workaround is to manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
K.12.4.8 CTM GUI Shows Mismatched Information Between GUI Views and Database Data
After an initial ooemc node resync, such as a coldstart process, GUI views such as Network Monitor tree view, Inspector View, and Chassis View display information that does not match newly provisioned or updated database data. The problem persists even after each subsequent manual or periodic resync.
Step 1 Determine which ooemc process manages the node and find the child ID of the ooemc process that manages your node. The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection. The debugging process for this issue mainly focuses on log file inspection.
Step 2 Initiate an initial node resync, such as coldstart; ooemc starts sending newly provisioned or updated database data to the network management (NM) server. The current supported database tables for NM server message forwarding include the following:
•node
•card
•line
•ausm_port
•cesm_port
•frp
•rpm_port
•svc_port
•virtual_port
•aps
•ima_group
•ima_link
•linedistribution
•au4tug3
•controller
•license_in_use
•mfr_bundle
•mfr_link
•peripheral
•redundantcard
•sensor
Step 3 To determine whether or not ooemc sent newly provisioned or updated database data to the NM server, grep ComposeNMSMsg from the ooemc log files. The key words in log messages that correspond to the supported database tables in step 2 are the following:
•EMC_DBProperty_Node_c::ComposeNMSMsg
•EMC_DBProperty_Card_c::ComposeNMSMsg
•EMC_DBProperty_Line_c::ComposeNMSMsg
•EMC_DBProperty_AusmPort_c::ComposeNMSMsg
•EMC_DBProperty_CesmPort_c::ComposeNMSMsg
•EMC_DBProperty_FrPort_c::ComposeNMSMsg
•EMC_DBProperty_RpmPort_c::ComposeNMSMsg
•EMC_DBProperty_SvcPort_c::ComposeNMSMsg
•EMC_DBProperty_VtPort_c::ComposeNMSMsg
•EMC_DBProperty_Aps_c::ComposeNMSMsg
•EMC_DBProperty_ImaGroup_c::ComposeNMSMsg
•EMC_DBProperty_ImaLink_c::ComposeNMSMsg
•EMC_DBProperty_LineDist_c::ComposeNMSMsg
•EMC_DBProperty_AU4TUG3_c::ComposeNMSMsg
•EMC_DBProperty_Ctrlr_c::ComposeNMSMsg
•EMC_DBProperty_LicenseInUse_c::ComposeNMSMsg
•EMC_DBProperty_MfrBundle_c::ComposeNMSMsg
•EMC_DBProperty_MfrLink_c::ComposeNMSMsg
•EMC_DBProperty_Peripheral_c::ComposeNMSMsg
•EMC_DBProperty_RedCard_c::ComposeNMSMsg
•EMC_DBProperty_Sensor_c::ComposeNMSMsg
Note Database data will not be sent to NMServer during the initial node resync. It is after the initial resync that ooemc starts sending updated or newly provisioned database data to NMServer. Each ooemc log message from grep contains detailed information on the identity of the NE object and the required fields from the corresponding database table. The following is an example of NMServer message for line tables:
ooemc10.24370.log.old.5:( 24370: 4) 23:15:17 INFO: N0:C7:B2:L2
<EMC_DBProperty_Line_c::ComposeNMSMsg> (MODIFY::LINE) aNMSEvent.node:0 type:4 subType:1
lpbkType:1 alarmState:1 adminState:1 sectStatus:6 pathState:2
In this example, the identity of the NE object is N0:C7:B2:L2, which is the line with node ID=0, slot=7, bay=2, and line#=2. The database operation is update. The rest of the message shows the values of the required fields from the line table.
Step 4 If grep returns log messages that indicate matched data, then you need to continue investigation on the NM server. See CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics for information on nmController.
Defect information—Collect ooemc and NMServer log files from the /opt/svplus/log directory.
Possible alternative workaround—Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
K.12.4.9 CTM Database and Switch Data Inconsistency Issues After Node Provisioning for MGX
This section includes the following information:
•Database Table Population Through Traps and SNMP Upload
•Switch Data Does Not Match CTM Database Table After Node Provisioning
K.12.4.9.1 Database Table Population Through Traps and SNMP Upload
Except for node and/or card resync, another mechanism that CTM ooemc uses to populate the database table entries is through trap processing followed by SNMP upload, if necessary. You can provision the switch through switch CLI, CTM GUI, or CTM service agents. All three cases will involve trap processing and SNMP upload on ooemc. Only connections can be provisioned by the CTM CM GUI. Other provisioning, such as line and port, have to go to CTM service agents. The MGX switch CLI can do all.
K.12.4.9.2 Switch Data Does Not Match CTM Database Table After Node Provisioning
After switch provisioning through the switch CLI, CTM GUI, or CTM service agents, the switch data does not match the CTM database.
Step 1 Study the log files to understand the root cause of the inconsistency problem. If the issue is caused by a trap (for example, if you provision a connection on a switch or through CTM GUI or service agent, and CTM does not populate the database entry or the information in the database entry does not match the switch data), it is possible that CTM did not receive the trap, or that it received the trap but the trap is buffered in the ooemc trap queue, and trap processing has been delayed. On the other hand, if the data in the CTM database is not correct, it is also possible that the SNMP upload failed to upload correct data.
Step 2 To determine whether or not a trap has been received and processed, there are key words in the ooemc log files that you can grep. For example, to find out the channel traps from node_id=4, slot=6, vpi=1, and vci=326, you can grep "TRAPLIST" as shown in the following example:
cwmult60% grep "TRAPLIST: N4:" ooemc* | grep "Channel Trap" | grep "C6" | grep "vpi 1 vci
326"
ooemc10.5760.log.old.55:( 5760: 10) 23:21:12 INFO: TRAPLIST: N4: Channel Trap 60310 from
C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 67
ooemc10.5760.log.old.55:( 5760: 10) 23:21:12 INFO: TRAPLIST: N4: Channel Trap 60310 <
PROCESSED > from C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2
operS 1 alarm 67
ooemc10.5760.log.old.73:( 5760: 10) 23:23:48 INFO: TRAPLIST: N4: Channel Trap 60310 from
C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 66
For other kinds of traps, you can use the following key words to supplement "TRAPLIST" in your grep statement:
•Port Trap
•RscPart Trap
•Svc Trap
•SonetLn Trap
•SctCard Trap
•SonetPath Trap
•FunMod Trap
•LineMod Trap
•RedCard Trap
•TrapMiss Trap
•VsiCtrlr Trap
•DS3Line Trap
•AtmPhy Trap
•Peripheral Trap
•CoreSwth Trap
•TrapLost Trap
•AtmAddr Trap
•Restart Trap
•Node Trap
•SonetAps Trap
•LMIPort Trap
•Pnni IF Trap
•Bulkfile Trap
•Vism Trap
•Vism Ann Trap
•SubIf Trap
•NBSMCnfg Trap
•NBSMChan Trap
•NBSMLine Trap
•AusmLine Trap
•AusmPort Trap
•AusmChan Trap
•AusmIma Trap
•FrChan Trap
•FrPort Trap
•CesmPort Trap
•CesmChan Trap
•HsFrPort Trap
•HsFrChan Trap
•LineDist Trap
•DS3Path Trap
•Chan Upload
•Party Trap
•PrefRoute Trap
•CardIma Trap
•DS1 Line Trap
•SctPort Trap
•SvcDerouteGroomTrap
•Channel Trap
•TUG3Path Trap
•Cug Trap
•AddrCug Trap
•RSC Upload
•APS Upload
•FrPort State Upload
•MPSM Upload
•Vism ToneDetect Trap
•License Trap
•PortAtmIf Trap
•VxsmPvcRed Trap
•ChanProt Trap
•VxsmGwDsp Trap
•VxsmGwIp Trap
•VxsmGw Trap
•VxsmSysRes Trap
•VxsmGw1 Trap
•VxsmMgc Trap
•VxsmMgcIp trap
•VxsmMgcGrpParam Trap
•VxsmMgcGrpMgc Trap
•VxsmMgcGrpProt Trap
•VxsmAal2Prof Trap
•VxsmCodec Trap
•VxsmSvc Trap
•VxsmAal2CrossConn Trap
•VxsmAal25DataProfileTrap
•VxsmSensor Trap
•VxsmSensorThrhd Trap
•VxsmModule Trap
•DS0Grp Trap
•VxsmAnnounce Trap
•VxsmAudioFile Trap
•VxsmDs0XConn Trap
•VxsmMegaco Trap
•VxsmCrr Trap
•VxsmTone Trap
•VxsmAs Trap
•VxsmAsp Trap
•VxsmAs Trap
•VxsmLapd Trap
•VismABCDBitTemplate Trap
Review the log messages using the traplist command and decide how you can grep the messages according to your needs. In the above example, there is another key word ("PROCESSED") that indicates that the trap has been processed. This gives you the starting point in the log file so that you can study the log messages. If the database is not populated correctly, then the subsequent log messages should provide you information on the database inconsistency problems. Some trap processing may invoke SNMP data upload. From the log messages, you can verify whether the uploaded data is correct or not. Other possible reasons for database inconsistency are:
•SNMP upload failure and maximum retries is exceeded; SNMP timeout; or throttle error. You can verify the problem in the snmpcomm log files.
•Database operation error. You can verify the problem in the ooemc log files.
If you do not know exactly what trap sequence is sent by the switch to CTM, you should try a working scenario and study the log for the trap sequence, or you can use HPOV to determine the trap sequence.
Step 3 To verify whether or not a trap has been received from the switch and forwarded to ooemc, refer to the nts log. If the trap has been buffered in the trap queue, the possible reasons are:
•Node is syncing due to regular node resync or due to -2 trap.
•Card is syncing and the trap is related to the card. The traps will be buffered in the queue until card resync completes.
•Summary alarm trap is processed. Summary alarm is being uploaded or parsed. You can grep the log files for information related to summary alarm traps.
To verify that the trap has been put into the queue, you can do the following:
grep EMC_TrapQueue_c::append ooemc_log | grep trap_num
You should see something like the following:
INFO: <EMC_TrapQueue_c::append> entering. ======> append trap# 60303 to trap queue;
getHdlrLevel=5, getTrapLevel=5, #=174
Defect information—Collect ooemc, nts, and snmpcomm log files from the /opt/svplus/log directory.
Possible alternative workaround—Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
K.12.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems
This section includes the following information:
•CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics
•Basic Issues
•Topology Discovery Issues
•Network Element Discovery Issues
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues
K.12.5.1 CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics
CTM manages MGX NEs. The topology module discovers the nodes and trunks. Once the nodes are discovered, the EM module syncs up with the nodes to discover the card, line, port, and so on. All the discovered nodes, trunks, and node elements are published by NMServer and are displayed in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report. The Configuration Center, Chassis View, Diagnostics Center, or Statistics Report has the tree view and Inspector View. The tree view and Inspector View feed off the data published by NMserver.
CTM is notified of all subsequent changes in the network through traps.
The following figure shows the end-to-end architecture.
Figure K-1 CTM End-to-End Architecture
NMServer provides inventory and alarm information for the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report. The node information is pushed by topod to NMServer. The initial inventory of card, lines, ports, and so on, is read from the database. The dynamic updates are pushed to NMServer from topod, ooemc, and sdbroker.
There are two utilities under the /opt/svplus/util/ directory that are shipped with CTM to debug issues in NMServer:
•nmControl—This utility provides the means to check the NMServer cache and its state. Output is redirected to /opt/svplus/log/nmControl.log. Cache dumps are redirected to /opt/svplus/log/nmControl.dump. The utility allows the following operations, all of which are nondestructive and perform the tasks in a passive mode.
Note Do not use the All Cache operation to dump the cache for a large network that has more than 1000 nodes.
–Resync Node—Resynchronizes the node (specified by the node_id).
–All Cache—Dumps all the data in its cache to the dump file.
–Topology Cache—Dumps topology (node) data to the dump file.
–Node Cache—Dumps a node's data (specified by node_id) to the dump file.
–CTM Alarm Cache—Dumps the CTM-specific alarms to the dump file.
–Client Manager Cache—Dumps the information about all the clients to the dump file.
–Error Statistics—Dumps the error information associated with the sync-up to the dump file.
–Sync-Up Status—Dumps the sync-up state to the dump file.
–Configuration—Dumps the configuration data to the dump file.
•nmClient—This utility is used to isolate a problem between client and server. It is used to query the NMServer the same way as the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI queries the server. Output is redirected to /opt/svplus/log/nmClient.log. Cache dumps are redirected to /opt/svplus/log/nmClient.dump.
You must perform the Register Client operation before performing any other operations. When you are finished using the client, perform the Unregister Client operation.
–Register Client—Registers with the server.
–Update Filter—Updates the subscription/filter with the server passing the FDN.
–Unregister Client—Unregisters with the server.
–Get Topology—Gets the topology information of networks and nodes.
–Get Children—Gets the list of children for a particular object in the tree.
–Get CwmInfo—Gets the CTM sync-up state information.
–Get ManagedObject—Gets the detailed information about an object in the tree.
–Subscribe for all events—Subscribes to all updates on the server.
–Unsubscribe for all event—Unsubscribes from all updates on the server.
The GUI client's log files (CMSCclient.log) are saved under log directories; for example, in Windows, D:Documents and Settings\<username>\log\, or in UNIX: /opt/svplus/log.
K.12.5.2 Basic Issues
This section includes the following information:
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show Any Nodes
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Is Unable to Connect to Server
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Newly Added Node
K.12.5.2.1 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show Any Nodes
Step 1 Check whether the nodes are discovered. See No Nodes Are Discovered.
Step 2 Check whether NMServer has the nodes and trunks in its cache. This can be done by using the command nmControl in the CTM CLI.
Step 3 If the nodes are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI.
K.12.5.2.2 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Is Unable to Connect to Server
Step 1 Check whether the client registers in NMServer.log.
Step 2 Enable orbix logs by creating the orbix directory in /opt/svplus/log/ directory.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log, NMServer.log
•Orbix logs
Possible alternative workaround—None.
K.12.5.2.3 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Newly Added Node
Step 1 Check whether the nodes have been discovered. See No Nodes Are Discovered for more information.
Step 2 Check whether NMServer has the nodes in its cache. This can be done by using the command nmControl in the CTM CLI.
Step 3 If the nodes are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI.
K.12.5.3 Topology Discovery Issues
This section includes the following information:
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Node
•Node Information Incorrect on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Node that Is Not in the Network
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows Incorrect SyncState for a Node
•Duplicate Nodes Are Displayed on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
K.12.5.3.1 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Node
Step 1 Check whether the nodes are discovered. See No Nodes Are Discovered for more information.
Step 2 Check whether NMServer has the nodes in its cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 If the nodes or trunks are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
K.12.5.3.2 Node Information Incorrect on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
Step 1 Check the node information in the database using selnd command.
Step 2 Check the node information in the NMServer's cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 Check the CMSCclient.log for the node information.
Step 4 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Step 5 If the node information is correct in the NMServer cache, open a new GUI and verify whether the node is shown correctly in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•selnd o/p
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
K.12.5.3.3 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Node that Is Not in the Network
Step 1 Check the node information in the database by querying the node, packet_line table. Verify whether the node is active.
Step 2 If the node is active, see No Nodes Are Discovered.
Step 3 Check the node in the NMServer's cache. This can be done by using the command nmControl in the CTM CLI.
Step 4 Check the CMSCclient.log for the node/trunk. Verify whether a delete message was received for the node.
Step 5 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and GUI.
Step 6 If the node is not present in the NMServer cache, open a new GUI and verify whether the node is shown in it.
Defect Information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, and NMServer.log.
•Dump outputs of ILMITopoc, topod and NMServer. The dump can be captured by issuing a kill -USR1 signal to the process.
•Output of the switch CLI, selnd, and dbnds.
K.12.5.3.4 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows Incorrect SyncState for a Node
Step 1 Check the node information in the database by querying the node table. This can be done by using the command selnd on the CTM CLI. Check the status.
Step 2 Check the node in the NMServer's cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 Check the CMSCclient.log for the node. Verify whether the correct message was received for the node.
Step 4 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Defect Information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, and NMServer.log.
•Dump outputs of ILMITopoc, topod and NMServer. The dump can be captured by issuing a kill -USR1 signal to the process.
•Output of the switch CLI, selnd, and dbnds.
K.12.5.3.5 Duplicate Nodes Are Displayed on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
Step 1 Check whether the node table has duplicate node IDs.
The node table can be queried using the following command:
tballraker18% echo "select (*) from node where node_id=2 and slot=1" | dbaccess stratacom
Step 2 Check whether both nodes in the database have active=1. If both nodes are active then collect the logs.
Step 3 If duplicate nodes do not exist in the database, check the cache of NMServer using nmControl.
Step 4 Check the CMSCclient.log for the node.
Step 5 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Step 6 If duplicate nodes are not present in the NMServer cache, open a new GUI and verify whether the node is shown in it.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under /opt/svplus/log. For PC Client, collect CMSCclient.log file under D:\Documents and Settings\<username>\log.
•Node data from node table for that specific node.
•ILMITopoc.log and topod.log, if there are duplicate nodes in the database.
Possible alternative workaround—None.
K.12.5.4 Network Element Discovery Issues
This section includes the following information:
•All the Cards Under a Node Are Missing
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Card, Line, or Port for a Node
•Card, Line, or Port Information Is Incorrect on Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Card, Line, or Port that Is Not Present in a Node
K.12.5.4.1 All the Cards Under a Node Are Missing
No cards are shown under a node in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synched up, wait for it to do so.
Step 3 If the node has synced up, check whether cards are populated in the card table. If the card table is empty, collect the EM logs. See Equipment Management Problems for more information.
Step 4 Check whether NMServer has cards in its cache. This can be done by using the command nmControl on the CTM CLI.
Step 5 If cards are not present in the NMServer cache, collect the logs.
Step 6 If cards are present in the NMServer cache, use nmClient and verify whether getChildren for node's FDN returns the cards in the nmClient.<pid>.dump file.
Step 7 If cards are present in the NMServer cache, open a new GUI and verify whether the cards are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
K.12.5.4.2 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Card, Line, or Port for a Node
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, wait for it to do so.
Step 3 If the node has synced up partially, check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 7.0 Database Schema.
Step 4 If the entry is not populated in the database, see Equipment Management Problems.
Step 5 If the entry is present in the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump
Step 6 If the element is not present in the NMServer cache, use nmClient and verify whether getChildren for FDN returns the entities in the nmClient.<pid>.dump file.
Step 7 If the element is present in the NMServer cache, open a new GUI and verify whether the missing elements are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
K.12.5.4.3 Card, Line, or Port Information Is Incorrect on Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
The entity information of an element shows incorrectly on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, then wait for it to do so.
Step 3 Check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 7.0 Database Schema.
Step 4 If the entry in the database matches the GUI, see Equipment Management Problems.
Step 5 If the entry doesn't match the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump.
Step 6 Use nmClient and verify whether getChildren for FDN returns the correct information for the element in the nmClient.<pid>.dump file.
Step 7 If the element is present in the NMServer cache, open a new GUI and verify whether the missing elements are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
K.12.5.4.4 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Card, Line, or Port that Is Not Present in a Node
An extra card, line, or port element is shown incorrectly on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, then wait for it to do so.
Step 3 Check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 7.0 Database Schema.
Step 4 If the element is present in the database but does not match what is shown in the GUI, see Equipment Management Problems.
Step 5 If element is not in the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump.
Step 6 Use nmClient and verify whether getChildren for FDN returns the element in the nmClient.<pid>.dump file.
Step 7 If element is not present in the NMServer cache, open a new GUI and verify whether the extra element is shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI.
2. Use nmControl and resynchronize the node, specifying the node ID.
K.12.5.5 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues
This section includes the following information:
•Alarm Processing Basics
•XML Schema for Alarm Rules
•Alarm Severity and Object Severity
•Severity Shown on Tree View Does Not Match Severity Shown on Platform
•Alarm List Shows Alarm that Does Not Exist on Platform
•Transient Event Has Disappeared Unexpectedly
•Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port
K.12.5.5.1 Alarm Processing Basics
Alarm processing is done within the NMServer processes. The following are some vital points with regard to alarm processing:
•Alarm rules are defined in an XML file (/opt/svplus/xml/ruledata.xml). For details on XML schema alarm rules, see Figure K-2.
•GUI clients register for alarms by providing the parent FDN.
•GUI clients pull initial alarms from NMServer. After initial pull of alarms, all alarms are pushed to registered clients by NMServer when alarms occur.
•Object severity for any and all network elements is determined by the XML alarm rule.
•Administrative states of network element states are not displayed in the alarm list, only alarm states.
•Alarm List displays mostly active alarms, however some events are also displayed. For details on events versus alarms, see XML Schema for Alarm Rules.
•Some alarms result in the network element having a different alarm severity then the actual alarm. For details on object severity versus alarm severity, see Alarm Severity and Object Severity.
Alarms for various network elements are stored in memory cache in the NMServer Object Tree. GUI clients register for alarms from NMServer by providing the FDN of the network element for which they would like alarms. All alarms for network elements under the FDN registered will be sent if the parent FDN is registered. GUI clients may also update alarms, for example by flagging an alarm as acknowledged or manually clearing an alarm. These updates are done through the Alarm Repository component of NMServer. The following figure illustrates the client/server architecture for alarm components in NMServer.
Figure K-2 Client/Server Architecture for Alarm Components
K.12.5.5.2 XML Schema for Alarm Rules
Alarm rules are defined in XML, specifically, in a file named $HOME/xml/ruledata.xml. These alarm rules are read once when NMServer starts. When events are processed, the rules are queried from memory to determine what action, with regard to alarms, should be taken on the event. There are three types of alarm rules defined in the XML schema: Correlated, Correlated Bitmap, and Transient. Specifics on each of these three types are as follows.
1. Correlated Alarm Rule Type
The correlated alarm rule type is the most common in ruledata.xml. This rule is used when a network element can be in only one of many states at any one time. If the entity were to change states, then the previous alarm state would be cleared. Most network elements managed by CTM fall into this category. A simple example is the database sync-up status of a node. The sync-up status can be one of Partial Sync-Up, Sync-Up Failed, or In Sync. Any one of these states correlates out any other. See the following figure.
Figure K-3 Correlated Alarm Rule Diagram
A CorrelatedRule can have any of the following:
•1 AlwaysClear (Used when a given element never has an alarm, such as in a top-level network)
•0 or more ClearAlarmCondIds
•1 new alarm (which consists of NewAlarmConditionID, NewAlarmServAffect, and so on.)
2. Correlated Bitmap Alarm Rule Type
The correlated bitmap alarm rule type is different from the correlated alarm rule type because the bitmap rule type can represent the many states an entity can be in at any one particular time. The correlated bitmap rule type is used primarily to represent line alarms. Lines can have many different alarms associated with them at any given time, such as Loss of Signal and Loss of Frame. See the following figure.
Figure K-4 Correlated Bitmap Alarm Rule Diagram
A CorrelatedBmRule can have any of the following:
•0 or more ClearAlarmCondIds
•1 new alarm (which consists of NewAlarmConditionID, NewAlarmServAffect, and so on)
3. Transient Alarm Rule Type
The transient alarm rule type is used to distinguish events from alarms. The alarm list shows some events. Events are normally associated with the NMS itself. Examples of events are Process Restarted or Primary Gateway Disconnected. These are NMS events that are not correlated, but are displayed in the alarm list. Another example of a transient event that is not related to the NMS is a card switchover. If automatic protection switching (APS) is enabled on a card and there is an APS switchover, an event will be displayed in the alarm list. The primary distinction between transient events and correlated alarms is that transient events are not cleared by the network, whereas correlated alarms are. See the following figure.
Figure K-5 Transient Alarm Rule Diagram
A transient alarm rule can have only one new alarm associated with it.
Note There is no ClearAlarmCondID associated with a transient rule. Transient alarms can be manually cleared by the user, cleared by the same transient alarm occurring twice, or purged by NMServer when a high threshold is hit.
K.12.5.5.3 Alarm Severity and Object Severity
Every alarm in an alarm rules XML file is assigned two severities: object and alarm. The two severities are called Object and Alarm severity. These severities do match for most alarms. There are some cases, however, where these severities are different. The following is an example of an XML alarm rule where object and alarm severity do not match:
<CorrelatedRule State="1 -1">
<ClearAlarmCondID>10302</ClearAlarmCondID>
<ClearAlarmCondID>10303</ClearAlarmCondID>
<ClearAlarmCondID>10304</ClearAlarmCondID>
<NewAlarmCondID>10301</NewAlarmCondID>
<NewAlarmServAffect>0</NewAlarmServAffect>
<NewAlarmTransient>0</NewAlarmTransient>
<NewAlarmDbPersistent>0</NewAlarmDbPersistent>
<NewAlarmSev>6</NewAlarmSev>
<NewAlarmObjSev>7</NewAlarmObjSev>
<NewAlarmDescrSuffix>Sync-Up has not started yet</NewAlarmDescrSuffix>
This alarm will occur if the southbound processes (EMs) send a node message with EM sync-up status as 1 or -1. If that does occur, then the node should be unreachable severity (value 7) in the tree view. Note that there is no unreachable severity in the alarm list. This is why there are two severities for each alarm. Unreachable alarms have Critical (value 6) severity in the alarm list.
Another instance when alarm and object severity do not match is for aggregate port alarms. Aggregate port alarms are alarms that summarize the condition of the connections on the port. Since these alarms should not affect the severity of the port, the object severity of these is alarms is Clear (value 3). The following is the XML for an aggregate port alarm:
<CorrelatedBmRule StateBm="1">
<NewAlarmCondID>40801</NewAlarmCondID>
<NewAlarmServAffect>0</NewAlarmServAffect>
<NewAlarmTransient>0</NewAlarmTransient>
<NewAlarmDbPersistent>1</NewAlarmDbPersistent>
<NewAlarmSev>5</NewAlarmSev>
<NewAlarmObjSev>3</NewAlarmObjSev>
<NewAlarmDescrSuffix>Aggregate Port alarm, One or more connections on this port are in
primary failure</NewAlarmDescrSuffix>
Note The alarm severity (NewAlarmSev tag) is Major (value 5).
K.12.5.5.4 Severity Shown on Tree View Does Not Match Severity Shown on Platform
Severity of a network element in tree view does not match the severity the switch is showing for the same network element.
Step 1 Verify whether the customer is comparing the correct severity icon.
a. There are two severities associated with every object in the tree view, the aggregate severity and the self severity. The aggregate severity is on the left, the self severity is on the right. Since the switch does aggregation of many faults, the customer should compare the aggregate severity of the object in the tree view with the severity of the switch.
b. If you are looking at the correct severity icon and there is still a discrepancy between the severities, continue to step 2.
Step 2 Verify whether CTM has synced up with the node.
a. Log in to the CTM server and type selnd.
b. Verify that the node mode in question is 3.
Step 3 If the mode is 3, verify whether the discrepancy is caused by aggregated connection alarms. NMServer does more aggregation of alarms than the platform. For instance, NMServer aggregates connection alarms up the port and the switch does not. Therefore, if the tree view displays a higher severity than the switch, it may be caused by one or more aggregated port alarms.
a. Right-click the NE in the tree view and choose show alarms. All of the alarms for this NE and its children should appear in the alarm list.
b. Filter on the alarms that have greater severity then what the platform is showing.
c. Verify whether or not the alarms are aggregate port alarms. If so, this is expected behavior and there is no defect. If there are alarms that have greater severity then what is shown on the platform, and they are not aggregate port alarms, continue to the next step to see whether the alarm is in the database.
Step 4 Verify whether the database has the correct alarm state. Refer to the Cisco Transport Manager Release 7.0 Database Schema for information on what table to look up for this particular entity type.
Defect Information—Collect the following information for further analysis:
•NMServer.log
•nmControl.dump
•CMSCclient.log on client machine
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
K.12.5.5.5 Alarm List Shows Alarm that Does Not Exist on Platform
There are several alarms that are correlated by NMServer and are not handled by the switch. This is the case because the NMS keeps track of alarm conditions that may not be relevant to the switch alone, but are relevant to the switch and the NMS. One example of such an alarm is the node sync-up state. The switch itself is not interested in the fact that the NMS may not be synced up with it, but the NMS is interested in this information. Therefore if the node is still syncing up with the NMS, an alarm will be displayed for this node in the alarm list, but there will not be such an alarm on the platform. The following is a summary of alarms that may be seen in the alarm list, but not on the platform.
Step 1 If there is an alarm in the alarm list, and not on the platform, see if the suspect alarm is included in any of the following lists:
•Node sync-up status alarms
•Node database sync-up status alarms
•Node Management State status alarms
•Node Aggregate alarm status
•Link0/Link1 Node alarms
•Card sync-up status alarms
•Aggregate Port (Connection) alarms
Step 2 If the suspect alarm is not included in any of the alarm lists mentioned in step 1, it may be a defect. Perform the following:
a. Verify whether the database has the correct alarm state. See the Cisco Transport Manager Release 7.0 Database Schema for information on what table to look up for this particular entity type.
b. Collect the following information:
–topod.log, linktopoc.log, ILMITopoc.log, NMServer.log, and fileTopoc.log
–nmControl.dump
–CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
K.12.5.5.6 Transient Event Has Disappeared Unexpectedly
Transient alarms behave somewhat differently depending on whether the entity is managed by the network element or the NMS itself. If the transient alarm is on a managed network element—for example, an alarm generated by an FTP transfer failure—then the alarm is self-clearing. This means if the same transient alarm happens more than twice, the previous active alarm is cleared by the latest.
If the transient alarm is not a managed network element, but rather the NMS itself, then the alarm is not self-clearing and there will be multiple occurrences of the same alarm conditions. Since these NMS alarms (or events) that pertain to the NMS are not cleared by the NMS, NMS alarms are not correlated. They must be cleared manually by the operator. If these events are not manually cleared, the list will grow only to the value of MAX_ACTIVE_NMS_EVENTS that is specified in the NMServer.conf file. Once the list of NMS alarms reaches MAX_ACTIVE_NMS_EVENTS, NMServer will automatically clear the oldest alarms in the list to make room for the new events. The number of events that will be cleared each time the maximum is reached is also specified in NMServer.conf. That configuration parameter is called EVENTS_TO_CLEAR_WHEN_MAX_REACHED.
If there was a transient event that has unexpectedly disappeared, it is most likely because NMServer purged the event.
Step 1 Filter the alarm list on CTM alarms.
Step 2 Check the alarm count and compare it with the MAX_ACTIVE_NMS_EVENTS value in NMserver.conf. If the alarm count is close to the MAX_ACTIVE_NMS_EVENTS, then this explains why the transient event has been purged.
Step 3 If the alarm count for NMS alarms has not reached the MAX_ACTIVE_NMS_EVENTS, the missing event may have been manually cleared. The NMServer log will indicate whether the event was manually cleared.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log, NMServer.log, and CMSCclient.log
•Capture NMServer dump, use nmControl
Possible alternative workaround—Restart the Alarm List GUI.
K.12.5.5.7 Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port
The tree view in CTM client GUIs displays network elements from the top-level physical view down to the port. Alarm severities are aggregated from children up to parents. Since the port is at the lowest level of the tree, the question that often arises is how a port can have an aggregate alarm if it has no children. The answer to this question is that connections are virtual entities under ports in the tree view. Virtually, you will not see connections in the tree. But connection alarms are aggregate up to the port in the tree view.
Step 1 Compare the connection alarms with the aggregate port alarms:
a. In the Configuration Center, click the Connections tab.
b. Find the port in the tree view and drag and drop it to the right window pane. The Connection view window appears.
c. Click the Get button. The connections should be listed, and the alarm status should match the alarm status in the alarm list.
Step 2 If the connection alarms do not match the aggregate port alarm(s) in Alarm List GUI, there may be a defect. Collect the following defect information:
•CMSCclient.log, NMServer.log, sdbroker*.log, xdbroker*.log
•Capture NMServer dump, use nmControl
Possible alternative workaround—None.
K.12.6 Chassis View Problems
This section includes the following information:
•Chassis View Basics
•Lines Not Displayed in the Chassis View
•Card Not Displayed in the Chassis View
•DCA/DCB Status Not Displayed on PXM Cards
•Ethernet Status Not Updated on PXM Cards
•HIST, CBRX, and CBTX Status Not Updated in MGX Nodes
•RPM Card Status Not Updated
•RPM Secondary Card Status Is Blue
•Lines Not Displayed on Secondary Card
•Lines Not Selectable
•Wrong Tooltip Is Displayed
K.12.6.1 Chassis View Basics
Chassis View is a read-only application that provides the physical view of WAN devices. For a specific node, it displays node, cards, and lines. Chassis View does not display ports. It can handle the following events:
•Node status changes
•Card status changes
•Line status changes
The alarms handled by Chassis View are:
•Card alarm
•Line alarm
Chassis View shows an empty slot in the chassis when displaying unsupported cards. For a reserved card, it shows the lines, but they are disabled. Chassis View does not show lines under a card when the card is in another state, other than active or standby.
Chassis View displays the chassis by combining static data from the XML file and dynamic data from the database. The first step in debugging is to determine if data is populated properly in the database. The next step is to check that the card or line is defined in the XML file.
K.12.6.2 Lines Not Displayed in the Chassis View
Step 1 Determine if data about the lines are populated for the corresponding lines in the line table in the database.
Step 2 Check whether the lines are defined in the XML file Chassis View.xml.
Step 3 Make sure that the line numbers in the database start from 1 and not from 2 or above.
Step 4 Make sure that the node is in sync (mode is 3).
If the details are not found in the database, this probably is an EM issue. If the details are found, then proceed as follows:
a. Use cwmver to get the CTM version.
b. Get the node table information for the node using the following command; then, save it in a file:
echo "select (*) from the node, where node_id=<NodeId> " | dbaccess
c. Get the card table information for the card using the following command; then, save it in a file:
echo "select (*) from the card, where node_id=<NodeId> and slot=<Slot>" | dbaccess
d. Get the line table information for the line using the following command; then, save it in a file:
echo "select (*) from the line, where node_id=<NodeId> and slot=<Slot>" | dbaccess
e. Save the log as CMSCclient.log to your local drive.
f. Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. You need to check that gif files used to draw the lines are available in the jar.
Possible alternative workaround—Select the lines from the tree view to launch other applications.
K.12.6.3 Card Not Displayed in the Chassis View
Step 1 Check that entries for the card are available in the card table.
Step 2 Check whether the lines are defined in the XML file ChassisView.xml.
Step 3 Make sure that the node is in sync (mode is 3).
If the details are not found in the database, this probably is an EM issue. If the details are found, then proceed as follows:
a. Use cwmver to get the CTM version.
b. Get the node table information for the node using the following command; then, save it in a file:
echo "select (*) from the node, where node_id=<NodeId> " | dbaccess
c. Get the card table information for the card using the following command; then, save it in a file:
echo "select (*) from the card, where node_id=<NodeId> and slot=<Slot>" | dbaccess
d. Save the log as CMSCclient.log to your local drive.
e. Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. You need to check that gif files used to draw the lines are available in the jar.
Possible alternative workaround—Select the cards from the tree view to launch other applications.
K.12.6.4 DCA/DCB Status Not Displayed on PXM Cards
DCA/DCB status is always grayed out in Chassis View. DCA/DCB status information is not received from the switch, so the status is not displayed or updated.
K.12.6.5 Ethernet Status Not Updated on PXM Cards
Ethernet status is received only when the Chassis View is launched. Dynamic changes in states of the Ethernet status are not updated in Chassis View.
K.12.6.6 HIST, CBRX, and CBTX Status Not Updated in MGX Nodes
For MGX nodes, only CPUOK LED status is updated in Chassis View. Chassis View does not manage the LEDs for HIST, CBRX, or CBTX.
K.12.6.7 RPM Card Status Not Updated
Dynamic event updates are not generated for RPM cards on MGX PXM1-based nodes, so the Chassis View does not get event updates on hot insertion or removal of RPM cards. The card will be identified when coldstart is done.
K.12.6.8 RPM Secondary Card Status Is Blue
For RPM cards, the Standby state shows the card status in blue because the card has only one LED (CPUOK) to show the status of the card, unlike the other cards. For other types of cards, the Standby state is indicated by a yellow LED.
K.12.6.9 Lines Not Displayed on Secondary Card
If two cards are in a redundancy relationship, the primary card (for example, the logical slot) is used to display the children and for all provisioning and troubleshooting activities, even if the primary slot becomes a standby. The secondary slot does not show any children under it, even if it becomes active. Hierarchical views in all applications behave in this manner. Similarly, provisioning is allowed only on the working line of an APS pair regardless of whether or not that line is currently active. However, monitoring is done on both working and protection lines.
K.12.6.10 Lines Not Selectable
Sometimes lines become unselectable. For example, when trying to select line 3, line 1 may get selected or vice versa.
Lines must be spaced sufficiently apart. If they are not, they may overlap, causing them to become unselectable.
Defect Information—Collect the following information for further analysis:
•Get the copy of the ChassisView.xml file used.
•Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. Verify that gif files were used to draw the lines.
Possible alternative workaround—Try selecting from the tree view.
K.12.6.11 Wrong Tooltip Is Displayed
When you move the cursor over a line, the wrong tooltip is displayed. For example, when moving the cursor below the last line in the card, the tooltip of the line gets displayed instead of the tooltip of the card.
Lines must be spaced sufficiently apart. If they are not, they may overlap, causing the wrong tooltip to be displayed.
Defect Information—Collect the following information for further analysis:
•Get the copy of the XML file used: ChassisView.xml for MGX nodes and BPXIGX.xml for BPX/IGX nodes.
•Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. Verify that gif files were used to draw the lines.
Possible alternative workaround—Try selecting from the tree view.
K.12.7 Configuration Center Management
The Configuration Center management functions are divided into the following categories:
•Network elements—Manages the nodes and their components. For network element management, the Configuration Center communicates with the Config Server process.
•Connections—Manages the connections between the nodes. For connection management, the Configuration Center communicates with the Connection Management (CM) Server process.
This section describes troubleshooting guidelines for the element management (Configuration Center; Elements Tab) section. The Connection Management (Configuration Center; Connections Tab) section describes troubleshooting guidelines related to Connection Management.
This section includes the following information:
•Configuration Center Framework
•Configuration Center—Element Management
K.12.7.1 Configuration Center Framework
The Configuration Center uses the CTM framework and workflow mechanism to launch applications and drag and drop objects across application.
•Network elements can be selected in the tree view and the Configuration Center (Elements Tab) can be launched to view or modify the selected object.
•Network elements can be dragged and dropped from other application to the Configuration Center's Elements tab for modification.
This section includes the following information:
•Cannot Launch the Configuration Center
•Cannot Launch Other Applications from the Configuration Center
•An Exception Is Raised when the Configuration Center Is Launched
•An Exception Is Raised when the Configuration Center Launches Another Application
•Element Tab—Internal Frame Does Not Launch
•Element Tab—Drag-and-Drop Functionality Does Not Launch An Internal Frame
•Element Tab—Create, Details, Modify, and Refresh Button Issues
•Element Management—Drag and Drop Within Configuration Center
•Cross Application—Configuration Center as Drag Source
•Cross Application—Configuration Center's Element Tab as Drop Target
•Element Tab—Internal Frame Displays Incorrect Object or Object Data
•Configuration Center's Element Tab Does Not Respond (GUI Is Grayed Out)
K.12.7.1.1 Cannot Launch the Configuration Center
The Configuration Center cannot be launched using one of the following methods:
•Click the Configuration Center icon from the Launch Center or from any application.
•Choose Tools > Configuration Center from any application.
•Right-click the selected node from the hierarchical tree, and choose
