Table Of Contents
Troubleshooting
J.1 Overview
J.2 Server Problems
J.2.1 Conditions that Affect CTM Server Performance
J.2.2 CTM Server Does Not Respond
J.2.3 Cannot Connect to the CTM Server
J.2.4 NE Connection State Is Listed as Unavailable
J.2.5 Launching Tables Results in Database Errors
J.2.6 SNMP Traps Are Not Forwarded from NEs
J.2.7 Trap Port Is Unavailable
J.2.8 Problems with ONS 15530 and ONS 15540 Configuration
J.2.9 Performance Monitoring Data Is Not Displayed
J.2.10 CTC-Based NE Is Not Discovered
J.2.11 CTC-Based NE Is Not Reachable
J.2.12 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable
J.2.13 Circuits Are Not Displayed
J.2.14 Circuit State for Monitor Circuits Reads "Duplicate ID"
J.2.15 NE Model Type Appears as Unknown
J.2.16 Cannot Copy CTC Binary to the CTM Server
J.2.17 Memory Backup, Memory Restore, or Software Download Fails
J.2.18 Memory Autobackup, Software Commit, or Software Revert Fails
J.2.19 The getinfo.sh Script Fails
J.2.20 Using the get nestate Command
J.3 Client Connectivity Problems
J.3.1 Database Is Not Available
J.3.2 Database Timeout Occurs
J.3.3 Are the CTM Client and the CTM Server Connected?
J.3.4 Cannot Log In as Provisioner or Operator
J.3.5 Cannot Authenticate User Message Appears
J.4 Client Operational Problems
J.4.1 Model Index Is Unknown
J.4.2 Cannot Delete an NE
J.4.3 CTC Fails to Start
J.4.4 Added a New Software Version to the Wrong NE
J.4.5 Cannot Delete a Subnetwork
J.4.6 Cannot Move an NE Between Subnetworks
J.4.7 Cannot Schedule Jobs
J.4.8 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms
J.4.9 Bandwidth-Intensive Operations Are Blocked
J.4.10 NE Displays Incorrect Configuration Management Data
J.4.11 Cannot Customize the Network Map
J.4.12 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation?
J.4.13 How Do I Collect Thread Dumps?
J.4.14 How Do I Enable or Disable the Automatic Refresh Data Action?
J.4.15 How Do I Replace the Alarm Interface Panel?
J.5 Client Debug Messages
J.6 CTM GateWay/TL1 Problems
J.7 CTM GateWay/CORBA Problems
J.8 Cisco CRS-1 and Catalyst 6509 Problems
J.8.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service
J.8.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service
J.8.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1
J.8.4 CTM Does Not Receive Traps from the Catalyst 6509
J.8.5 CTM Does Not Receive syslog Events from the Catalyst 6509
J.8.6 CTM Does Not Collect Historical PM Data for the CRS-1
J.8.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval
J.8.8 Memory Backup or Restore Fails for the CRS-1
J.8.9 How Do I Back Up the CRS-1 Configuration File Automatically Whenever the Configuration Is Changed on the Router?
J.8.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs
J.8.11 Cannot Delete an Out-of-Service NE from CTM
J.8.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty
J.8.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer
J.8.14 How Do I Configure Telnet on the CRS-1?
J.8.15 How Do I Configure SSH on the CRS-1?
J.8.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs?
J.8.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509?
J.8.18 How Do I View the SNMP Configuration on the Catalyst 6509?
J.8.19 How Do I Load a Software Image on the Catalyst 6509?
J.9 XR 12000 Problems
J.9.1 Reporting XR 12000 Issues
J.9.2 CTM Cannot Discover an XR 12000 Node
J.9.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer
J.9.4 XR 12000 Alarm and Event Reporting Issues
J.9.5 XR 12000 PM Collection Issues
J.9.6 XR 12000 BGP and CDP Neighbor Discovery Issues
J.9.7 XR 12000 NE Explorer Applications Are Disabled
J.10 Problems with MGX Voice Gateway Devices
J.10.1 Discovery Mechanism
J.10.2 Discovery Issues at Startup
J.10.3 Discovery Issues at Runtime
J.10.4 Equipment Management Problems
J.10.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems
J.10.6 Chassis View Problems
J.10.7 Configuration Center Management
J.10.8 Connection Management Problems
J.10.9 Diagnostics Center Problems
J.10.10 Performance Management Collection and Parsing Problems
J.10.11 Statistics Report Problems
J.10.12 Service Agent Problems
J.10.13 Miscellaneous Problems
Troubleshooting
This chapter offers troubleshooting steps to help solve high-level problems while operating CTM or CTM GateWay. Follow the troubleshooting procedures described in this chapter before contacting Cisco technical support.
This chapter includes the following troubleshooting information:
•
Overview
•Server Problems
•Client Connectivity Problems
•Client Operational Problems
•Client Debug Messages
•CTM GateWay/TL1 Problems
•CTM GateWay/CORBA Problems
•Cisco CRS-1 and Catalyst 6509 Problems
•XR 12000 Problems
•Problems with MGX Voice Gateway Devices
Note For information about troubleshooting CiscoView problems, see I.9 Troubleshooting, page I-6 in Appendix I, "Using CiscoView to Configure and Monitor ONS 15501, ONS 15530, and ONS 15540 NEs."
J.1 Overview
Troubleshooting involves:
1. Identifying the source of the problem—Which devices, links, interfaces, hosts, or applications have the problem?
2. Locating the problem on the network—On what VLAN, subnet, or segment is the problem occurring?
3. Comparing current network performance against an established baseline—Is the performance better or worse?
4. Finding out when the problem started—When did you first see the problem? Is it recurring?
5. Determining the extent of the problem—How widespread is the problem? Is it getting worse?
Note This chapter assumes that the server is installed under the default /opt/CiscoTransportManagerServer directory and the client is installed under the default /opt/CiscoTransportManagerClient (Solaris) or C:\Cisco\TransportManagerClient directory (Windows). If a directory other than the default installation directory is specified, replace the default path with the installed path.
J.2 Server Problems
This section describes troubleshooting procedures for CTM server-related problems.
Note Log in as the root user on the Solaris workstation where the CTM server is installed to perform any operations on the Solaris workstation.
J.2.1 Conditions that Affect CTM Server Performance
The "System Requirements" chapter in the Cisco Transport Manager Installation Guide lists server, CPU, CPU speed, disk space, and RAM requirements for different types of CTM installations. These requirements are derived from guidelines based on scalability simulation testing.
Actual CTM server performance varies for each customer and is affected by:
•The total number of NEs actively managed by CTM
•The number of NE-related services (including northbound services) running on the server
•The rate of circuit provisioning and the methods used for provisioning
•The rate of network growth (for example, adding 50 NEs to CTM at once)
•The rate of configuration change updates received from the NEs
•The number of circuits provisioned with no signals that generate threshold crossing alerts (TCAs)
•The number of unacknowledged alarms in the CTM database
•The rate of alarm bursts received from the NEs
•The actual number of circuits provisioned in the CTM database
The following conditions might indicate that your CTM installation is near or at capacity:
•Your daily CPU utilization is consistently above 80%
•You experience loss of connectivity (LOC) to 1% to 5% of nodes at random
•Your RAM usage is 100% or your system swaps frequently
•You receive many configuration updates from the NEs
•Your system experiences high Java garbage collection time
Note The Java garbage collection time is determined by engineering after analyzing your CPU utilization data.
If you are experiencing any of these conditions, contact your account representative. Your account representative can engage Cisco's Advanced Services team to audit your CTM server and recommend a higher-capacity server and resources.
J.2.2 CTM Server Does Not Respond
Step 1 Log in as the root user on the Solaris workstation where the CTM server is installed.
Step 2 Enter the following command to view the status of the CTM server processes:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
If there is a line containing /CTMServer, the CTM server is running.
If there is no line containing /CTMServer, the CTM server is not running. Proceed to Step 3.
Note You can also check the ctmop.log file in /opt/CiscoTransportManagerServer/log to determine if the server was stopped by another user or if it stopped abnormally. If it stopped abnormally, proceed to Step 3.
Step 3 Run the getinfo.sh CTM server tool and send the data to Cisco technical support for analysis.
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 4 Start the CTM server by using the ctms-start script in the /opt/CiscoTransportManagerServer/bin directory.
a. Log in as the root user.
b. Change the directory to /opt/CiscoTransportManagerServer/bin and enter the following command:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
If the preceding procedure does not solve the problem, complete the following steps:
Step 1 Verify that the /opt/CiscoTransportManagerServer/cfg/CTMServer.cfg file is not corrupted. The file should contain the db-config-mode = auto parameter in the [database] section. If the entry is missing, the CTM server configuration file is corrupt. Reinstall the CTM server.
Step 2 Verify that the first entry in the /var/opt/oracle/oratab file looks similar to CTM6_0:/oraclesw9i/product/9.2:Y. If this entry is missing, the Oracle database might not be installed. The Oracle database is a prerequisite for installing the CTM server.
J.2.3 Cannot Connect to the CTM Server
Step 1 Ping the server's IP address from the client PC or workstation.
Step 2 If the ping fails, resolve the IP connectivity problem and try again.
Step 3 Telnet to the server and log in as the root user.
Step 4 Enter the following command to verify that CTM is running:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 5 The server should have at least the following four processes running:
root 3778 0.1 0.1567 9592 ? S 16:57:36 0:00 /opt/CiscoTransportManagerServer/bin/CTMServer
root 3771 0.1 0.4 6208 pts/1 S 16:57:34 0:00
/opt/CiscoTransportManagerServer/bin/CTMServer
root 3876 0.5 0.6129464 8648 ? R 16:58:12 SnmpTrapService
root 3798 26.8 4.115732060968 ? S 16:57:37 0:29 SMService
Step 6 Manually stop the server if you see fewer than four processes running. Enter the following command:
Step 7 If you changed the server IP address, verify that the configuration files shown in Table 4-13 on page 4-44 have been updated. See 4.4.2 Updating the Configuration Files After Changing the CTM Server IP Address, page 4-44.
Step 8 Verify that the Oracle database is accepting connections:
a. Enter the following command to log in as the Oracle user:
b. Enter the following command to open an SQL*Plus session:
omu-u60-3% sqlplus ctmanager/ctm123!
Step 9 Reboot the server if you receive another error message and the SQL prompt does not appear. Wait enough time for the server to boot up and try to run the client. The SQL prompt indicates that the Oracle database is running and accepting connections.
Step 10 Enter the following commands to manually restart CTM:
If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:
Step 11 Wait for 5 minutes and run the client.
J.2.4 NE Connection State Is Listed as Unavailable
If the connection state of an NE is listed as Unavailable in the Domain Explorer window, a connectivity or configuration problem exists. Wait 5 to 10 minutes after adding the NE to the CTM domain; then, complete the following steps:
Step 1 To see the NE IP address, select the NE in the Domain Explorer window. The Address tab of the Network Element Properties pane lists the IP address of the selected NE.
Step 2 From the CTM server, enter the following command to verify connectivity between the CTM server and the NE:
Step 3 If the ping fails, a physical or configuration problem exists in the data communications network (DCN).
a. If connectivity existed earlier:
•If there were no configuration changes made to any routers in the DCN or to the CTM server, the problem is a physical problem.
•If the configuration was changed, the change might have introduced problems. Verify the changed configuration.
b. If connectivity was never established:
•Verify that there are no problems on the physical level.
•Verify the DCN configuration.
Step 4 If the ping succeeds, verify that the NE software version is listed in the Supported NE Table. See 4.3.7 Adding a New NE Software Version to the CTM Domain, page 4-37.
Step 5 For ONS 15501, ONS 15530, and ONS 15540 NEs, SNMP read and write community strings must be set up properly on each NE. The running configuration on each NE should contain entries similar to the following example:
snmp-server community public RO
snmp-server community private RW
The community strings configured on the server should match the strings configured on the NE. For information on configuring community strings in CTM for ONS 15501, ONS 15530, and ONS 15540 NEs, see 3.5.1.4 Prerequisites for Adding ONS 15501, ONS 15530, and ONS 15540 NEs, page 3-10.
J.2.5 Launching Tables Results in Database Errors
If the Oracle database or Oracle listener that CTM is using is down, launching tables will generate database errors. To troubleshoot table launching errors:
Step 1 Log in as the Oracle user and enter the following command:
sqlplus ctmanager/ctm123!
If the login is successful, an SQL> prompt appears, which indicates that the Oracle database has been installed and the database server is up and running. If login fails, either the Oracle database has not been installed or the database server is not running.
Step 2 To start the Oracle database, log into the Solaris workstation as the Oracle software owner user.
Step 3 Enter the following command at the shell prompt to start the Oracle database:
Step 4 Enter the following command at the shell prompt to start the Oracle listener:
If there are still problems with starting the Oracle database or the Oracle listener, refer to the Oracle documentation or contact Oracle support.
J.2.6 SNMP Traps Are Not Forwarded from NEs
SNMP traps might not be forwarded, either because the trap port is already in use, or because the NE is not properly configured.
J.2.7 Trap Port Is Unavailable
The CTM server requires exclusive access to the SNMP trap port to receive SNMP traps from the NE.
Step 1 To verify that the standard SNMP trap port (number 162) is not being used by another application running on the same Solaris workstation, enter the following command:
If the following line is seen, the SNMP trap port is being used by another application:
Step 2 If the trap port is being used by another application, stop the other application.
J.2.8 Problems with ONS 15530 and ONS 15540 Configuration
ONS 15530 and ONS 15540 NEs must be properly configured to send traps to the server.
Step 1 The running configuration on the NE should contain entries similar to the following example:
snmp-server enable traps snmp authentication warmstart
snmp-server enable traps bgp
snmp-server enable traps oscp
snmp-server enable traps config
snmp-server enable traps entity
snmp-server enable traps fru-ctrl
snmp-server enable traps topology throttle-interval 60
snmp-server enable traps optical monitor min-severity not-alarmed
snmp-server enable traps rf
snmp-server enable traps aps
snmp-server enable traps patch
snmp-server enable traps cdl all
snmp-server enable traps alarms
snmp-server enable traps threshold min-severity degrade
snmp-server host 172.20.126.214 version 2c public snmp bgp oscp
config entity fru-ctrl topology optical rf aps patch cdl alarms threshold
In this example, the host 172.20.126.214 will receive the specified traps from the NE.
Step 2 You can also run the following commands on the server and device:
a. As the root user, enter the following command on the server:
b. On the device, enter the following command to generate a configuration change:
If the device is configured correctly, the snoop command reports a configuration change trap.
J.2.9 Performance Monitoring Data Is Not Displayed
Step 1 Use the Domain Explorer NE Properties pane to see if PM collection is disabled for the selected NE.
Step 2 Choose Administration > Control Panel and expand PM Service. Select the NE type to see if the service status is Active or Not Active.
Step 3 Choose Administration > Audit Log to see if PM was ignored for the selected NE.
Step 4 Choose Administration > Control Panel and click PM Service. Look at the PM Data Storage field. Choose Optimized to save only nonzero values in the database. Choosing Optimized requires less space than saving all values, including zero values. If you choose Normal in the PM Data Storage field, the database saves all PM values, including zero values.
Note The Optimized PM data storage feature is available only for CTC-based and ONS 155xx NEs.
Step 5 Choose Administration > Error Log to see if there are any critical, major, or minor errors against the PM service.
Step 6 Verify that the NE is reachable during the duration that PM data is collected. If the NE is not reachable, there is an entry in the Audit Log table (Administration > Audit Log).
J.2.10 CTC-Based NE Is Not Discovered
Step 1 Verify that the NE is up and running.
Step 2 Verify that the NE IP address and default route are configured correctly.
Step 3 To verify that the NE is running the correct version of system software, open a CTC session to the NE and check the NE software version.
Step 4 Verify that the NE that acts as the gateway NE (GNE) (through which this node is discovered) is added to CTM and is available.
Step 5 Open CTC from the GNE and see if CTC can discover the NE.
J.2.11 CTC-Based NE Is Not Reachable
Step 1 Verify that the NE is up and running.
Step 2 Verify that the NE IP address and default route are correctly configured.
Step 3 Wait for five poll cycles while CTM reestablishes connectivity with the NE.
Step 4 To test IP connectivity to the NE from the CTM server, enter the following command from the Solaris workstation running CTM:
Step 5 To verify that the NE is running the version of system software supported by CTM, open a CTC session to the NE and check the NE software version.
Step 6 Verify that the username and password that CTM uses to reach the NE exist on the NE.
J.2.12 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable
Step 1 Enter the following command on the server to verify IP connectivity between the server and the NE:
Step 2 If the NE is reachable through IP, verify that the correct read community string is configured on the server.
•On the server, choose Administration > ONS 155XX > ONS 155XX SNMP Settings Table to view the community strings set in CTM.
•On the NE, enter the following command to view the read community string set on the NE:
•In the running configuration, the community strings are shown in entries similar to the following:
snmp-server community public RO
snmp-server community private RW
In this example, the read string is public, and the read/write string is private.
Step 3 If IP connectivity and community strings are verified and the device is still unavailable, check the error log for any errors reported by ONS 155xx NE Service.
J.2.13 Circuits Are Not Displayed
If CTM fails to display some circuits or if the information displayed by CTC and CTM differs, complete the following steps:
Step 1 Wait for two minutes while the CTM server synchronizes with the NE. If the Circuit table is not in autorefresh mode, click the Refresh Data tool when the tool flashes.
Step 2 Verify that the Circuit table is opened from either the source or destination NE of the circuit to be viewed.
Step 3 Verify that the NEs that are part of the circuit (source, destination, or drop) are available in CTM.
Step 4 Open another CTC session and compare the circuit information in the newly opened CTC session with the circuit information displayed by CTM.
J.2.14 Circuit State for Monitor Circuits Reads "Duplicate ID"
CTM generates a unique number that is appended to the circuit name to make the monitor circuit name unique. On rare occasions, CTM might create two or more monitor circuits with the same name. The circuit state for these circuits reads "Duplicate ID." This is a known issue that has been tracked as DDTS number CSCdz87566.
When the circuit state reads "Duplicate ID," you cannot see the actual circuit state. In this case, you must change the duplicate name to a unique name, so that you can see the correct circuit state. Use the Modify Circuit wizard to enter a unique name for each monitor circuit.
J.2.15 NE Model Type Appears as Unknown
If an in-service NE is added to the Domain Explorer, but the model type appears as unknown, the software version of the NE might not be prepopulated in the database. In other words, CTM cannot match the NE with a recognizable version.
Use the NE Software Table to add the NE software version string to the CTM database. See 4.3.5 Viewing Software Versions and Restarting the NE with a New Software Image, page 4-35.
J.2.16 Cannot Copy CTC Binary to the CTM Server
If a CTC binary fails to be copied to the CTM server, the <CTM_install_directory>/cms/ directory might be missing or write-protected. If the directory is missing, create a cms directory with write-access permissions. Otherwise, change the permissions of the existing cms directory to allow write access.
J.2.17 Memory Backup, Memory Restore, or Software Download Fails
Step 1 In the Domain Explorer window, choose Administration > Job Monitor. The Job Monitor table shows the status of the operation. The reason for the failure is shown in the Additional Information column.
Step 2 Return to the Domain Explorer window and choose Administration > Error Log. The Error Log table shows information about the backup, memory restore, or download failure.
Step 3 See Appendix H, "Error Messages" for the correct action to take in response to the error.
J.2.18 Memory Autobackup, Software Commit, or Software Revert Fails
Step 1 In the Domain Explorer window, choose Administration > Audit Log. The Audit Log table shows the status of the operation.
Step 2 Return to the Domain Explorer window and choose Administration > Error Log. The Error Log table shows the reason for the failure.
Step 3 See H.2 CTM Server Error Messages, page H-94 for the correct action to take in response to the error.
J.2.19 The getinfo.sh Script Fails
You might experience a problem where the getinfo.sh script fails with the following error:
"Error: permission denied while running `rsh.' Check the environment and make sure the root is permitted to run .rsh on host <IP_address>."
This problem occurs because the getinfo.sh script does not run unless an entry is made in the /.rhosts file. This problem occurs even on a local server and database configuration. This is a known issue that has been tracked as DDTS number CSCin66495.
The solution is to add an entry for the server IP address to the /.rhosts file.
J.2.20 Using the get nestate Command
The get nestate command-line interface (CLI) command available in the ctm utility in the EMS server bin directory returns the provisioned operational state (NE_Info_Table: NEState) and the read-only modifiers to the operational state (NE_Info_Table: NEDiscoveryState). The get nestate command returns the following possible values:
•Preprovisioned
•In Service
•Out of Service
•Under Maintenance
•In Service-Initializing
•In Service-Sync Configuration
In addition, the show nelist command lists all of the NEs with the following information:
•NE database ID
•NE IP address
•Model type
•SID
•Network partition ID
•Operational state
J.3 Client Connectivity Problems
The CTM client might not be able to connect to the CTM server for various reasons. Complete the following procedures in the order listed until the problem is resolved.
J.3.1 Database Is Not Available
If the CTM client cannot connect to the CTM server, verify that the database is available.
Step 1 Log into the CTM server as the Oracle user.
Step 2 Enter the following command to connect to the database:
sqlplus ctmanager/ctm123!
Step 3 If the error message "maximum processes exceeded" is received, the maximum number of database connections has been reached. Close several clients or ask the database administrator to increase the maximum number of processes for the database.
J.3.2 Database Timeout Occurs
Step 1 Reduce the scope of the query by selecting a group or an NE and not the entire domain before opening tables such as the Alarm Browser, Alarm Log, Audit Log, and PM tables.
Step 2 Increase the client database query timeout period by editing the ems-client.cfg file in the C:\Cisco\TransportManagerClient\config or /opt/CiscoTransportManagerClient/config directory.
Step 3 Add hardware resources to the Oracle database server.
Step 4 Ping the Oracle database server from the client to verify the response time. Increase the bandwidth if the round-trip response time is inadequate.
J.3.3 Are the CTM Client and the CTM Server Connected?
If the database is available, check connectivity between the client and the server.
Step 1 To see the CTM server IP address, enter the following command on the Solaris workstation that is running the CTM server:
The command output looks similar to the following example:
hme0:flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST>mtu 1500
inet 192.168.120.93 netmask ffffff00 broadcast 192.168.120.255
The IP address is the address following the inet field.
Step 2 To verify that the physical connection between the CTM client and the CTM server does not have problems, enter the following command from the CTM client:
where the IP address belongs to the Solaris workstation that runs the CTM server.
Step 3 If the ping command is not successful, fix the physical connectivity; then, log into the CTM client.
J.3.4 Cannot Log In as Provisioner or Operator
Step 1 Use the default username and password to log into the CTM client:
Username: SysAdmin
Password: Ctm123!
Step 2 If the SysAdmin login is successful, but the provisioner or operator login fails, verify that the provisioner or operator exists and is not disabled. In the Domain Explorer window, choose Administration > CTM Users to view a table of all configured CTM users.
Step 3 If the provisioner or operator is not in the CTM Users table, the user is not configured. Configure the provisioner or operator; then, log in as that user.
Step 4 If the provisioner or operator is in the CTM Users table, select the row corresponding to that user and click the Modify User Properties tool to bring up the Modify CTM User Properties wizard. Verify that the Login State field is set to Enabled. If the login state is disabled, enable it and log in as the provisioner or operator. The user might have been disabled after attempting to log in with an incorrect password.
Step 5 If the password is not correct, set a new password for the user and log in again.
J.3.5 Cannot Authenticate User Message Appears
If the "Cannot authenticate user" error message is received when logging into the CTM client, the CTM server might be initializing. Wait for five minutes while the CTM server finishes initializing; then, try to log in again. Alternately, check your username and password and enter them again. The username and password are case sensitive.
J.4 Client Operational Problems
This section describes troubleshooting procedures for CTM client operational problems.
J.4.1 Model Index Is Unknown
If a new NE is added to the CTM domain and marked as In Service, but the model index is unknown, complete the following steps:
Step 1 Verify that the IP address and other parameters are entered correctly.
Step 2 The CTM server might not be able to establish connectivity to the GNE for the NE. Verify that the GNE for the NE is marked as In Service, and that the GNE is correctly populated in the CTM domain.
Step 3 Verify that the GNE is DCC-connected to the NE.
Step 4 The CTM server might not recognize the NE's software version. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity. If the NE is available, contact Cisco Systems and verify that the version of the software on the NE is compatible with the CTM server being used. Use the Supported NE Table to provide the new NE software version that the CTM server will recognize.
Step 5 The CTM server might be configured to attempt communication with new NEs after a certain delay. To change the health poll frequency:
a. In the Domain Explorer window, choose Administration > Control Panel and click NE Service.
b. In the NE Poller tab, look at the NE Health Poll Interval field. The default is 240 seconds. If the value is too large, there might be a communication delay between CTM and the NEs. Under normal conditions, the CTM server attempts communication with new NEs for no more than twice the number of seconds specified in the NE Health Poll Interval field.
J.4.2 Cannot Delete an NE
If an NE cannot be deleted, verify that the user logged in as SuperUser or NetworkAdmin—not as a Provisioner or Operator. Provisioners and Operators cannot delete NEs. If the user logged in as Provisioner or Operator, restart the CTM client session and log in as SuperUser.
J.4.3 CTC Fails to Start
If CTC cannot be started for a CTC-based NE, complete the following steps:
Step 1 If an error occurs while opening the NE Explorer yet the CTC progress screen and login dialog boxes appear, the CTC username or password might be incorrect.
a. Choose Administration > CTM Users.
b. In the CTM Users table, choose Edit > Modify. The Modify CTM User Properties wizard opens.
c. Click Next to move to the CTC/Craft User Properties panel and change the CTC username and password to match those configured on the device. Click Finish.
Step 2 If the CTC progress screen idles for a long time before the login dialog box appears, there might be problems with network connectivity to the device.
a. From a DOS command window or a Sun Solaris terminal, ping the device.
b. If no response is received, set up routes so that the device is available; then, restart CTC. If CTC does not start, the workstation might have resource constraints.
c. Close some open applications and try again.
d. Close some instances of CTC that are managing another group of CTC-based NEs and try again.
Step 3 If the GNE for the selected NE was not found, there might be a mismatch between the GNE specified for the device and the available GNEs. This means that the data in the database is corrupt. Contact Cisco technical support for assistance.
J.4.4 Added a New Software Version to the Wrong NE
Step 1 In the Domain Explorer window, choose Administration > Supported NE Table.
Step 2 In the Supported NE Table, select the incorrect entry and choose Edit > Delete.
Step 3 Click OK in the confirmation dialog box.
Step 4 Select the correct NE row from the Supported NE Table.
Step 5 Add the correct software version.
Step 6 In the Domain Explorer > Network Element Properties pane, set the operational state of all NEs that are behaving erroneously to Out of Service.
Step 7 Click Save.
Step 8 In the Network Element Properties pane, set the operational state of all the NEs back to In Service.
Step 9 Click Save.
J.4.5 Cannot Delete a Subnetwork
If a subnetwork cannot be deleted from the Subnetwork Explorer, complete the following steps:
Step 1 Verify that the user logged in as a SuperUser—not as a Provisioner or an Operator. Provisioners and Operators cannot delete subnetworks. If the user logged in as a Provisioner or Operator, restart the CTM client session and log in as a SuperUser.
Step 2 Verify that the target subnetwork does not contain NEs. Move all NEs to another subnetwork before deleting the empty target subnetwork.
J.4.6 Cannot Move an NE Between Subnetworks
When automatic grouping of NEs in subnetworks is enabled, you cannot move NEs between subnetworks. To move NEs from one subnetwork to another:
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 In the Control Panel, click UI Properties.
Step 3 Under Subnetwork Grouping, uncheck the Automatically Group NEs in Subnetworks check box.
Step 4 Click Save.
J.4.7 Cannot Schedule Jobs
The CTM client can schedule three types of administrative tasks:
•Software download
•Memory backup
•Memory restore
CTM maintains an Error Log and Audit Log to track potential problems. To view the Error Log or Audit Log:
Step 1 In the Domain Explorer window, choose Administration > Audit Log or Error Log.
Step 2 Look for errors related to software download, memory backup, or memory restore.
J.4.8 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms
Occasionally, ONS 15800, ONS 15801, and ONS 15808 NEs generate so many alarms that the CTM client cannot process all of them. Set the operational state of the NE to Under Maintenance while debugging the cause of the alarms:
Step 1 In the CTM Domain Explorer window, right-click the ONS 15800, ONS 15801, or ONS 15808 that is generating the large number of alarms and choose Mark Under Maintenance.
Step 2 Debug the ONS 15800, ONS 15801, or ONS 15808 NE to see which card is in error. Find the card that is generating the alarms and mark it as Under Maintenance.
Step 3 While the card is under maintenance for debugging, set the ONS 15800, ONS 15801, or ONS 15808 NE back to In Service.
J.4.9 Bandwidth-Intensive Operations Are Blocked
Bottlenecks in the DCN affect bandwidth-intensive operations, such as software download. Tune the timeout and retry values to match DCN performance. Log into the NE and enter the following commands:
•ip tftp min-timeout—Sets the minimum wait time, in seconds, before retrying the read/write request.
•ip tftp max-timeout—Sets the maximum wait time, in seconds, before retrying the read/write request.
•ip tftp backoff-delay—Sets the number of seconds to extend wait time if the read/write request times out.
•ip tftp write-retries—Sets the maximum number of times to retry the TFTP read/write request before declaring a failure.
J.4.10 NE Displays Incorrect Configuration Management Data
Step 1 In the Domain Explorer window, choose Administration > Service Monitor and verify that the NE Service is running. If it is not running, start the NE Service.
Step 2 Test NE connectivity.
a. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity. The message "<NE_name> (<IP_address>) is available" should be generated.
b. If the NE is unavailable, establish connectivity to the NE before retrieving configuration data.
J.4.11 Cannot Customize the Network Map
If an image file is not displayed while changing the Network Map background or while changing a node icon, complete the following steps:
Step 1 Choose another image file. The file might be corrupt.
Step 2 Check the size of the image file. The image file might be larger than 100 KB, which is too big to load. If the file is too big, use a smaller image file.
Step 3 Verify that the image file exists in the \images\mapbkgnds\shapefiles directory or the \images\mapicons directory. If the file is missing, it has been deleted. Reinstall the CTM client.
Note The client is bundled primarily with shape files (*.shp) and only a few map background GIF files.
J.4.12 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation?
To run the client on a Sun Ultra 5 workstation, the color map must be changed from 8 bit to 24 bit. If the colors do not look right, change the color map from 8 bit (the default) to 24 bit as follows:
Step 1 To see the current color settings, enter the following command:
/usr/sbin/m64config -prconf
Command output looks similar to the following example:
--- Hardware Configuration for /dev/fbs/m640 ---
Card possible resolutions: 720x400x88, 640x480x60, 640x480x72, 640x480x75, 800x600x56,
800x600x60, 800x600x72, 800x600x75, 1024x768x87, 1024x768x60, 1024x768x70, 1024x768x75,
1280x1024x75, 1280x1024x60, 1152x900x66, 1152x900x76, 1280x1024x67, 1280x800x76,
1280x1024x85
1280x1024x76, 1152x864x75, 1024x768x77, 1024x800x84, vga, svga, 1152, 1280, 800x600,
1024x768, 1280x1024, 1152x900
Monitor possible resolutions: 720x400x70, 720x400x88, 640x480x60, 640x480x67, 640x480x72,
640x480x75, 800x600x56, 800x600x60, 800x600x72, 800x600x75, 832x624x75, 1024x768x60,
1024x768x70, 1024x768x75, 1280x1024x75, 1152x870x75, 1152x900x66, 1152x900x76,
1280x1024x67, 1280x1024x76, vga, svga, 1152, 1280, 800x600, 1024x768, 1280x1024, 1152x900
Current resolution setting: 1280x1024x76
Step 2 To change the color setting to 24 bit, log in as the root user and enter the following commands:
/usr/sbin/m64config -depth 24 -res 1152x900x76
Step 3 Reboot the workstation for the new color settings to take effect.
Note The resolution in 24-bit depth is a little lower than is possible with 8-bit depth (1152 x 900 x 76 versus 1280 x 1024 x 76), but the difference is hardly noticeable.
J.4.13 How Do I Collect Thread Dumps?
Thread dumps are helpful references when debugging the CTM process. To collect thread dumps:
Step 1 Log into the server workstation as the root user.
Step 2 On the command line, enter the following command:
thread_dumper [\<Group/Service>\]
where:
•Group is the group name for which to collect thread dumps. It can be:
–SM
–SNMPTRAP
–NE
–PM
–GW
•Service is the service name for which the thread dump is required. It can be:
–SMService
–SNMPTrapService
–CORBAGWService
–ONS15216NEService
–ONS15302NEService
–ONS15305NEService
–ONS15310NEService
–ONS15327NEService
–ONS15454NEService
–ONS15454SDHNEService
–ONS15501NEService
–ONS15530NEService
–ONS15540NEService
–ONS15600NEService
–ONS15600SDHNEService
–ONS15800NEService
–ONS15801NEService
–ONS15808NEService
–UnmanagedNEService
–ONS15216PMService
–ONS15302PMService
–ONS15305PMService
–ONS15310PMService
–ONS15327PMService
–ONS15454PMService
–ONS15454SDHPMService
–ONS15501PMService
–ONS15530PMService
–ONS15540PMService
–ONS15600PMService
–ONS15600SDHPMService
–ONS15800PMService
–ONS15801PMService
–ONS15808PMService
J.4.14 How Do I Enable or Disable the Automatic Refresh Data Action?
The automatic Refresh Data feature automatically refreshes all data being displayed by CTM. When automatic refresh is enabled, you receive the following prompt:
Refresh Data action suggested. This action will result in closing all windows and might
take some time. Do you want to continue? {Yes | No}
In an unstable environment where NEs are synchronizing or NEs change their operational state frequently, you might receive the preceding prompt continuously. To disable the prompt:
Step 1 In the Domain Explorer window, choose Edit > User Preferences. The User Preferences dialog box opens.
Step 2 Click the Miscellaneous tab.
Step 3 To disable the prompt, uncheck the Enable Refresh Data Timer check box. (To enable the prompt, leave the check box checked.)
Step 4 Click OK.
J.4.15 How Do I Replace the Alarm Interface Panel?
The Alarm Interface Panel (AIP) provides surge protection for CTC-based NEs. This panel has a nonvolatile memory chip that stores the unique node address known as the MAC address. The MAC address identifies the nodes that support circuits. It allows CTM to determine circuit sources, destinations, and spans.
If an AIP fails, an alarm will be generated and the LCD display on the fan-try assemblies of the NEs will go blank. To perform an in-service replacement of the AIP, you must contact Cisco technical support. For contact information, go to the technical support website at http://www.cisco.com/tac.
You can replace the AIP on an in-service system without affecting traffic by using the circuit repair feature. See 7.2.10 Repairing a Circuit, page 7-66.
J.5 Client Debug Messages
On Windows, you can start the CTM client with a console window that displays exceptions and client debug messages. You can use these messages to help troubleshoot any client problems.
Step 1 Depending on your network size, enter one of the following commands to start the client with a console window:
/bin/ctm-medium-debug.exe
/bin/ctm-highend-debug.exe
Step 2 When the CTM client exits, the console window closes. If you want to capture and save the messages when the client exits, you must redirect the output to a file. In this case, the console will not be displayed. To save exceptions to a file, open one of the following files that corresponds to your network size:
•/bin/ctm-small-debug.lax
•/bin/ctm-medium-debug.lax
•/bin/ctm-large-debug.lax
•/bin/ctm-highend-debug.lax
Note It is strongly recommended that you save a copy of the original .lax file before modifying it.
Step 3 Using a text editor, open the appropriate .lax file and change the following lines by replacing "console" with a filename:
lax.stderr.redirect=console
lax.stdout.redirect=console
When specifying filenames, make sure to use escaped backslashes (such as c:\\myfolder\\client.log). For example, to save messages to the C:\myfolder\client.log file, change the lines to read:
lax.stderr.redirect=c:\\myfolder\\client.log
lax.stdout.redirect=c:\\myfolder\\client.log
Step 4 After modifying the .lax file, enter one of the following commands to launch the CTM client:
/bin/ctm-medium-debug.exe
/bin/ctm-highend-debug.exe
On Solaris, you can start the CTM client with a console window that displays exceptions and client debug messages. You can use these messages to help troubleshoot any client problems.
Depending on your network size, enter one of the following commands to start the client with a console window:
J.6 CTM GateWay/TL1 Problems
If the OSS cannot connect to CTM GateWay/TL1, or if the OSS does not receive a response, check the cables, address bindings, and configuration.
Step 1 If the connection times out, check the cables and connections.
Step 2 Verify that the address bindings between the OSS and the Solaris workstation running the CTM GateWay/TL1 service are correct.
Step 3 Using the ping command, verify connectivity between the OSS and the Solaris workstation running the CTM GateWay/TL1 service.
Step 4 Verify that the TCP/IP socket connection is still open for the OSS-to-CTM GateWay/TL1 port.
Step 5 Choose Administration > Control Panel. Click GateWay/TL1 Service and confirm that the service status is Active.
Step 6 Verify that the username and password under Control Panel > Security Properties match the user configuration of the NEs that CTM GateWay/TL1 will manage.
Step 7 Verify that the username and password in the Domain Explorer > Network Element Properties > NE Authentication tab match the user configuration of the NEs that CTM GateWay/TL1 will manage.
J.7 CTM GateWay/CORBA Problems
If the CTM server is installed without the CTM GateWay/CORBA option, and the CTM GateWay/CORBA option is installed later, the Control Panel does not refresh automatically after the CORBA installation to show that CTM GateWay/CORBA is installed.
This problem occurs because the CTM GateWay/CORBA installation is handled by a script that notifies the database that the component is installed, but does not notify the CTM server.
To work around this problem, click the Refresh Data button in the Control Panel. It might take some time for the Control Panel to show that CTM GateWay/CORBA is installed. You might need to close and reopen the Control Panel to see the change.
J.8 Cisco CRS-1 and Catalyst 6509 Problems
This section describes troubleshooting procedures for problems related to the Cisco CRS-1 and the Catalyst 6509.
J.8.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service
Step 1 Verify that the CRS-1 is configured with "hfrems" as the username and "hfrems" as the password with "root-system" privileges. By default, CTM expects "hfrems" as both the username and password. You can change the username and password in the Control Panel > Security Properties > CRS-1 tab.
Step 2 Enter the following commands to verify that the XML CORBA agent service is running:
Router(config)# xml agent corba [ssl]
Note SSL is optional and enables SSL CORBA services.
Step 3 If SSL CORBA services are running, enter the following commands to verify that HTTP over SSL is running:
Router(config)# http server ssl
Step 4 The CORBA agent on the router accepts connectivity only when the connection is done through hostname. Verify that the device IP address has an entry in DNS. If not, add an entry to the /etc/hosts file.
Step 5 Verify that CTM supports the software image that is running on the router. In the Domain Explorer, choose Administration > Supported NE Table and verify that the software version is supported.
Step 6 When the CRS-1 is added to CTM, the NE type is retrieved from the NE to verify that the NE is a CRS-1. If the device does not report CRS-1 as the NE type, CTM reports a loss of communication (LOC) to the NE and raises an NE type mismatch alarm.
Step 7 Verify that the NE hostname specified in DNS or in the CTM server's /etc/hosts file matches the hostname configured on the NE. If the hostnames do not match, CTM reports an LOC to the NE.
J.8.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service
Step 1 Verify that the IP address entered in CTM is configured on the Catalyst 6509. Enter the following command to configure the IP address:
cat6509 (enable)# set interface <logical_interface_name_like_sc0/sc1>
<IP_address_of_the_Catalyst_6509> <mask>
Step 2 Enter the following command to verify that the device is configured correctly with the default route:
cat6509 (enable)# set ip route default <gateway_IP_address>
Step 3 Verify that CTM supports the software image that is running on the Catalyst 6509. In the Domain Explorer, choose Administration > Supported NE Table and verify that the software version is supported.
Note For the Catalyst 6509, the supported software version is Release 6.3(7) or later.
Step 4 Verify that the community string shown in the Domain Explorer > Address tab is same as the community string on the device. If there is a mismatch, enter the following command to change the SNMP community string on the device:
cat6509 (enable)# set snmp community read-only <community_string>
For example:
cat6509 (enable)# set snmp community read-only public
J.8.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1
Step 1 Enter the following commands to verify that the CTM server hostname and "logging-events-level" are configured in the running configuration:
Router# sh run | include domain ipv4 host
Router# sh run | include logging events level informational
Step 2 If the CTM server hostname and "logging-events-level" are not configured in the running configuration, enter the following commands:
Router(config)# domain ipv4 host <CTM_server_hostname> <IP_address>
Router(config)# logging events level informational
J.8.4 CTM Does Not Receive Traps from the Catalyst 6509
Step 1 Enter the following command to configure the CTM server as the trap receiver on the Catalyst 6509:
cat6509 (enable)# set snmp trap <CTM_server_IP_address> <community_string>
Step 2 Enter the following command to configure the Catalyst 6509 to enable only specific traps:
cat6509 (enable)# set snmp trap enable <specific_trap>
where <specific_trap> is auth, bridge, chassis, config, entity, ippermit, module, stpx, syslog, systemp, vmps, or vtp.
Step 3 Enter the following command to configure the Catalyst 6509 to enable all traps:
cat6509 (enable)# set snmp trap enable all
Note The set snmp trap enable all command enables traps only on physical ports. For logical ports such as sc0 or sc1, you must enable traps separately. Go to Step 4.
Step 4 Enter the following command to enable traps on logical ports:
cat6509 (enable)# set interface trap <logical_port_name> enable
For example, to enable traps on the logical interface sc0, enter:
cat6509 (enable)# set interface trap sc0 enable
J.8.5 CTM Does Not Receive syslog Events from the Catalyst 6509
The Catalyst 6509 sends syslog traps only if the trap severity is both:
•Less than or equal to facility severity (which is set with the set logging level command)
•Less than or equal to history severity (which is set with the set logging history severity command)
It is recommended that you set the history severity to debugging (7).
The show logging command shows both the history severity and the severity for all facilities. For example, in the following output, the history severity is 7 and facility severity is 5. Syslog events of severity X will be triggered from the device if X is less than or equal to 5 and less than or equal to 7.
Cat6509 (enable)# show logging
timestamp option: enabled
{10.77.56.151, 172.19.75.86}
server severity: debugging(7)
Current Logging Session: enabled
Callhome Functionality: disabled
Callhome Severity: LOG_CRIT(2)
SMTP servers are not configured for callhome messages.
Destination addresses are not configured for callhome messages.
From Address is not configured for callhome messages.
Reply-To Address is not configured for callhome messages.
Facility Default Severity Current Session Severity
------------- ----------------------- ------------------------
0(emergencies) 1(alerts) 2(critical)
3(errors) 4(warnings) 5(notifications)
6(information) 7(debugging)
J.8.6 CTM Does Not Collect Historical PM Data for the CRS-1
Step 1 Verify that PM collection is enabled for the CRS-1. To do this, go to the Network Element Properties > Status tab and in the PM Collection area, check the 15 Min check box. Click Save. Also, verify that the CRS-1 PM Service Status is Active in the Control Panel window.
Step 2 Enter the following commands to verify that the correct PM configuration is in the running configuration on the CRS-1:
Router(config)# show run perf
performance-mgmt resources tftp-server <IP_address_of_server> directory
<directory_on_TFTP_server>
performance-mgmt statistics bgp template default
performance-mgmt statistics mpls ldp template default
performance-mgmt statistics node cpu template default
performance-mgmt statistics interface generic-counters template default
performance-mgmt statistics interface data-rates template default
performance-mgmt statistics mpls TE-link template default
performance-mgmt statistics node memory template default
performance-mgmt statistics node process template default
performance-mgmt statistics mpls TE-tunnel template default
performance-mgmt statistics mpls interface template default
Step 3 If the files are not being dumped to the target directory on the TFTP server, verify that the TFTP server is using the correct tftpd (tftp daemon binary). The tftpd should be a version that allows the CRS-1 to create new files in the specified directory. The default tftpd only allows pre-existing TFTP files to be written. The default tftpd does not create new TFTP files.
Step 4 If the files appear on the TFTP server but do not appear in the PM table, verify that the CTM server is being notified of PM TFTP file dumping events. To do this, choose Fault > Alarm Log and check for events that arrive every 15 minutes and resemble the following output:
pm_collector[301]: %PM_COLLECTOR-6-TFTP_SEND : Successfully dumped
file:tftp://<IP_address><TFTP_directory><NE_name>_node_process_1100886887.212000
Step 5 If events do not appear after 15 minutes, verify that configuration change notifications are enabled. See CTM Does Not Receive Configuration Changed Notifications from the CRS-1.
Step 6 If events are present, verify that the TFTP server is reachable from the CTM server. To do this, ping the IP address that is present in the event notification. If the IP address is not reachable, configure the routes so that it becomes reachable. Alternately, include a mapping file in the /opt/CiscoTransportManagerServer/cfg/hfr/IPAddressMapping.cfg directory that includes the following mapping:
<IP_address_in_event_notification> <reachable_IP_address_of_the_same_TFTP_server>
For example:
172.19.64.51 223.255.254.254
J.8.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval
You might experience a problem where the PM data collection interval is set to 15 minutes, but the CRS-1 PM tables are collecting data more frequently (or less frequently).
The CRS-1 collects and dumps data to a binary file on a TFTP server at user-configured intervals based on the IOS-XR CLI. After dumping the PM data, the CRS-1 notifies CTM with the location of the file. CTM then collects PM data from the CRS-1. Therefore, you should configure the CRS-1 to collect data every 15 minutes.
J.8.8 Memory Backup or Restore Fails for the CRS-1
Step 1 Verify that the NE is in service and the communication state is Available.
Step 2 Verify that you can ping the CTM server from the NE. The NE is used as a TFTP client for downloading the configuration file.
Step 3 In the /etc/inetd.conf file, verify that the TFTP server is enabled on the CTM server. Also, verify that the specified tftpboot directory has the correct permissions.
J.8.9 How Do I Back Up the CRS-1 Configuration File Automatically Whenever the Configuration Is Changed on the Router?
Step 1 In the Domain Explorer, choose Administration > Control Panel.
Step 2 Click NE Service.
Step 3 In the NE AutoBackup tab, choose Cisco CRS-1 as the NE model. Alternately, choose All NE Models.
Step 4 Check the Enable Auto Backup check box.
Step 5 Specify the number of backup copies and the backup time.
Step 6 Click Save.
J.8.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs
Step 1 Verify that there is a physical link between the two NEs.
Step 2 Complete one of the following options, depending on your configuration:
•If the link is an Ethernet link, bring up the interfaces on both NEs. For example, enter the following commands on the first router:
Router-1(config)# int GigabitEthernet0/2/0/0
Router-1(config-if) ipv4 address <IP_address> <mask>
Router-1(config-if) no shutdown
Router-1(config-if) commit
Enter the following commands on the second router:
Router-2(config)# int GigabitEthernet0/1/0/0
Router-2(config-if) ipv4 address <IP_address> <mask>
Router-2(config-if) no shutdown
Router-2(config-if) commit
•If the link is a POS link, enter the following commands on the first router:
Router-1(config)# int POS 0/2/0/0
Router-1(config-if) ipv4 address <IP_address> <mask>
Router-1(config-if) no shutdown
Router-1(config-if) commit
Enter the following commands on the second router:
Router-2(config)# int POS 0/1/0/0
Router-2(config-if) ipv4 address <IP_address> <mask>
Router-2(config-if) no shutdown
Router-2(config-if) commit
Verify that the SONET controllers are up. For example, enter the following commands:
Router-1(config)# controller SONET 0/2/0/0
Router-1(config-sonet)# no shutdown
Router-1(config-sonet)# commit
Router-2(config)# controller SONET 0/1/0/0
Router-2(config-sonet)# no shutdown
Router-2(config-sonet)# commit
Step 3 Enter the following commands to enable CDP on both NEs:
J.8.11 Cannot Delete an Out-of-Service NE from CTM
If you cannot delete an out-of-service NE from CTM, follow the procedure in 3.5.7.3 Using the prune_ne.sh Script to Remove an Out-of-Service NE from the Database, page 3-27.
J.8.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty
If you cannot see the software version in the Domain Explorer > Identification tab and NE Software table is empty, complete the following steps:
Step 1 Verify that the NE is in service and the communication state is Available.
Step 2 Verify that the device is configured with either Telnet or SSH.
J.8.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer
If you cannot use Configuration > Telnet/SSH from the CRS-1 NE Explorer, try pinging the NE from the CTM client. You might be able to reach the NE from the CTM server but not from the client if they are in different subnets.
J.8.14 How Do I Configure Telnet on the CRS-1?
To configure Telnet on the CRS-1, log into the router through the console and enter the following commands:
Router(config)# telnet ipv4 server
J.8.15 How Do I Configure SSH on the CRS-1?
Step 1 Enter the following command to verify that the security PIE(k9) is installed and activate on the router. The PIE(k9) is required for encryption, decryption, IPSec, SSH, SSL, and PKI:
Router# sh install active
Step 2 Enter the following command to verify that DSA keys have been generated on the router:
Router# crypto key generate dsa
Step 3 Enter the following command to verify that RSA keys have been generated on the router:
Router# crypto key generate rsa
Step 4 Enter the following command to verify that SSH is enabled on the router:
Router# crypto key generate rsa
Step 5 Enter the following command to enable the SSH server on the router:
Router(config)# ssh server enable
J.8.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs?
See 5.4.25 Configuring Secure Socket Layer for the CRS-1 and XR 12000, page 5-188.
J.8.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509?
Complete one of the following options:
•To trigger a linkDown trap from the Catalyst 6509, enter the following command:
cat6509 (enable)# set port disable <module_number/port_number>
For example:
cat6509 (enable)# set port disable 5/1
•To trigger a linkUp trap from the Catalyst 6509, enter the following command:
cat6509 (enable)# set port enable <module_number/port_number>
For example:
cat6509 (enable)# set port enable 5/1
J.8.18 How Do I View the SNMP Configuration on the Catalyst 6509?
To view the SNMP configuration on the Catalyst 6509, enter the following command:
cat6509 (enable)# show snmp
J.8.19 How Do I Load a Software Image on the Catalyst 6509?
Complete the following steps:
Step 1 Enter the following command to view all the software images on the Catalyst 6509:
cat6509 (enable)# dir bootflash:
The output is similar to the following example:
-#- -length- -----date/time------ name
2 5849256 Aug 09 2004 14:41:25 cat6000-sup2.6-3-7.bin
3 8801412 Aug 09 2004 15:30:57 cat6000-sup2k8.8-3-2.bin
4 5713736 Aug 11 2004 13:08:02 cat6000-sup2.6-3-2a.bin
5 8182352 Aug 12 2004 09:15:00 cat6000-sup2k8.8-2-1.bin
17 1786 Aug 23 2004 14:16:30 6509svt-171-180
19 1688 Sep 24 2004 15:26:40 6509svt-171-180.cfg
3346628 bytes available (28634940 bytes used)
Step 2 Enter the following command to retrieve the image name that is currently on the device:
cat6509 (enable)# whichboot
The output is similar to the following example:
Boot image name is 'bootflash:cat6000-sup2.6-3-7.bin'.
Step 3 Enter the following command to delete the existing image:
cat6509 (enable)# del <image_name>
Step 4 Enter the following command to remove the deleted image:
cat6509 (enable)# squeeze bootflash:
Step 5 Enter the following commands to copy the image to flash memory:
cat6509 (enable)# copy tftp flash
IP address or name of remotehost[]? <TFTP_server_IP_address>
Name of file to copy from []? <Full_path_name_and_filename_of_the_image_location>
For example:
tftpboot/cat6k/7-2/<image_name>
Step 6 Enter the following command to verify that the downloaded image exists on the device:
cat6509 (enable)# dir bootflash:
Step 7 Enter the following command to verify that the download was successful:
cat6509 (enable)# verify <image_name>
Step 8 Enter the following command to clear the flash system:
cat6509 (enable)# clear boot system flash bootflash:<previous_image_name>
Step 9 Enter the following command to set the boot system flash with the new image name:
cat6509 (enable)# set boot system flash bootflash:<new_image_name> prepend
Step 10 Enter the following command to reset the device:
J.9 XR 12000 Problems
This section describes troubleshooting procedures for problems related to the XR 12000.
J.9.1 Reporting XR 12000 Issues
When reporting an XR 12000 problem, include detailed steps to reproduce the problem and the software version(s) of the NE(s) to which the problem applies. Also, enter the following router CLI commands on the XR 12000 and collect the data:
show ipv4 interface brief
show log events buffer all-in-buffer
Turn on trace level debugging in the Control Panel for the service in question, wait until the problem occurs, and collect the GSRIOXNEService logs from /opt/CiscoTransportManagerServer/log.
J.9.2 CTM Cannot Discover an XR 12000 Node
Complete the following steps if CTM cannot discover an XR 12000 node:
Step 1 Verify that the NE is configured with "hfrems" as the username and "hfrems" as the password with "root-system" privileges. By default, CTM expects the username and password to be hfrems/hfrems. You can change the security properties from the Control Panel > Security Properties pane > XR 12000 tab. CTM will use the new username and password to authenticate all XR 12000 devices in the domain.
Step 2 Check whether the XML CORBA agent service is running. To configure the XML CORBA agent, enter:
Router(config)# xml agent corba [ssl]
Note ssl is optional and enables SSL CORBA services.
If SSL CORBA services are activated, enter the following commands to check whether HTTP over SSL is running:
Router(config)# http server ssl
Step 3 The CORBA agent on the router accepts connections only when using the hostname. Check to see if the device IP address has an entry in DNS. If DNS is not used, add an entry in the /etc/hosts file.
Step 4 Verify that CTM supports the software version that is running on the router. The CTM release notes list the CTM-supported NE software versions. Go to http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/optnet/ctm/ctmreln/index.htm.
Step 5 When an NE is added to CTM, the NE type is retrieved to verify whether the NE is an XR 12000. If the device is not of NE type XR 12000, the NE is moved to an LOC state in CTM and an NE type mismatch alarm is raised.
Step 6 The NE hostname specified in DNS or in the CTM server /etc/hosts file must match the hostname configured on the NE. If the hostnames do not match, the NE is moved to an LOC state in CTM.
J.9.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer
If the LAN cable is removed, there is a delay before the status of the XR 12000 is updated in the Domain Explorer. It might take several minutes or more to reflect the status as unreachable if the router is disconnected.
If you want the status to be updated sooner, you must modify the jacorb.connection.client.pending_reply_timeout parameter in the /opt/CiscoTransportManagerServer/openfusion/classes/jacorb.properties file. The default timeout is 5 minutes (300000 ms). Updates take approximately the health poll interval plus two times the jacorb.connection.client.pending_reply_timeout value.
J.9.4 XR 12000 Alarm and Event Reporting Issues
If CTM does not receive XR 12000 alarm and event notifications, complete the following steps:
Step 1 Enter the following commands to verify that the CTM server hostname and "logging-events-level" are configured in the running configuration:
Router# sh run | include domain ipv4 host
Router# sh run | include logging events level informational
Step 2 If the CTM server hostname and "logging-events-level" are not configured in the running configuration, enter the following commands:
Router(config)# domain ipv4 host <CTM_server_hostname> <IP_address>
Router(config)# logging events level informational
J.9.5 XR 12000 PM Collection Issues
The XR 12000 collects and dumps data to a binary file on a TFTP server at intervals configured by the user through the CLI. After dumping the performance data, the XR 12000 sends a notification indicating the location of the file. Upon receiving this notification, CTM collects performance data from the XR 12000. Therefore, the network operator should configure the XR 12000 to collect data every 15 minutes.
Step 1 Enter the following command to verify whether the TFTP server and directory are configured in the running configuration:
performance-mgmt resources tftp-server <server_IP_address> directory
<directory_on_TFTP_server>
Step 2 Enter the following commands to verify that the router's running configuration contains the correct PM configuration:
performance-mgmt statistics bgp template default
performance-mgmt statistics mpls ldp template default
performance-mgmt statistics node cpu template default
Step 3 After dumping the performance data, the XR 12000 sends a notification indicating the location of the file. This notification is logged in the Alarm Log every 15 minutes and triggers the CTM server connecting to the TFTP server. Check for notifications in the Alarm Log that look similar to the following:
pm_collector[301]: %PM_COLLECTOR-6-TFTP_SEND : Successfully dumped
file:tftp://<IP_address><TFTP_directory><NE_name>_node_process_1100886887.212000
J.9.6 XR 12000 BGP and CDP Neighbor Discovery Issues
CTM discovers all the CDP and BGP neighbor routers and adds them to the tree view if the CDP or BGP is enabled on the router. CDP and BGP neighbor discovery occurs after the NE is synchronized. If the neighbors are not discovered, enter the following CLI commands to check whether CDP or BGP is enabled on the router and interfaces:
J.9.7 XR 12000 NE Explorer Applications Are Disabled
The CTM NE Explorer enables only those applications for which the XML version matches on CTM and the router. If an NE Explorer application is disabled, there might be a version mismatch. Do the following:
•Collect all the XML files from the <CTM_home>/cfg/metadata directory.
•Run the <CTM_home>/bin/vwdata.sh script to collect the hfr_version_info database table entries.
On Windows, if an NE Explorer or GUI application returns an error, repeat the same operation after invoking the GUI by entering one of the following commands, depending on your network size:
<CTM_home>/bin/ctm-small-debug.exe
<CTM_home>/bin/ctm-medium-debug.exe
<CTM_home>/bin/ctm-large-debug.exe
This opens a console that logs all the exceptions. Capture the console log for debugging.
On UNIX, enter the <CTM_home>/bin/ctmcdebug-start command to start the client and repeat the same operation. This generates the error and exception messages on the terminal. Capture the terminal messages for debugging.
J.10 Problems with MGX Voice Gateway Devices
This section describes troubleshooting procedures for problems related to the MGX Voice Gateway devices.
J.10.1 Discovery Mechanism
CTM manages the Private Network-Network Interface (PNNI). The ILMITopoc process uses SNMP protocol to discover the physical PNNI network. All the discovered nodes are displayed in the MGX NE GUIs (Configuration Center, Diagnostics Center, Statistics Report, and Chassis View). CTM is notified of all subsequent changes in the network through traps for MGX nodes.
Once the ILMITopoc process discovers all the nodes, it sends all the nodes to the topod process. Topod then sends these nodes and trunks to other processes in CTM such as ooemc, NMServer, nts, and so on.
J.10.2 Discovery Issues at Startup
This section includes the following information:
•No Nodes Are Discovered
•Node Name Is Incorrect in the Database
•Node IP Is Incorrect in the Database
•Node Alarm Is Shown Incorrectly in the Database
•Reachable Node Is Shown as Unreachable
•CTM State Is Incorrect
J.10.2.1 No Nodes Are Discovered
If no nodes are in the CTM database, complete the following steps:
Step 1 Verify that the nodes are IP-reachable. Complete the following substeps:
a. Enter the following CLI commands to retrieve the primary IP address of the node:
b. Enter the following command to check whether CTM has IP connectivity to the node:
Step 2 If CTM has IP connectivity to the nodes, verify that the community strings of the gateway nodes are correct. Complete the following substeps:
a. Enter the following command on the switch CLI of the node that cannot be discovered:
b. Compare this community string with the community string in the node_info table. The community string in the node_info table is encrypted. You will need to decrypt this string and verify it.
Step 3 For MGX nodes, if persistent topology is enabled on the switch gateway and some nodes in a peer group are not discovered, complete the following substeps:
a. Enter the following CLI command to verify that persistent topology is enabled on the gateway in the peer group:
b. If the gateway flag is enabled, enter the following CLI command to verify that the node is part of the list of nodes on the gateway:
Defect information—Collect the following information for further analysis:
•Topod.log and ILMITopoc.log.
•Data in the node_info table.
•Output of dspndparms, dspsnmp, and dspipif on the MGX node.
J.10.2.2 Node Name Is Incorrect in the Database
If the node name of the standalone MGX node is incorrect, complete the following steps:
Step 1 If the node table contains the correct name, complete the following substeps:
a. Enter the following command to dump the cache of topod and ILMITopoc:
b. From the data collected, verify which process contains the incorrect information. If the information is correct in all processes, open a new instance of the GUI and verify whether the problem still exists.
Step 2 If the node table contains the incorrect name, check the ILMITopoc.log file to see from which node it collected the wrong information. Complete the following substeps:
a. Enter the following command:
snmpget -c <community> <IP_address> 1.3.6.1.2.1.1.5
b. Verify that the name received as a response is correct. If the name received is incorrect, verify the name on the switch.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
J.10.2.3 Node IP Is Incorrect in the Database
Step 1 If the node table contains the correct IP address, complete the following substeps:
a. Enter the following command to dump the cache of topod and ILMITopoc:
b. From the data collected, verify which process has the incorrect information. If the information is correct in all processes, open a new instance of the GUI and verify whether the problem still exists.
Step 2 If the node table does not have the IP address set correctly, verify whether the node is managed by the LAN IP address. In CTM, since the node does not have any PNNI trunks and the persistent topology feature is not enabled, the node is managed by the LAN IP address by default.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Collect the output of the switch CLI, selnd, and dbnds.
J.10.2.4 Node Alarm Is Shown Incorrectly in the Database
Step 1 Enter the following command to dump the cache of topod and ILMITopoc:
Step 2 From the data collected, verify which process contains the incorrect information.
Step 3 If the node table does not have the node alarm status set correctly, check the ILMITopoc.log file and enter the following command:
snmpget -c <community string> <IP_address> 1.3.6.1.4.1.351.110.1.1.14
The snmpget command returns the alarm state of the node, where 1 = Clear, 2 = Minor, 3 = Major, and 4 = Critical.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log and topod.log.
•Dump outputs of ILMITopoc and topod. Enter the following command:
kill -USR1 signal <process>
•Collect the output of the switch CLI, selnd, and dbnds.
J.10.2.5 Reachable Node Is Shown as Unreachable
If a node that is reachable from CTM is shown as unreachable in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table reports the alarm state of this node as minor, major, critical, or clear, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /opt/svplus/log directory and verify whether the node alarm status is correct. If the alarm status is correct in the cache dump, open a new instance of the GUI and check whether the node shows the correct alarm status in that window.
Step 2 If the node table shows the alarm state of an MGX node as unreachable, complete the following substeps:
a. Verify that the node is IP-reachable from CTM.
b. Ping the active IP address of the node. The active IP address is the IP address of the node that is populated in the node table.
c. Enter the following CLI command to retrieve the community string of the node:
d. Verify that the community string in the node_info table is the same as the community string on the node.
e. Check the nts logs to see if nts has declared the node as reachable. See NTS. If nts has declared the node as reachable, check the topod.log for "nonRoutingNodeMsg" for this node. Verify that the alarm status in this message is the correct alarm status.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.*.log, nts.*.log, and ooemc*log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Output of the switch CLI, selnd, and dbnds.
J.10.2.6 CTM State Is Incorrect
If the management state of the node in the node table (mgmt_state column) shows as DOWN (2) or UNKNOWN (0) even when the node is reachable by IP, complete the following steps:
Step 1 Check the trap manager registration of the CTM station on the node. Log into the node and enter the following command to check whether the CTM station has registered to the node for traps. This is required for the node to be declared as manageable (mgmt_state = UP).
Step 2 Enter the community strings (SNMP-GET and SNMP-SET) of the node through the Domain Explorer > Network Element Properties pane > NE Authentication tab. The SNMP strings might be incorrect in the node_info table. The strings can be verified if the decrypt tool for the encrypted strings is available in the database.
Step 3 Check the nts.log to see if trap registration succeeded for the node. See NTS.
Step 4 Save ILMITopoc.log, topod.log, and ooemc*.log for analysis.
J.10.3 Discovery Issues at Runtime
This section includes the following information:
•Node Is Not Shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Node Name Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Node IP Address Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
J.10.3.1 Node Is Not Shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If a dynamically added routing node is not shown in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /usr/user/svplus/log directory and verify whether the node is present in the cache dumps. If the node is present, open a new instance of the GUI and check whether the node is visible in that GUI.
Step 2 If the node table does not have an entry for the node, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 70005 and 70201 trap (if the persistent topology feature is enabled).
b. If you see the trap in the log, verify that ILMITopoc's SNMP requests issued after receiving the trap are successful. If an SNMP request does not go through, verify whether CTM has IP and SNMP connectivity to the newly added node.
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Step 3 Complete the following substeps to collect defect information for analysis:
a. Save ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
b. Enter the following command to collect the dump outputs of ILMITopoc, topod, and NMServer:
kill -USR1 signal <process>
c. Collect the output of the switch CLI, selnd, and dbnds.
Step 4 If the preceding steps do not solve the problem, delete the node from the network and add it again.
J.10.3.2 Node Name Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If changes to the node name are not reflected on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node with the correct name, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /usr/user/svplus/log directory and verify whether the node name is correct in the cache dumps. If the node name is correct, open a new instance of the GUI and check whether the node name is correct in that GUI.
Step 2 If the node table does not have an entry for the node with the correct name, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 60006 and 70202 trap.
b. If you see the trap in the log, verify that ILMITopoc updates its cache based on the trap. If it fails to update the node name, ILMITopoc.log will contain the message "%ILMITopoc-3-updateFailed: <Node_c::updateName> Failed to update node name."
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Collect the output of the switch CLI, selnd, and dbnds.
J.10.3.3 Node IP Address Change Is Not Updated in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
If the node IP address is not updated dynamically in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report, complete the following steps:
Step 1 If the node table has an entry for the node with the correct IP address, complete the following substeps:
a. Enter the following command to dump the cache of NMServer, topod, and ILMITopoc:
kill -USR1 signal <process>
b. Check the data collected in the /opt/svplus/log directory and verify whether the IP address is correct in the cache dumps. If the IP address is correct, open a new instance of the GUI and check whether the IP address is correct in that GUI.
Step 2 If the node table does not have an entry for the node with the correct IP address, complete the following substeps:
a. Check the ILMItopoc log and verify whether it received a 60007 and 70202 trap.
b. If you see the trap in the log, verify that ILMITopoc updates its cache based on the trap.
c. If you do not see the traps in the ILMITopoc.log, see NTS.
Defect information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, NMServer.log, and nts.*.log.
•Dump outputs of ILMITopoc, topod, and NMServer. Enter the following command:
kill -USR1 signal <process>
•Collect the output of the switch CLI, selnd, and dbnds.
J.10.4 Equipment Management Problems
This section describes troubleshooting procedures for equipment management problems on the Cisco MGX devices.
J.10.4.1 Equipment Management Declares Successful Node Resync when Node Mode Is 3
MGX nodes are managed by the ooemc component in CTM. Equipment management (EM) uses coldstart, warmstart, and periodic resync to synchronize network data. Each of these resync methods involves configuration file upload, parsing, and database population. The node mode in the node table indicates the state of the synchronization process. The possible node modes are 1, 2, 3, and 5. The expected mode for a successful node resync is 3. If the node mode stays in other modes for a long time, it indicates a resync problem. Any synchronization problem could be caused by the switch, SNMP, FTP, or ooemc. You must investigate different areas to identify the root cause of any synchronization problem.
The following is the flow of events in the ooemc coldstart synchronization process:
1. The topo process discovers a node in the network. The process changes the node mode in the node database table to -1. It then notifies the ooemc component for the discovery. The ooemc changes the node mode from -1 to 1 after the component begins managing the node.
2. When the nts component is able to register with the node, it sends a link up message to the ooemc that manages the node. The ooemc component changes the node mode from 1 to 2 to signify the start of the node synchronization.
3. The ooemc component sends an SNMP bulk file creation request to the switch. If the request is allowed, the switch sends a trap 60901 to CTM to signify the start of bulk file creation. When bulk file creation is complete on the switch, the switch sends a trap 60902 to CTM. After CTM receives the trap, it begins to FTP the config upload files.
4. If the upload and parsing of all config files completes without errors, the ooemc changes the node mode from 2 to 3. If the upload or parsing encounters errors for any service module on the switch, the node synchronizes partially and the ooemc changes the node mode from 2 to 4 at the end of the node synchronization process. If the error occurs on the shelf generic card file (CARD_01_CC.CF) or pnni file (PNNI_01_CC_CF), the ooemc changes the node mode from 2 to 5 to signify node sync-up failure.
J.10.4.2 Network Setup and Configuration Required for OOEMC Sync-Up Process
The EM requires network setup and CTM configuration prerequisites before it can start the sync-up process. On the switch, perform the following:
•Use the dsptrapmgr CLI command to verify that the CTM IP address has been added to the trap manager. If your CTM IP address is not registered with the switch, add the CTM IP address to the trap manager by entering the addtrapmgr <cwm_ip> 2500 CLI command, where 2500 is the port number and cwm_ip is the IP address of your CTM machine. You might need to delete other IP addresses from the trap manager list if the list is full. If the list is not full, the registration takes place automatically if CTM is configured to manage the node.
•The trap IP should have been configured on the switch properly. Whenever you configure the switch to use any IP (atm0 IP or lnPci0 IP) of the switch as primary IP, the topology component of CTM should use that IP for node discovery. You can display the primary and secondary IP information of the switch by typing the CLI command dspndparms. To configure the primary and secondary IP addresses, you can enter cnfndparms and the CLI will prompt you for options. After you have set up the primary and secondary IP addresses, you should also configure the primary IP as trap IP on the switch by entering the CLI command cnftrapip configured_IP, where configured_IP is the IP that you have chosen to use as the primary IP in the CLI command cnfndparms.
•On CTM, the following is a list of configurable parameters in the file /opt/svplus/config/emd.conf of your CTM machine that you should set up for proper problem logging:
–"OODebug": For debugging purposes, this parameter should be set to level 6 or above; for example, the log level statement in emd.conf should be "OODebug Level 6."
–"OOKeep": For debugging purposes, this parameter should be set to a value depending on how many log files you want to keep; for example, the statement in emd.conf should be "OOKeep 100 ooemc log files per oochild."
Note Sometimes, a different network environment requires other parameters in the same files to be tuned for correct functioning of CTM.
J.10.4.3 Node Mode Remains in 1
The node has been discovered by the network topology process in CTM, its node mode has been changed from -1 to 1 in node table entry, and it stays in mode 1 for a long period of time.
Step 1 If CTM stays in mode 1 for a long time after CTM core has been started, you need to check the trap manager on the switch. See Network Setup and Configuration Required for OOEMC Sync-Up Process for trap manager and trap IP setup.
Step 2 If step 1 is not an issue, then check for rtm link up message that nts sends to EMD. The ooemc will start node resync once EMD notifies the ooemc that the node is active. Therefore, the second step for debugging this mode 1 problem is to see whether rtm link up message has been received by EMD and whether it has been forwarded to the ooemc.
For example, to find the rtm link up messages received by EMD for node with ID 9, issue the following command:
grep RTM_LINK_UP emd* | grep "Node id 9"
Once the location of the rtm link up message is found in the log, view the log file for more information on notification to ooemc.
Step 3 If the rtm link up message for the node is found and the log indicates that notification to ooemc has been sent, then collect the log files and report the problem.
Step 4 If the rtm link up message cannot be found, then search for link down message. Grep RTM_LINK_DOWN from the emd log files. If the rtm link down message is found, then the node is not reachable by the nts process. Check with the network administrator and see NTS if RTM_LINK_UP and RTM_LINK_DOWN messages are not found.
Defect information—Collect the following information for further analysis:
•Ooemc and nts log files from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.4.4 Node Mode Remains in 2
After EMD receives the rtm link up message and the node active message has been sent to ooemc, ooemc will change the node mode from 1 to 2 and trigger the node resync process. The node resync time varies depending on the switch configuration and network activities. If the node mode stays in mode 2 for longer than the normal time for node resynchronization, there can be a problem with the node resync process.
Step 1 The debugging process for this problem is mainly focused on log file inspection. Make sure that log level of all related CTM processes is set at the right level. See Network Setup and Configuration Required for OOEMC Sync-Up Process for more information.
Step 2 Go to the /opt/svplus/tmp directory to check for configuration upload files.
Step 3 Determine which files have been uploaded for a node. For example, issue the following command on a node with node_id=9:
If files have been uploaded for this node, then the node resync process is ongoing. Make sure that the time stamps for these files refer to the current time.
Note Refer to step 6 for a list of configuration upload files.
Step 4 If there are no files uploaded for a node, it is possible that the trap IP address on the switch does not match the IP address used for node discovery. It is also possible that an SNMP request failure occurred. To check the IP addresses, perform the following:
a. Use the CLI command dbnds to display the IP address used for node discovery.
b. Use the CLI command dsptrapip to display the trap IP address.
c. If there is a mismatch, use the CLI command cnftrapip to reconfigure the trap IP address. Use the IP address used by CTM for node discovery. See Network Setup and Configuration Required for OOEMC Sync-Up Process for more information.
Step 5 If the trap IP address is not the issue, check whether there is an SNMP request failure. Normally, after the node resync process is triggered, ooemc will send SNMP requests to the switch to start the configuration upload file creation process. If the request fails, then ooemc will continue resending the SNMP requests until it exceeds the maximum number of retries and declares node resync failure with node mode equal to 5.
Note It should not take long to declare node resync failure because of the SNMP request failure. To verify that there is an SNMP failure, check the ooemc and snmpcomm log files.
Note Using ooemc10.6568.log as an example of an ooemc log filename, 10 is the child ID, the 6568 is the ooemc process ID. The child ID is calculated from the following formula: Remainder (NEDBACCSSID / number of ooemc child) + 1. To retrieve the process ID, use the CLI command psg em.
Step 6 Grep RESYNC from the log file. The grep result should include the node ID and node mode. Review the log messages in the log file to determine if an SNMP failure has occurred.
Step 7 Do the following:
a. Check the log file for trap 60903. If you find it, contact the platform team to investigate the bulk file creation failure on the switch.
Very often, the node mode 2 problem is caused by a bulk file creation abort process on the switch. If there are no SNMP request failures, the switch should send traps 60901 and 60902 to ooemc. Trap 60901 indicates that the switch will start bulk file creation, while trap 60902 indicates bulk file creation done. When the ooemc receives a trap 60902, it will start to FTP the bulk file, which is a shelf generic configuration file. This is also the first file to be uploaded in every resync mode (coldstart, warmstart, or periodic resync.) The commonly seen mode 2 problem is that the switch will abort the bulk file creation and send trap 60903 to CTM. CTM will reschedule the next SNMP request unless it exceeds maximum retries.
b. Check the log file for trap 60902.
CARD_01_CC.CF.10. This file is a shelf generic file and is the first file to be uploaded after CTM receives trap 60902 from the switch. The 10 is the node ID. If you find trap 60902 in the log file but no config file has been uploaded after a reasonable amount of time, it is possible that the FTP failed.
To find out if FTP failure has occurred, do the following:
a. Grep "to ftp file" and the config filename, or just "FTP" from the log files.
b. Trace the log messages in the log file.
For a more detailed information on SNMP request failure and FTP failure, check the snmpcomm and cwmftpd log files and the cwmftp.request_log. Search for the filename at the time the error happened in both files. The cwmftp.request_log gives the summary or final result of the FTP operation, and any error is reported. The cwmftp.log shows the FTP operation details.
CTM will upload and parse a set of config files from the switch. The following files are uploaded from the MGX NE, which contains VISM, AXSM, VXSM, SRM, and RPM/RPM-PR cards:
1. CARD_01_CC.CF
2. SM_1_slot#.CF
3. SM_1_slot#.CS
4. SM_CARD_01_slot#.CF
5. SM_CONN_01_slot#.CF
6. SM_ALARM_01_slot#.CF
7. SM_CON_UPDATE_01_slot#.CF
8. SM_CARD_01_SRM.CF
9. SM_CARD_01_RPM.CF
10. PNNI_01_CC.CF
Files 1 and 2 are uploaded for each NBSM. Files 4 to 7 are uploaded for each AXSM. File 9 is uploaded for all RPM/RPM-PR cards on the switch.
The following files are uploaded from the MGX NE. It includes VISM, AXSM, VXSM, SRM, RPM/RPM-PR, and RPM-XF cards:
1. CARD_01_CC.CF
2. SM_1_slot#.CF
3. SM_1_slot#.CS
4. SM_CARD_01_slot#.CF
5. SM_CONN_01_slot#.CF
6. SM_ALARM_01_slot#.CF
7. SM_CON_UPDATE_01_slot#.CF
8. SM_SC_slot#_transactionID_date.CF
9. SM_IC_slot#_transactionID_date.CF
10. SM_CARD_01_SRM.CF
11. SM_CARD_01_RPM.CF
12. PNNI_01_CC.CF
The difference between the items in these lists is in the files uploaded for AXSM cards and RPM-XF cards. For the switch software release 4.0 or later, the static file (SM_SC_slot#_transactionID_date.CF) and incremental file (SM_IC_slot#_transactionID_date.CF) are uploaded. For the switch software release 3.0 and earlier, conn, alarm, and conn update files are uploaded for each AXSM. Files 4 to 7 in the second list are uploaded for each RPM-XF card, and file 11 is uploaded for all RPM/RPM-PR cards on the switch.
Step 8 In another scenario in which the node mode stays in 2 for a very long time, the parsing of one particular config upload file has taken a very long time and has not completed. To view the parsing process, enter the following command:
Step 9 Collect the ooemc, nts, snmpcomm, and cwmftpd log files from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.4.5 Node Resync Mode Is 5
Node mode 5 indicates that node resync has failed. This could be caused by the config upload or parsing failure of the shelf generic file (CARD_01_CC.CF.13, for example) or pnni file (PNNI_01_CC.CF.13, for example). In general, mode 5 signifies that a problem has happened in one or more stages of the entire resync process. The resync process is composed of the following stages and each stage alone can lead to mode 5 problems:
1. The ooemc triggers node resync.
2. The ooemc sends an SNMP request to the switch for bulk file creation.
3. The ooemc receives bulk file creation-related traps: 60901, 60902, and 60903.
4. The ooemc FTPs the config upload files from the switch after it has received 60901 and 60902 from the switch.
5. The ooemc parses the config upload files.
6. The ooemc declares sync-up done.
Step 1 Determine if the problem is caused by the SNMP request:
a. Grep RESYNC from the ooemc log files. The ooemc will change node mode to 2 when it starts the node resync process. You should see something like the following from the log:
NOTICE: N17 <EMC_Node_c::InSync> SENDING RESYNC STATUS 2 FOR NODE 17 TO GUI - Node is
synchronizing.
This message tells you that ooemc will start the node resync process for nodes with ID equal to 17 (N17). If you look further in the log, you should see log messages related to SNMP requests and SNMP responses. If the SNMP request is successful, the switch will respond to the request. The ooemc will then process the SNMP response by invoking response function, which may do nothing:
<EMC_SnmpFunc_c::ProcFunc_GenNodeBulkFile_1> entering
b. If the log messages indicate SNMP error, refer to the snmpcomm log files for more failure information.
Step 2 Determine if the problem is caused by bulk creation traps. Grep 60901, 60902, and 60903 from the log files. (See Node Mode Remains in 1 for more information.) You should see something like the following:
INFO: <EMC_TrapClientImpl::onIncomingTrap> NTS NodeId 17 genericTrap 6 specificTrap 60901
INFO: <EMC_TrapClientImpl::onIncomingTrap> NTS NodeId 17 genericTrap 6 specificTrap 60902
Step 3 Determine if the problem is caused by the config file FTP. Check the FTP or the FTP plus config upload filename. You should see something like the following:
NOTICE: N17 <EMC_NodeFsmHandler_c::LoadShelf> to ftp /opt/svplus/tmp/CARD_01_CC.CF.17
INFO: <ParseFile_c::CheckFile> OOEMC9 CHECKSUM OK FOR FTP FILE
/opt/svplus/tmp/CARD_01_CC.CF.17
These messages indicate that FTP is successful and there is no checksum error in the file. From the log file, you should able to find out if there is an FTP problem. Refer to the cwmftpd log files for more failure information.
Step 4 Determine if the problem is caused by shelf generic file or pnni file parsing. After you have located where to do a checksum check on the shelf generic file (CARD_01_CC.CF.17, for example) or pnni file (PNNI_01_CC.CF.17, for example), continue to trace the log messages to see whether or not parsing error has occurred. If there are no errors, you should see the following:
INFO: N17 <EMC_NodeFsmHandler_c::FinishShelf> Parse /opt/svplus/tmp/CARD_01_CC.CF.17
successfully.
Defect information:
•Collect the ooemc, nts, snmpcomm, and cwmftpd log files from the /opt/svplus/log directory.
•Collect the config upload files from the /opt/svplus/tmp directory.
Possible alternative workaround—None.
J.10.4.6 CTM Database Is Inconsistent with Switch Data After Successful Coldstart (CTM Server Stop/Start) or Periodic Resync
After a successful node resynchronization triggered by the periodic resync, the CTM database is found to be inconsistent with the switch data.
Step 1 Collect the ooemc log files and all config upload files for the node. The ooemc implements a node-based cache. You can dump and save the cache.
Step 2 Get the process ID of the ooemc process that manages the node.
Step 3 Save the config upload files and ooemc log files.
Defect information—Collect the ooemc and config upload files from the /opt/svplus/log directory.
Possible alternative workaround: Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
J.10.4.7 CTM CM GUI Shows Incomplete Connections After Successful Node Resync and Dbroker Sync-Up
CTM shows that some of connections are incomplete after the node resync process and dbroker synchronization process.
Step 1 Determine which ooemc process manages the node. The ooemc will manage the connection segment in the CTM database, which terminates on the MGX. Find the child ID of the ooemc process that manages your node. The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection. The debugging process for this issue mainly focuses on log file inspection.
Step 2 After you have identified the ooemc log files, grep NotifyDataBroker from the files to see some log message examples before you modify your grep format. This will help you more efficiently grep the pair of log messages that correspond to the local and remote ends of one endpoint (primary endpoint or secondary endpoint) of a connection segment. In other words, each end (either primary end or secondary end) of a connection segment managed by ooemc should be logged with one pair of messages, one for local end and one for remote end. This pair of messages corresponds to one atm_connection segment entry in the database. Verify the information that ooemc sends to dbroker. Dbroker will refer to these messages from ooemc and construct a user_connection database entry. CTM Connection Management (CM) GUI displays connection information based on the user_connection database entry. Whether or not the connection is displayed as complete or incomplete depends on the information in the user_connection database entry. It is important to verify that ooemc sent the correct information to dbroker. Also verify that the segment database entries are populated correctly.
Step 3 If the correct number of messages have been forwarded to dbroker and the data in the messages is correct, but the user_connection database entry still contains invalid data, see Data Inconsistency for more debugging information, or contact the dbroker development engineer (DE) for further investigation. If the problem is due to wrong data populated in segment database entries, then the EM DE should look at the problem.
Defect information:
•Collect ooemc and dbroker log files from the /opt/svplus/log directory.
•Collect config upload files from the /opt/svplus/tmp directory.
Possible alternative workaround is to manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
J.10.4.8 CTM GUI Shows Mismatched Information Between GUI Views and Database Data
After an initial ooemc node resync, such as a coldstart process, GUI views such as Network Monitor tree view, Inspector View, and Chassis View display information that does not match newly provisioned or updated database data. The problem persists even after each subsequent manual or periodic resync.
Step 1 Determine which ooemc process manages the node and find the child ID of the ooemc process that manages your node. The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection. The debugging process for this issue mainly focuses on log file inspection.
Step 2 Initiate an initial node resync, such as coldstart. Ooemc starts sending newly provisioned or updated database data to the network management (NM) server. The current supported database tables for NM server message forwarding include the following:
•node
•card
•line
•ausm_port
•cesm_port
•frp
•rpm_port
•svc_port
•virtual_port
•aps
•ima_group
•ima_link
•linedistribution
•au4tug3
•controller
•license_in_use
•mfr_bundle
•mfr_link
•peripheral
•redundantcard
•sensor
Step 3 To determine whether or not ooemc sent newly provisioned or updated database data to the NM server, grep ComposeNMSMsg from the ooemc log files. The key words in log messages that correspond to the supported database tables in step 2 are the following:
•EMC_DBProperty_Node_c::ComposeNMSMsg
•EMC_DBProperty_Card_c::ComposeNMSMsg
•EMC_DBProperty_Line_c::ComposeNMSMsg
•EMC_DBProperty_AusmPort_c::ComposeNMSMsg
•EMC_DBProperty_CesmPort_c::ComposeNMSMsg
•EMC_DBProperty_FrPort_c::ComposeNMSMsg
•EMC_DBProperty_RpmPort_c::ComposeNMSMsg
•EMC_DBProperty_SvcPort_c::ComposeNMSMsg
•EMC_DBProperty_VtPort_c::ComposeNMSMsg
•EMC_DBProperty_Aps_c::ComposeNMSMsg
•EMC_DBProperty_ImaGroup_c::ComposeNMSMsg
•EMC_DBProperty_ImaLink_c::ComposeNMSMsg
•EMC_DBProperty_LineDist_c::ComposeNMSMsg
•EMC_DBProperty_AU4TUG3_c::ComposeNMSMsg
•EMC_DBProperty_Ctrlr_c::ComposeNMSMsg
•EMC_DBProperty_LicenseInUse_c::ComposeNMSMsg
•EMC_DBProperty_MfrBundle_c::ComposeNMSMsg
•EMC_DBProperty_MfrLink_c::ComposeNMSMsg
•EMC_DBProperty_Peripheral_c::ComposeNMSMsg
•EMC_DBProperty_RedCard_c::ComposeNMSMsg
•EMC_DBProperty_Sensor_c::ComposeNMSMsg
Note Database data will not be sent to NMServer during the initial node resync. It is after the initial resync that ooemc starts sending updated or newly provisioned database data to NMServer. Each ooemc log message from grep contains detailed information on the identity of the NE object and the required fields from the corresponding database table. The following is an example of NMServer message for line tables:
ooemc10.24370.log.old.5:( 24370: 4) 23:15:17 INFO: N0:C7:B2:L2
<EMC_DBProperty_Line_c::ComposeNMSMsg> (MODIFY::LINE) aNMSEvent.node:0 type:4 subType:1
lpbkType:1 alarmState:1 adminState:1 sectStatus:6 pathState:2
In this example, the identity of the NE object is N0:C7:B2:L2, which is the line with node ID=0, slot=7, bay=2, and line#=2. The database operation is update. The rest of the message shows the values of the required fields from the line table.
Step 4 If grep returns log messages that indicate matched data, then you need to continue investigation on the NM server. See CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics for information on nmController.
Defect information—Collect ooemc and NMServer log files from the /opt/svplus/log directory.
Possible alternative workaround—Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
J.10.4.9 CTM Database and Switch Data Inconsistency Issues After Node Provisioning for MGX
This section includes the following information:
•Database Table Population Through Traps and SNMP Upload
•Switch Data Does Not Match CTM Database Table After Node Provisioning
J.10.4.9.1 Database Table Population Through Traps and SNMP Upload
Except for node and/or card resync, another mechanism that CTM ooemc uses to populate the database table entries is through trap processing followed by SNMP upload, if necessary. You can provision the switch through switch CLI, CTM GUI, or CTM service agents. All three cases will involve trap processing and SNMP upload on ooemc. Only connections can be provisioned by the CTM CM GUI. Other provisioning, such as line and port, have to go to CTM service agents. The MGX switch CLI can do all.
J.10.4.9.2 Switch Data Does Not Match CTM Database Table After Node Provisioning
After switch provisioning through the switch CLI, CTM GUI, or CTM service agents, the switch data does not match the CTM database.
Step 1 If the issue is caused by a trap (for example, if you provision a connection on a switch or through CTM GUI or service agent, and CTM does not populate the database entry or the information in the database entry does not match the switch data), it is possible that CTM did not receive the trap, or that it received the trap but the trap is buffered in the ooemc trap queue, and trap processing has been delayed. On the other hand, if the data in the CTM database is not correct, it is also possible that the SNMP upload failed to upload correct data. Study the log files to understand the root cause of the inconsistency problem.
Step 2 To determine whether or not a trap has been received and processed, there are key words in the ooemc log files that you can grep. For example, to find out the channel traps from node_id=4, slot=6, vpi=1, and vci=326, you can grep "TRAPLIST" as shown in the following:
cwmult60% grep "TRAPLIST: N4:" ooemc* | grep "Channel Trap" | grep "C6" | grep "vpi 1 vci
326"
ooemc10.5760.log.old.55:( 5760: 10) 23:21:12 INFO: TRAPLIST: N4: Channel Trap 60310 from
C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 67
ooemc10.5760.log.old.55:( 5760: 10) 23:21:12 INFO: TRAPLIST: N4: Channel Trap 60310 <
PROCESSED > from C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2
operS 1 alarm 67
ooemc10.5760.log.old.73:( 5760: 10) 23:23:48 INFO: TRAPLIST: N4: Channel Trap 60310 from
C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 66
For other kinds of traps, you can use the following key words to supplement "TRAPLIST" in your grep statement:
•Port Trap
•RscPart Trap
•Svc Trap
•SonetLn Trap
•SctCard Trap
•SonetPath Trap
•FunMod Trap
•LineMod Trap
•RedCard Trap
•TrapMiss Trap
•VsiCtrlr Trap
•DS3Line Trap
•AtmPhy Trap
•Peripheral Trap
•CoreSwth Trap
•TrapLost Trap
•AtmAddr Trap
•Restart Trap
•Node Trap
•SonetAps Trap
•LMIPort Trap
•Pnni IF Trap
•Bulkfile Trap
•Vism Trap
•Vism Ann Trap
•SubIf Trap
•NBSMCnfg Trap
•NBSMChan Trap
•NBSMLine Trap
•AusmLine Trap
•AusmPort Trap
•AusmChan Trap
•AusmIma Trap
•FrChan Trap
•FrPort Trap
•CesmPort Trap
•CesmChan Trap
•HsFrPort Trap
•HsFrChan Trap
•LineDist Trap
•DS3Path Trap
•Chan Upload
•Party Trap
•PrefRoute Trap
•CardIma Trap
•DS1 Line Trap
•SctPort Trap
•SvcDerouteGroomTrap
•Channel Trap
•TUG3Path Trap
•Cug Trap
•AddrCug Trap
•RSC Upload
•APS Upload
•FrPort State Upload
•MPSM Upload
•Vism ToneDetect Trap
•License Trap
•PortAtmIf Trap
•VxsmPvcRed Trap
•ChanProt Trap
•VxsmGwDsp Trap
•VxsmGwIp Trap
•VxsmGw Trap
•VxsmSysRes Trap
•VxsmGw1 Trap
•VxsmMgc Trap
•VxsmMgcIp trap
•VxsmMgcGrpParam Trap
•VxsmMgcGrpMgc Trap
•VxsmMgcGrpProt Trap
•VxsmAal2Prof Trap
•VxsmCodec Trap
•VxsmSvc Trap
•VxsmAal2CrossConn Trap
•VxsmAal25DataProfileTrap
•VxsmSensor Trap
•VxsmSensorThrhd Trap
•VxsmModule Trap
•DS0Grp Trap
•VxsmAnnounce Trap
•VxsmAudioFile Trap
•VxsmDs0XConn Trap
•VxsmMegaco Trap
•VxsmCrr Trap
•VxsmTone Trap
•VxsmAs Trap
•VxsmAsp Trap
•VxsmAs Trap
•VxsmLapd Trap
•VismABCDBitTemplate Trap
Review the log messages using the traplist command and decide how you can grep the messages according to your needs. In the above example, there is another key word ("PROCESSED") that indicates that the trap has been processed. This gives you the starting point in the log file so that you can study the log messages. If the database is not populated correctly, then the subsequent log messages should provide you information on the database inconsistency problems. Some trap processing may invoke SNMP data upload. From the log messages, you can verify whether the uploaded data is correct or not. Other possible reasons for database inconsistency are:
•SNMP upload failure and maximum retries is exceeded; SNMP timeout; or throttle error. You can verify the problem in the snmpcomm log files.
•Database operation error. You can verify the problem in the ooemc log files.
If you do not know exactly what trap sequence is sent by the switch to CTM, you should try a working scenario and study the log for the trap sequence, or you can use HPOV to determine the trap sequence.
Step 3 To verify whether or not a trap has been received from the switch and forwarded to ooemc, refer to the nts log. If the trap has been buffered in the trap queue, the possible reasons are:
•Node is syncing due to regular node resync or due to -2 trap.
•Card is syncing and the trap is related to the card. The traps will be buffered in the queue until card resync completes.
•Summary alarm trap is processed. Summary alarm is being uploaded or parsed. You can grep the log files for information related to summary alarm traps.
To verify that the trap has been put into the queue, you can do the following:
grep EMC_TrapQueue_c::append ooemc_log | grep trap_num
You should see something like the following:
INFO: <EMC_TrapQueue_c::append> entering. ======> append trap# 60303 to trap queue;
getHdlrLevel=5, getTrapLevel=5, #=174
Defect information—Collect ooemc, nts, and snmpcomm log files from the /opt/svplus/log directory.
Possible alternative workaround—Manually resync the node. If the problem persists, perform a coldstart. If the problem is still not resolved, collect the log files and report the problem.
J.10.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems
This section includes the following information:
•CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics
•Basic Issues
•Topology Discovery Issues
•Network Element Discovery Issues
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues
J.10.5.1 CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics
CTM manages MGX NEs. The topology module discovers the nodes and trunks. Once the nodes are discovered, the EM module syncs up with the nodes to discover the card, line, port, and so on. All the discovered nodes, trunks, and node elements are published by NMServer and are displayed in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report. The Configuration Center, Chassis View, Diagnostics Center, or Statistics Report have the tree view and Inspector View. The tree view and Inspector View feed off the data published by NMserver.
CTM is notified of all subsequent changes in the network through traps.
Figure J-1 shows the end-to-end architecture.
Figure J-1 CTM End-to-End Architecture
NMServer provides inventory and alarm information for the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report. The node information is pushed by topod to NMServer. The initial inventory of card, lines, ports, and so on, is read from the database. The dynamic updates are pushed to NMServer from topod, ooemc, and sdbroker.
There are two utilities under the /opt/svplus/util/ directory that are shipped with CTM to debug issues in NMServer:
•nmControl—This utility provides the means to check the NMServer cache and its state. Output is redirected to /opt/svplus/log/nmControl.log. Cache dumps are redirected to /opt/svplus/log/nmControl.dump. The utility allows the following operations, all of which are nondestructive and perform the tasks in a passive mode.
Note Do not use the All Cache operation to dump the cache for a large network that has more than 1000 nodes.
–Resync Node—Resyncs the node (specified by the node_id).
–All Cache—Dumps all the data in its cache to the dump file.
–Topology Cache—Dumps topology (node) data to the dump file.
–Node Cache—Dumps a node's data (specified by node_id) to the dump file.
–CTM Alarm Cache—Dumps the CTM-specific alarms to the dump file.
–Client Manager Cache—Dumps the information about all the clients to the dump file.
–Error Statistics—Dumps the error information associated with the sync-up to the dump file.
–Sync-Up Status—Dumps the sync-up state to the dump file.
–Configuration—Dumps the configuration data to the dump file.
•nmClient—This utility is used to isolate an issue between client and server. It is used to query the NMServer the same way as the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI queries the server. Output is redirected to /opt/svplus/log/nmClient.log. Cache dumps are redirected to /opt/svplus/log/nmClient.dump.
You must perform the Register Client operation before performing any other operations. When you are finished using the client, perform the Unregister Client operation.
–Register Client—Registers with the server.
–Update Filter—Updates the subscription/filter with the server passing the FDN.
–Unregister Client—Unregisters with the server.
–Get Topology—Gets the topology information of networks and nodes.
–Get Children—Gets the list of children for a particular object in the tree.
–Get CwmInfo—Gets the CTM sync-up state information.
–Get ManagedObject—Gets the detailed information about an object in the tree.
–Subscribe for all events—Subscribes to all updates on the server.
–Unsubscribe for all event—Unsubscribes from all updates on the server.
The GUI client's log files (CMSCclient.log) are saved under log directories; for example, in Windows, D:Documents and Settings\<username>\log\, or in Unix: /opt/svplus/log.
J.10.5.2 Basic Issues
This section includes the following information:
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show Any Nodes
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Is Unable to Connect to Server
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Newly Added Node
J.10.5.2.1 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show Any Nodes
Step 1 Check whether the nodes are discovered. See No Nodes Are Discovered.
Step 2 Check whether NMServer has the nodes and trunks in its cache. This can be done by using the command nmControl in the CTM CLI.
Step 3 If the nodes are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI.
J.10.5.2.2 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Is Unable to Connect to Server
Step 1 Check whether the client registers in NMServer.log.
Step 2 Enable orbix logs by creating the orbix directory in /opt/svplus/log/ directory.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log, NMServer.log
•Orbix logs
Possible alternative workaround—None.
J.10.5.2.3 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Newly Added Node
Step 1 Check whether the nodes have been discovered. See No Nodes Are Discovered for more information.
Step 2 Check whether NMServer has the nodes in its cache. This can be done by using the command nmControl in the CTM CLI.
Step 3 If the nodes are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI.
J.10.5.3 Topology Discovery Issues
This section includes the following information:
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Node
•Node Information Incorrect on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Node that Is Not in the Network
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows Incorrect SyncState for a Node
•Duplicate Nodes Are Displayed on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
J.10.5.3.1 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Node
Step 1 Check whether the nodes are discovered. See No Nodes Are Discovered for more information.
Step 2 Check whether NMServer has the nodes in its cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 If the nodes or trunks are in the NMServer cache, open a new GUI and verify whether the nodes are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
J.10.5.3.2 Node Information Incorrect on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
Step 1 Check the node information in the database using selnd command.
Step 2 Check the node information in the NMServer's cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 Check the CMSCclient.log for the node information.
Step 4 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Step 5 If the node information is correct in the NMServer cache, open a new GUI and verify whether the node is shown correctly in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•selnd o/p
•nmControl.dump
•CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
J.10.5.3.3 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Node that Is Not in the Network
Step 1 Check the node information in the database by querying the node, packet_line table. Verify whether the node is active.
Step 2 If the node is active, see No Nodes Are Discovered.
Step 3 Check the node in the NMServer's cache. This can be done by using the command nmControl in the CTM CLI.
Step 4 Check the CMSCclient.log for the node/trunk. Verify whether a delete message was received for the node.
Step 5 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and GUI.
Step 6 If the node is not present in the NMServer cache, open a new GUI and verify whether the node is shown in it.
Defect Information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, and NMServer.log.
•Dump outputs of ILMITopoc, topod and NMServer. The dump can be captured by issuing a kill -USR1 signal to the process.
•Output of the switch CLI, selnd, and dbnds.
J.10.5.3.4 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows Incorrect SyncState for a Node
Step 1 Check the node information in the database by querying the node table. This can be done by using the command selnd on the CTM CLI. Check the status.
Step 2 Check the node in the NMServer's cache. This can be done by using the command nmControl on the CTM CLI.
Step 3 Check the CMSCclient.log for the node. Verify whether the correct message was received for the node.
Step 4 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Defect Information—Collect the following information for further analysis:
•ILMITopoc.log, topod.log, and NMServer.log.
•Dump outputs of ILMITopoc, topod and NMServer. The dump can be captured by issuing a kill -USR1 signal to the process.
•Output of the switch CLI, selnd, and dbnds.
J.10.5.3.5 Duplicate Nodes Are Displayed on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
Step 1 Check whether the node table has duplicate node IDs.
The node table can be queried using the following command:
tballraker18% echo "select (*) from node where node_id=2 and slot=1" | dbaccess stratacom
Step 2 Check whether both nodes in the database have active=1. If both nodes are active then collect the logs.
Step 3 If duplicate nodes do not exist in the database, check the cache of NMServer using nmControl.
Step 4 Check the CMSCclient.log for the node.
Step 5 From nmClient, use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI.
Step 6 If duplicate nodes are not present in the NMServer cache, open a new GUI and verify whether the node is shown in it.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under /opt/svplus/log. For PC Client, collect CMSCclient.log file under D:\Documents and Settings\<username>\log.
•Node data from node table for that specific node.
•ILMITopoc.log and topod.log, if there are duplicate nodes in the database.
Possible alternative workaround—None.
J.10.5.4 Network Element Discovery Issues
This section includes the following information:
•All the Cards Under a Node Are Missing
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Card, Line, or Port for a Node
•Card, Line, or Port Information Is Incorrect on Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
•Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Card, Line, or Port that Is Not Present in a Node
J.10.5.4.1 All the Cards Under a Node Are Missing
No cards are shown under a node in the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synched up, wait for it to do so.
Step 3 If the node has synced up, check whether cards are populated in the card table. If the card table is empty, collect the EM logs. See Equipment Management Problems for more information.
Step 4 Check whether NMServer has cards in its cache. This can be done by using the command nmControl on the CTM CLI.
Step 5 If cards are not present in the NMServer cache, collect the logs.
Step 6 If cards are present in the NMServer cache, use nmClient and verify whether getChildren for node's FDN returns the cards in the nmClient.<pid>.dump file.
Step 7 If cards are present in the NMServer cache, open a new GUI and verify whether the cards are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
J.10.5.4.2 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Does Not Show a Card, Line, or Port for a Node
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, wait for it to do so.
Step 3 If the node has synced up partially, check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 6.0 Database Schema.
Step 4 If the entry is not populated in the database, see Equipment Management Problems.
Step 5 If the entry is present in the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump
Step 6 If the element is not present in the NMServer cache, use nmClient and verify whether getChildren for FDN returns the entities in the nmClient.<pid>.dump file.
Step 7 If the element is present in the NMServer cache, open a new GUI and verify whether the missing elements are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
J.10.5.4.3 Card, Line, or Port Information Is Incorrect on Configuration Center, Chassis View, Diagnostics Center, or Statistics Report
The entity information of an element shows incorrectly on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, then wait for it to do so.
Step 3 Check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 6.0 Database Schema.
Step 4 If the entry in the database matches the GUI, see Equipment Management Problems.
Step 5 If the entry doesn't match the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump.
Step 6 Use nmClient and verify whether getChildren for FDN returns the correct information for the element in the nmClient.<pid>.dump file.
Step 7 If the element is present in the NMServer cache, open a new GUI and verify whether the missing elements are shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
2. Use nmControl and perform "Resync Node," specifying the node ID.
J.10.5.4.4 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Shows a Card, Line, or Port that Is Not Present in a Node
An extra card, line, or port element is shown incorrectly on the Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
Step 1 Check whether the node is synced up. This can be done by using the command selnd on the CTM CLI.
Step 2 If the node has not synced up, then wait for it to do so.
Step 3 Check whether the card, line, or port is populated in the database. Refer to the Cisco Transport Manager Release 6.0 Database Schema.
Step 4 If the element is present in the database but does not match what is shown in the GUI, see Equipment Management Problems.
Step 5 If element is not in the database, check the NMServer cache. This can be done by using the command nmControl on the CTM CLI. Check the nmControl.<pid>.dump.
Step 6 Use nmClient and verify whether getChildren for FDN returns the element in the nmClient.<pid>.dump file.
Step 7 If element is not present in the NMServer cache, open a new GUI and verify whether the extra element is shown in it.
Defect Information—Collect the following information for further analysis:
•topod.log, ILMITopoc.log, and NMServer.log
•nmControl.dump
•CMSCclient.log
Possible alternative workarounds:
1. Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report GUI.
2. Use nmControl and perform Resynch node, specifying the node ID.
J.10.5.5 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues
This section includes the following information:
•Alarm Processing Basics
•XML Schema for Alarm Rules
•Alarm Severity and Object Severity
•Severity Shown on Tree View Does Not Match Severity Shown on Platform
•Alarm List Shows Alarm that Does Not Exist on Platform
•Transient Event Has Disappeared Unexpectedly
•Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port
J.10.5.5.1 Alarm Processing Basics
Alarm processing is done within the NMServer processes. The following are some vital points with regard to alarm processing:
•Alarm rules are defined in an XML file (/opt/svplus/xml/ruledata.xml). For details on XML schema alarm rules, see Figure J-2.
•GUI clients register for alarms by providing the parent FDN.
•GUI clients pull initial alarms from NMServer. After initial pull of alarms, all alarms are pushed to registered clients by NMServer when alarms occur.
•Object severity for any and all network elements is determined by the XML alarm rule.
•Administrative states of network element states are not displayed in the alarm list, only alarm states.
•Alarm List displays mostly active alarms, however some events are also displayed. For details on events versus alarms, see XML Schema for Alarm Rules.
•Some alarms result in the network element having a different alarm severity then the actual alarm. For details on object severity versus alarm severity, see Alarm Severity and Object Severity.
Alarms for various network elements are stored in memory cache in the NMServer Object Tree. GUI clients register for alarms from NMServer by providing the FDN of the network element for which they would like alarms. All alarms for network elements under the FDN registered will be sent if the parent FDN is registered. GUI clients may also update alarms, for example by flagging an alarm as acknowledged or manually clearing an alarm. These updates are done through the Alarm Repository component of NMServer. Figure J-2 illustrates the client/server architecture for alarm components in NMServer.
Figure J-2 Client/Server Architecture for Alarm Components
J.10.5.5.2 XML Schema for Alarm Rules
Alarm rules are defined in XML, specifically, in a file named $HOME/xml/ruledata.xml. These alarm rules are read once when NMServer starts. When events are processed, the rules are queried from memory to determine what action, with regard to alarms, should be taken on the event. There are three types of alarm rules defined in the XML schema: Correlated, Correlated Bitmap, and Transient. Specifics on each of these three types are as follows.
1. Correlated Alarm Rule Type
The correlated alarm rule type is the most common in ruledata.xml. This rule is used when a network element can be in only one of many states at any one time. If the entity were to change states, then the previous alarm state would be cleared. Most network elements managed by CTM fall into this category. A simple example is the database sync-up status of a node. The sync-up status can be one of Partial Sync-Up, Sync-Up Failed, or In Sync. Any one of these states correlates out any other. See Figure J-3.
Figure J-3 Correlated Alarm Rule Diagram
A CorrelatedRule can have any of the following:
•1 AlwaysClear (Used when a given element never has an alarm, such as in a top-level network)
•0 or more ClearAlarmCondIds
•1 new alarm (which consists of NewAlarmConditionID, NewAlarmServAffect, and so on.)
2. Correlated Bitmap Alarm Rule Type
The correlated bitmap alarm rule type is different from the correlated alarm rule type because the bitmap rule type can represent the many states an entity can be in at any one particular time. The correlated bitmap rule type is used primarily to represent line alarms. Lines can have many different alarms associated with them at any given time, such as Loss of Signal and Loss of Frame. See Figure J-4.
Figure J-4 Correlated Bitmap Alarm Rule Diagram
A CorrelatedBmRule can have any of the following:
•0 or more ClearAlarmCondIds
•1 new alarm (which consists of NewAlarmConditionID, NewAlarmServAffect, and so on)
3. Transient Alarm Rule Type
The transient alarm rule type is used to distinguish events from alarms. The alarm list shows some events. Events are normally associated with the NMS itself. Examples of events are Process Restarted or Primary Gateway Disconnected. These are NMS events that are not correlated, but are displayed in the alarm list. Another example of a transient event that is not related to the NMS is a card switchover. If automatic protection switching (APS) is enabled on a card and there is an APS switchover, an event will be displayed in the alarm list. The primary distinction between transient events and correlated alarms is that transient events are not cleared by the network, whereas correlated alarms are. See Figure J-5.
Figure J-5 Transient Alarm Rule Diagram
A transient alarm rule can have only one new alarm associated with it.
Note There is no ClearAlarmCondID associated with a transient rule. Transient alarms can be manually cleared by the user, cleared by the same transient alarm occurring twice, or purged by NMServer when a high threshold is hit.
J.10.5.5.3 Alarm Severity and Object Severity
Every alarm in an alarm rules XML file is assigned two severities: object and alarm. The two severities are called Object and Alarm severity. These severities do match for most alarms. There are some cases, however, where these severities are different. The following is an example of an XML alarm rule where object and alarm severity do not match:
<CorrelatedRule State="1 -1">
<ClearAlarmCondID>10302</ClearAlarmCondID>
<ClearAlarmCondID>10303</ClearAlarmCondID>
<ClearAlarmCondID>10304</ClearAlarmCondID>
<NewAlarmCondID>10301</NewAlarmCondID>
<NewAlarmServAffect>0</NewAlarmServAffect>
<NewAlarmTransient>0</NewAlarmTransient>
<NewAlarmDbPersistent>0</NewAlarmDbPersistent>
<NewAlarmSev>6</NewAlarmSev>
<NewAlarmObjSev>7</NewAlarmObjSev>
<NewAlarmDescrSuffix>Sync-Up has not started yet</NewAlarmDescrSuffix>
This alarm will occur if the southbound processes (EMs) send a node message with EM sync-up status as 1 or -1. If that does occur, then the node should be unreachable severity (value 7) in the tree view. Note that there is no unreachable severity in the alarm list. This is why there are two severities for each alarm. Unreachable alarms have Critical (value 6) severity in the alarm list.
Another instance when alarm and object severity do not match is for aggregate port alarms. Aggregate port alarms are alarms that summarize the condition of the connections on the port. Since these alarms should not affect the severity of the port, the object severity of these is alarms is Clear (value 3). The following is the XML for an aggregate port alarm:
<CorrelatedBmRule StateBm="1">
<NewAlarmCondID>40801</NewAlarmCondID>
<NewAlarmServAffect>0</NewAlarmServAffect>
<NewAlarmTransient>0</NewAlarmTransient>
<NewAlarmDbPersistent>1</NewAlarmDbPersistent>
<NewAlarmSev>5</NewAlarmSev>
<NewAlarmObjSev>3</NewAlarmObjSev>
<NewAlarmDescrSuffix>Aggregate Port alarm, One or more connections on this port are in
primary failure</NewAlarmDescrSuffix>
Note The alarm severity (NewAlarmSev tag) is Major (value 5).
J.10.5.5.4 Severity Shown on Tree View Does Not Match Severity Shown on Platform
Severity of a network element in tree view does not match the severity the switch is showing for the same network element.
Step 1 Verify whether the customer is comparing the correct severity icon.
a. There are two severities associated with every object in the tree view, the aggregate severity and the self severity. The aggregate severity is on the left, the self severity is on the right. Since the switch does aggregation of many faults, the customer should compare the aggregate severity of the object in the tree view with the severity of the switch.
b. If you are looking at the correct severity icon and there is still a discrepancy between the severities, continue to step 2.
Step 2 Verify whether CTM has synced up with the node.
a. Log in to the CTM server and type selnd.
b. Verify that the node mode in question is 3.
Step 3 If the mode is 3, verify whether the discrepancy is caused by aggregated connection alarms. NMServer does more aggregation of alarms than the platform. For instance, NMServer aggregates connection alarms up the port and the switch does not. Therefore, if the tree view displays a higher severity then the switch, it may be caused by one or more aggregated port alarms.
a. Right-click the NE in the tree view and choose show alarms. All of the alarms for this NE and its children should appear in the alarm list.
b. Filter on the alarms that have greater severity then what the platform is showing.
c. Verify whether or not the alarms are aggregate port alarms. If so, this is expected behavior and there is no defect. If there are alarms that have greater severity then what is shown on the platform, and they are not aggregate port alarms, continue to the next step to see whether the alarm is in the database.
Step 4 Verify whether the database has the correct alarm state. Refer to the Cisco Transport Manager Release 6.0 Database Schema for information on what table to look up for this particular entity type.
Defect Information—Collect the following information for further analysis:
•NMServer.log
•nmControl.dump
•CMSCclient.log on client machine
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
J.10.5.5.5 Alarm List Shows Alarm that Does Not Exist on Platform
There are several alarms that are correlated by NMServer and are not handled by the switch. This is the case because the NMS keeps track of alarm conditions that may not be relevant to the switch alone, but are relevant to the switch and the NMS. One example of such an alarm is the node sync-up state. The switch itself is not interested in the fact that the NMS may not be synced up with it, but the NMS is interested in this information. Therefore if the node is still syncing up with the NMS, an alarm will be displayed for this node in the alarm list, but there will not be such an alarm on the platform. The following is a summary of alarms that may be seen in the alarm list, but not on the platform.
Step 1 If there is an alarm in the alarm list, and not on the platform, see if the suspect alarm is included in the following list:
•Node sync-up status alarms
•Node database sync-up status alarms
•Node Management State status alarms
•Node Aggregate alarm status
•Link0/Link1 Node alarms
•Card sync-up status alarms
•Aggregate Port (Connection) alarms
Step 2 If the suspect alarm is not included in any of the alarm list mentioned in step 1, it may be a defect. Perform the following:
a. Verify whether the database has the correct alarm state. See the Cisco Transport Manager Release 6.0 Database Schema for information on what table to look up for this particular entity type.
b. Collect the following information:
–topod.log, linktopoc.log, ILMITopoc.log, NMServer.log, and fileTopoc.log
–nmControl.dump
–CMSCclient.log
Possible alternative workaround—Open a new GUI and a new Configuration Center, Chassis View, Diagnostics Center, or Statistics Report.
J.10.5.5.6 Transient Event Has Disappeared Unexpectedly
Transient alarms behave somewhat differently depending on whether the entity is managed by the network element or the NMS itself. If the transient alarm is on a managed network element; for example, an alarm generated by an FTP transfer failure, then the alarm is self-clearing. This means if the same transient alarm happens more than twice, the previous active alarm is cleared by the latest.
If the transient alarm is not a managed network element, but rather the NMS itself, then the alarm is not self-clearing and there will be multiple occurrences of the same alarm conditions. Since these NMS alarms (or events) that pertain to the NMS are not cleared by the NMS, NMS alarms are not correlated. They must be cleared manually by the operator. If these events are not manually cleared, the list will grow only to the value of MAX_ACTIVE_NMS_EVENTS that is specified in the NMServer.conf file. Once the list of NMS alarms reaches MAX_ACTIVE_NMS_EVENTS, NMServer will automatically clear the oldest alarms in the list to make room for the new events. The number of events that will be cleared each time the maximum is reached is also specified in NMServer.conf. That configuration parameter is called EVENTS_TO_CLEAR_WHEN_MAX_REACHED.
If there was a transient event that has unexpectedly disappeared, it is most likely because NMServer purged the event.
Step 1 Filter the alarm list on CTM alarms.
Step 2 Check the alarm count and compare it with the MAX_ACTIVE_NMS_EVENTS value in NMserver.conf. If the alarm count is close to the MAX_ACTIVE_NMS_EVENTS, then this explains why the transient event has been purged.
Step 3 If the alarm count for NMS alarms has not reached the MAX_ACTIVE_NMS_EVENTS, the missing event may have been manually cleared. The NMServer log will indicate whether the event was manually cleared.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log, NMServer.log, and CMSCclient.log
•Capture NMServer dump, use nmControl
Possible alternative workaround—Restart the Alarm List GUI.
J.10.5.5.7 Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port
The tree view in CTM client GUIs displays network elements from the top-level physical view down to the port. Alarm severities are aggregated from children up to parents. Since the port is at the lowest level of the tree, the question that often arises is how a port can have an aggregate alarm if it has no children. The answer to this question is that connections are virtual entities under ports in the tree view. Virtually, you will not see connections in the tree. But connection alarms are aggregate up to the port in the tree view.
Step 1 Compare the connection alarms with the aggregate port alarms:
a. In the Configuration Center, click the Connections tab.
b. Find the port in the tree view and drag and drop it to the right window pane. The Connection view window appears.
c. Click the Get button. The connections should be listed, and the alarm status should match the alarm status in the alarm list.
Step 2 If the connection alarms do not match the aggregate port alarm(s) in Alarm List GUI, there may be a defect. Collect the following defect information:
•CMSCclient.log, NMServer.log, sdbroker*.log, xdbroker*.log
•Capture NMServer dump, use nmControl
Possible alternative workaround—None.
J.10.6 Chassis View Problems
This section includes the following information:
•Chassis View Basics
•Lines Not Displayed in the Chassis View
•Card Not Displayed in the Chassis View
•DCA/DCB Status Not Displayed on PXM Cards
•Ethernet Status Not Updated on PXM Cards
•HIST, CBRX, and CBTX Status Not Updated in MGX Nodes
•RPM Card Status Not Updated
•RPM Secondary Card Status Is Blue
•Lines Not Displayed on Secondary Card
•Lines Not Selectable
•Wrong Tooltip Is Displayed
J.10.6.1 Chassis View Basics
Chassis View is a read-only application that provides the physical view of WAN devices. For a specific node, it displays node, cards, and lines. Chassis View does not display ports. It can handle the following events:
•Node status changes
•Card status changes
•Line status changes
The alarms handled by Chassis View are:
•Card alarm
•Line alarm
Chassis View shows an empty slot in the chassis when displaying unsupported cards. For a reserved card, it shows the lines, but they are disabled. Chassis View does not show lines under a card when the card is in another state, other than active or standby.
Chassis View displays the chassis by combining static data from the XML file and dynamic data from the database. The first step in debugging is to determine if data is populated properly in the database. The next step is to check that the card or line is defined in the XML file.
J.10.6.2 Lines Not Displayed in the Chassis View
Step 1 Determine if data about the lines are populated for the corresponding lines in the line table in the database.
Step 2 Check whether the lines are defined in the XML file Chassis View.xml.
Step 3 Make sure that the line numbers in the database start from 1 and not from 2 or above.
Step 4 Make sure that the node is in sync (mode is 3).
If the details are not found in the database, this probably is an EM issue. If the details are found, then proceed as follows:
a. Use cwmver to get the CTM version.
b. Get the node table information for the node using the following command; then, save it in a file:
echo "select (*) from the node, where node_id=<NodeId> " | dbaccess
c. Get the card table information for the card using the following command; then, save it in a file:
echo "select (*) from the card, where node_id=<NodeId> and slot=<Slot>" | dbaccess
d. Save the line table information for the line using the following command:
echo "select (*) from the line, where node_id=<NodeId> and slot=<Slot>" | dbaccess
e. Save the log as CMSCclient.log to your local drive.
f. Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. You need to check that gif files used to draw the lines are available in the jar.
Possible alternative workaround—Select the lines from the tree view to launch other applications.
J.10.6.3 Card Not Displayed in the Chassis View
Step 1 Check that entries for the card are available in the card table.
Step 2 Check whether the lines are defined in the XML file ChassisView.xml.
Step 3 Make sure that the node is in sync (mode is 3).
If the details are not found in the database, this probably is an EM issue. If the details are found, then proceed as follows:
a. Use cwmver to get the CTM version.
b. Get the node table information for the node using the following command; then, save it in a file:
echo "select (*) from the node, where node_id=<NodeId> " | dbaccess
c. Get the card table information for the card using the following command; then, save it in a file:
echo "select (*) from the card, where node_id=<NodeId> and slot=<Slot>" | dbaccess
d. Save the log as CMSCclient.log to your local drive.
e. Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. You need to check that gif files used to draw the lines are available in the jar.
Possible alternative workaround—Select the cards from the tree view to launch other applications.
J.10.6.4 DCA/DCB Status Not Displayed on PXM Cards
DCA/DCB status is always grayed out in Chassis View. DCA/DCB status information is not received from the switch, so the status is not displayed or updated.
J.10.6.5 Ethernet Status Not Updated on PXM Cards
Ethernet status is received only when the Chassis View is launched. Dynamic changes in states of the Ethernet status are not updated in Chassis View.
J.10.6.6 HIST, CBRX, and CBTX Status Not Updated in MGX Nodes
For MGX nodes, only CPUOK LED status is updated in Chassis View. Chassis View does not manage the LEDs for HIST, CBRX, or CBTX.
J.10.6.7 RPM Card Status Not Updated
Dynamic event updates are not generated for RPM cards on MGX PXM1-based nodes, so the Chassis View does not get event updates on hot insertion or removal of RPM cards. The card will be identified when coldstart is done.
J.10.6.8 RPM Secondary Card Status Is Blue
For RPM cards, the standby state shows the card status in blue because the card has only one LED (CPUOK) to show the status of the card, unlike the other cards. For other types of cards, the standby state is indicated by a yellow LED.
J.10.6.9 Lines Not Displayed on Secondary Card
If two cards are in a redundancy relationship, the primary card (for example, the logical slot) is used to display the children and for all provisioning and troubleshooting activities, even if the primary slot becomes a standby. The secondary slot does not show any children under it, even if it becomes active. Hierarchical views in all applications behave in this manner. Similarly, provisioning is allowed only on the working line of an APS pair regardless of whether or not that line is currently active. However, monitoring is done on both working and protection lines.
J.10.6.10 Lines Not Selectable
Sometimes lines become unselectable. For example, when trying to select line 3, line 1 may get selected or vice versa.
Lines must be spaced sufficiently apart. If they are not, they may overlap, causing them to become unselectable.
Defect Information—Collect the following information for further analysis:
•Get the copy of the ChassisView.xml file used.
•Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. Verify that gif files were used to draw the lines.
Possible alternative workaround—Try selecting from the tree view.
J.10.6.11 Wrong Tooltip Is Displayed
When you move the cursor over a line, the wrong tooltip is displayed. For example, when moving the cursor below the last line in the card, the tooltip of the line gets displayed instead of the tooltip of the card.
Lines must be spaced sufficiently apart. If they are not, they may overlap, causing the wrong tooltip to be displayed.
Defect Information—Collect the following information for further analysis:
•Get the copy of the XML file used: ChassisView.xml for MGX nodes and BPXIGX.xml for BPX/IGX nodes.
•Take a copy of the chassisview.jar from the /opt/svplus/java/jars/cwm/ directory. Verify that gif files were used to draw the lines.
Possible alternative workaround—Try selecting from the tree view.
J.10.7 Configuration Center Management
The Configuration Center management functions are divided into the following categories:
•Network elements—Manages the nodes and their components. For network element management, the Configuration Center communicates with the Config Server process.
•Connections—Manages the connections between the nodes. For connection management, the Configuration Center communicates with the Connection Management (CM) Server process.
This section describes troubleshooting guidelines for the element management (Configuration Center; Elements Tab) section. The Connection Management (Configuration Center; Connections Tab) section describes troubleshooting guidelines related to Connection Management.
This section includes the following information:
•Configuration Center Framework
•Configuration Center—Element Management
J.10.7.1 Configuration Center Framework
The Configuration Center uses the CTM framework and workflow mechanism to launch applications and drag and drop objects across application.
•Network elements can be selected in the tree view and the Configuration Center (Elements Tab) can be launched to view or modify the selected object.
•Network elements can be dragged and dropped from other application to the Configuration Center's Elements tab for modification.
This section includes the following information:
•Cannot Launch the Configuration Center
•Cannot Launch Other Applications from the Configuration Center
•An Exception Is Raised when the Configuration Center Is Launched
•An Exception Is Raised when the Configuration Center Launches Another Application
•Element Tab—Internal Frame Does Not Launch
•Element Tab—Drag-and-Drop Functionality Does Not Launch An Internal Frame
•Element Tab—Create, Details, Modify, and Refresh Button Issues
•Element Management—Drag and Drop Within Configuration Center
•Cross Application—Configuration Center as Drag Source
•Cross Application—Configuration Center's Element Tab as Drop Target
•Element Tab—Internal Frame Displays Incorrect Object or Object Data
•Configuration Center's Element Tab Does Not Respond (GUI Is Grayed Out)
J.10.7.1.1 Cannot Launch the Configuration Center
The Configuration Center cannot be launched using one of the following methods:
•Click the Configuration Center icon from the Launch Center or from any application.
•Choose Tools > Configuration Center from any application.
•Right-click the selected node from the hierarchical tree, and choose Configuration Center.
Step 1 Check the configcenter.jar file. Make sure that the configcenter.jar file is located in the /opt/svplus/java/jars/cwm directory of the target machine.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.2 Cannot Launch Other Applications from the Configuration Center
The Configuration Center does not launch other applications using one of the following methods:
•Choose an application from the Tools menu item.
•Right-click on the selected object from the hierarchical tree and choose the target application.
Step 1 Check the target application jar file. Make sure the target application jar file is located in the /opt/svplus/java/jars/cwm directory of the target machine.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.3 An Exception Is Raised when the Configuration Center Is Launched
When the Configuration Center is launched using one of the following methods, an exception is raised and the Java Console shows the exception trace information:
•Choose an application from the Tools menu item.
•Right-click on the selected object from the hierarchical tree and choose the target application.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.4 An Exception Is Raised when the Configuration Center Launches Another Application
When the Configuration Center launches another application, using one of the following methods, an exception is raised and the Java Console shows the exception trace information:
•Choose an application from the Tools menu item.
•Right-click on the selected object from the hierarchical tree and choose the target application.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.5 Element Tab—Internal Frame Does Not Launch
When the Element tab is selected and you double-click on a supported network element, an internal frame is not created or the content of an existing internal frame is not recycled.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.6 Element Tab—Drag-and-Drop Functionality Does Not Launch An Internal Frame
When the Element tab is selected and you drag and drop supported network elements to the Element tab's content pane, an internal frame is not created or the content of an existing internal frame is not recycled.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.7 Element Tab—Create, Details, Modify, and Refresh Button Issues
When the Element tab is selected and an object is displayed in an internal frame, the Create, Details, Modify, and Refresh buttons either do not launch other internal frames for further provisioning or they do not update the user interface with data for the selected operation.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.8 Element Management—Drag and Drop Within Configuration Center
The drag-and-drop functionality of a network element from the Configuration Center tree view to the Element tab's content pane does one of the following:
•Fails to open an internal frame
•Fails to recycle the contents of an existing frame to display the object's attributes
•Results in the Operation Not Supported message box
Step 1 Determine if the drag-and-drop functionality is supported for the dropped object.
The following objects can be dropped from the tree view to the content pane:
•For the Element tab, the Network, Node, Card, Line, Port, IMA, and IMA link objects are supported and Folder objects are not supported.
•For the Connection tab, the Node, Card, Line, and Port objects are supported and Folder, IMA, and IMA link objects are not supported.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information, specifically any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.9 Cross Application—Configuration Center as Drag Source
As drag source, the drag-and-drop functionality of an object from the Configuration Center's tree view or Element tab to another CTM application fails to display the selected object in the target application.
Step 1 Determine if the drag-and-drop functionality is supported for the selected object.
Step 2 Determine if the target application supports the drag-and-drop functionality for the dropped object.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information, specifically any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.10 Cross Application—Configuration Center's Element Tab as Drop Target
The drag-and-drop functionality of a network element from another CTM application to the Configuration Center's Element tab content pane does one of the following:
•Fails to open an internal frame
•Fails to recycle the contents of an existing frame to display the dropped object's attributes
•Results in the Operation Not Supported message dialog box
Step 1 Determine if the drag-and-drop functionality is supported for the dropped object.
The following objects can be dropped from the tree view to the content pane:
•For the Element tab, the Network, Node, Card, Line, Port, IMA, and IMA link objects are supported and Folder objects are not supported.
•For the Connection tab, the Node, Card, Line, and Port objects are supported and Folder, IMA and IMA link objects are not supported.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information, specifically any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.11 Element Tab—Internal Frame Displays Incorrect Object or Object Data
The Element tab successfully creates the internal frame but either it displays information related to another object or the object's attribute values are not valid.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information, specifically any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.1.12 Configuration Center's Element Tab Does Not Respond (GUI Is Grayed Out)
The Configuration Center's Element tab does not respond and the GUI is grayed out.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file from the /opt/svplus/log directory.
•Configserver.log file from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.7.2 Configuration Center—Element Management
The Configuration Center can be used to configure different network element objects. The Configuration Center's Element tab is the main window for creating, modifying, and viewing network elements. The Element tab uses the CTM Internal Frame mechanism to display the attributes (groups and categories) associated with a particular network element. The following sections describe guidelines for troubleshooting issues related to creating, modifying, and viewing network elements.
J.10.7.2.1 XML Parsing Error
XML parsing error is seen while launching the Configuration Center (either by the drag-and-drop method or the right-click method) for a network element. The popup window says:
Internal Error: XML Parsing Error.
Step 1 Verify whether the network element, on which you want to launch the Configuration Center, is supported in the CTM version being used. If this error is seen for a supported node/card go to step 2.
Step 2 Open the /opt/svplus/log/configserver.log file and look for the message information showing when this error occurred. A typical message in the log looks like the following:
ERR: Fatal Error at file, line 0, char 0, Message: An exception occurred!
Type:RuntimeException, Message:The primary document entity could not be opened.
Id=/opt/svplus/xml/configcenter/XXX/XXX-XXX.xml
( <someNumber>: <x>) <someTime Stamp> ERR: InternalError: XML Parsing Error
If the .xml filename mentioned above has two consecutive hyphens (for example: ABC--XYZ.xml) or if it has a preceding hyphen (for example: -ABC.xml) or terminates with hyphen before the file extension (for example: ABC-.xml) proceed to step 4.
Step 3 See if the .xml file mentioned in the log message exists. If it does not exist, contact CTM engineers.
Step 4 Perform the following substeps to investigate incorrectly formed XML filename strings. The format of XML filenames is <Platform>-<Card>-<InterfaceType>-<SomeEntityName>.xml. This format is generic with a few exceptions. Also note that Platform and InterfaceType are optional and will not be seen in many files. For example, ABC-Card.xml is a valid XML filename.
a. If the Platform part of the XML filename is incorrect/missing, check the Node table to see if it is correctly populated.
b. If the Card part of the XML filename is incorrect/missing, check the Card table to verify if it is correctly populated.
c. If the Interface Type part of the XML filename is incorrect/missing, check the appropriate table to verify if it is correctly populated. This table generally corresponds to the Line or Port table of the card for which this error occurred.
d. If the Entity Name part of the XML filename is incorrect, contact CTM engineers, with the defect information required.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•/opt/svplus/log/configserver.log.
•Dspcd command output from the switch for the controller card and the service module where this error is detected.
•Ddspln or dspport command output, if this error seen for a line or port.
Possible alternative workaround—None.
J.10.7.2.2 SNMP No Data Error
An SNMP NO DATA error is seen while launching the Configuration Center (either by the drag-and-drop method or the right-click method) for a network element.
See if the network element for which this error occurred is active, and that the node is synced up. To do this, use the selnd command on the CTM machine.
Check whether the correct community strings are used. If this error is noticed for a properly synced up node on an active element, perform the following:
Step 1 Use an SNMP Manager tool to determine whether the switch is responding to SNMP queries.
The SNMP MIB objects to be queried for a particular dialog box can be obtained from the /svplus/log/configserver.log file. Refer to the entries in this log file when this error occurs and perform the following:
a. Using HP-OV SNMP operations, perform a walk (on "system" MIB table for this example, using public community string) as /opt/OV/bin/snmpwalk -c public nodeName system.
b. If an error is seen during the SNMP operations (from the SNMP manager tool), check the switch to verify if this information should really be present. If yes, proceed to step 2.
Step 2 Collect the information from the /svplus/log/configserver.log file when this error occurs. Perform the following:
a. From the configserver.log file, identify the MIB objects that caused the error.
b. Identify the MIB to which these objects belong.
c. Verify whether these MIB objects are supported in this release. If the objects are supported, but still the SNMP agent is not responding for SNMP queries, contact CTM engineers.
Defect Information—Collect the following information for further analysis:
•Complete screen snapshots for the investigative commands used for debugging this issue.
•Configserver.log.
•Related information on the switch.
Possible alternative workaround—None.
J.10.7.2.3 Details, Create, Delete, and Refresh Buttons Are Not Enabled
The Details, Create, Delete, and Refresh buttons are enabled or disabled based on the requirement in the tabular view. These buttons are disabled when a corresponding operation is not supported.
Step 1 Open the /opt/svplus/log/configserver.log file for the MIB.
Step 2 If the Create button is disabled, use any SNMP tool to create the object using the SNMP SET operation.
•If the SNMP SET operation passes, contact CTM engineers.
•If the SNMP SET operation fails, SSH or Telnet to the CLI and try to add the element. If the operation succeeds, contact CTM engineers.
Step 3 If the Delete button is disabled, use any SNMP tool to delete the object using the SNMP SET operation.
•If the SNMP SET operation passes, contact CTM engineers.
•If the SNMP SET operation fails, SSH or Telnet to CLI and try to delete the element. If the operation succeeds, contact CTM engineers.
Step 4 The only reason why a Details button is disabled is that all the information related to the element is already displayed in the tabular view. Check the configserver.log file to determine if any MIB entry appropriate to the element is available. If a MIB entry is available, perform an SNMP walk on the MIB entry using any SNMP tool. If there are is information in addition to what is displayed in the tabular view, and if it is important to be displayed in the frame, contact CTM engineers.
Step 5 The only reason why a Refresh button is disabled is that all the information displayed in the frame is read-only. If there are any editable fields and the Refresh button is disabled, contact CTM engineers.
Defect Information—Collect the following information for further analysis:
•Complete screen snapshots for the investigative commands used for debugging this issue.
•Configserver.log
Possible alternative workaround—For creating or deleting elements, use either the SNMP SET operation on the CTM machine or CLI commands on the switch.
J.10.7.2.4 SNMP Timeout Error
This error occurs when you try to set or get values to or from SNMPCOMM and the values look like the following:
Check the sync-up, link0, link1 status of the Node in the inspectorview.
Step 1 If the status is Unmanaged, Unreachable, Failed in Sync, or Partially Synced, check the connectivity of the node using the ping command, Telnet, or any other utility.
Defect Information—Collect the following information for further analysis:
•Complete screen snapshots for the investigative commands used for debugging this issue.
•Configserver.log
Possible alternative workaround—None.
J.10.7.2.5 SNMP Set Error
This error occurs when the SNMP SET operation is done on an uneditable MIB object or when you use invalid values.
Step 1 Open the /opt/svplus/log/configserver.log file.
Step 2 Check for the last part of the document name.
For example, if the document name is AAA-BBB-CCC.xml, then the document AAA-BBB-CCC.xml can be found in the /opt/svplus/xml/configcenter/CCC/ directory.
Step 3 If the file is not available, contact CTM engineers.
Step 4 Open the document you got in step 2 and do the following:
•Check the Access field of the MIB object in the XML file.
•Check the Access field in the /opt/svplus/mibs directory.
If the access field from XML file and MIB definition are not same, contact CTM engineers.
Step 5 Check the Range field of the MIB object in the XML file and MIB definition. If they are not the same, contact CTM engineers.
Step 6 Perform an SNMP SET operation on the MIB object with the same value shown in the GUI and appropriate community string (as displayed in /opt/svplus/log/configserver.log) using HP-OV. A syntax of the SNMP SET operation is shown as follows:
/opt/OV/bin/snmpset -c private nodeName MIBVariable MIBDataType MIBValue
Step 7 If the SNMP SET operation succeeds, contact CTM engineers with appropriate defect information.
Step 8 SSH or Telnet to the switch and try to configure the same parameter with the value given in the GUI and SNMP tool. If the CLI does not report errors, contact CTM engineers with defect information.
Defect Information—Collect the following information for further analysis:
•Complete screen snapshots for the investigative commands used for debugging this issue.
•Configserver.log
Possible alternative workaround—None.
J.10.7.2.6 Object Not Found In Tree View Error
This error occurs when a network element is launched from the tabular view of the Configuration Center.
Step 1 Collect the information of the object being launched. This can be done in two ways.
•In the tabular view, there will be some minimal information such as node name, slot number, line number, trunk number, port number, and so on.
•Some minimal information regarding the object being launched will be displayed in the title of the window in one of the following formats:
–For a node, the window title will have <NodeName>
–For a slot, the window title will have <NodeName>.<SlotNumber>
–For a line, the window title will have <NodeName>.<SlotNumber>.<LineNumber>
–For a port, the window title will have <NodeName>.<SlotNumber>.<PortNumber>
Step 2 With the information obtained from Step 1, see if that object appears in the tree view.
•If it does appear, go to step 3.
•If it doesn't appear, refresh the tabular view two or three times and see if the object still exists in the tabular view, and if it continues giving the same error, contact CTM engineers with the appropriate defect information.
Step 3 If the object being launched is a node or card, verify whether that particular node or card is supported in the CTM version being used.
Step 4 If the node or card is supported, open the /opt/svplus/log/CMSCclient.log file and check whether the FDN of the object being launched matches the information from Step 1. Contact CTM engineers with appropriate defect information. A typical example of the log follows:
[DD Mon YYYY HH:MM:SS] DEBUG - [appsConfigCenter] :: CcElementInternalFrame :
modifyActionPerformed()
[DD Mon YYYY HH:MM:SS] DEBUG - [appsConfigCenter] CcTabularView : modify()
[DD Mon YYYY HH:MM:SS] INFO - [appsConfigCenter] Selected fdn =
/NW=X/RND=XX/CD=X/LN=X/PT=X
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>og directory.
•Java Console information.
•Contents of /opt/svplus/log/configserver.log file.
Possible alternative workaround—None.
J.10.7.2.7 Element Data Inconsistent with Switch
Information displayed in the GUI and the switch is not the same.
Step 1 Make sure that the window has been launched for the correct element in the GUI. If not, contact CTM engineers with the defect information.
Step 2 SSH or Telnet to the switch, change a few parameters, and see if the change is reflected in the GUI. If not, contact CTM engineers.
Step 3 Open the /opt/svplus/log/configserver.log file and find the MIB OID of the element that displays the inconsistent data.
Step 4 See if the MIB OID is pointing to the information that you are looking for. If not, contact CTM engineers.
Step 5 Use any SNMP tool to perform an SNMP GET operation on the MIB object which shows inconsistent. If the value retrieved using the SNMP GET operation and the value shown in the GUI are not the same, contact CTM engineers. The following shows the syntax of the SNMP GET operation in a HP-OV:
snmpget [-Cf] [options...] <hostname> {<community>} [<objectID> ...]
Step 6 Check the data type of the MIB object that shows inconsistent data. If the data type of the MIB object in the configserver.log is not the same as the data type defined in the MIB object definition (available at /opt/svplus/mibs/ directory), contact CTM engineers with defect information.
Step 7 Check the data type of the MIB object:
a. If the data type of the MIB object is Bitmap, then convert the decimal value of the MIB object to a binary value and compare the bits SET/RESET with the values defined in the MIB object definition and displayed in the GUI. Contact CTM engineers with defect information.
b. If the data type of the MIB object is an IpAddress, verify whether the value is in the appropriate format, for example, <aaa.bbb.ccc.ddd>. If the value is not in proper format, contact CTM engineers.
Defect Information—Collect the following information for further analysis:
•Screen snapshots for the investigative commands used for debugging this issue
•/opt/svplus/log/configserver.log
Possible alternative workaround—None.
J.10.7.2.8 Config Server Reports Error Messages
This section describes the various kinds of errors that could happen on the config server.
Step 1 Open the /opt/svplus/log/configserver.log file.
Step 2 If data type mismatch occurs, an error message is displayed in the GUI and the corresponding error message seen in configserver.log file is as follows:
( 24560: 9) 07:49:14 CRIT: %SnmpCommException-2-NOT_INTEGER_VALUE:
SnmpVar::getInteverValue() incorrectly called for ASN type [4].
In this case, the data type of the MIB object defined in the XML file is not the same as the data type of the MIB object definition. Contact CTM engineers with defect information.
Step 3 If you enter an invalid value, an error message is displayed in the GUI with the parameter name appended. The corresponding error message seen in configserver.log file is as follows:
( 17252: 6) 08:31:20 ERR: %SnmpCommException-3-ERR_BADVALUE: Snmp Error[3]: Bad Value;
object [] (index[2]) :
Step 4 If you try to up/add more than one element, an error message is thrown in the GUI and the corresponding error message thrown in the /opt/svplus/log/configserver.log file is as follows:
( 23868: 6) 14:01:09 ERR: %SnmpCommException-3-ERR_GENERR: Snmp Error[5]: General Error;
object [] (index[0])
Defect Information—Collect the following information for further analysis:
•Collect all complete screen snapshots for the investigative commands used for debugging this issue.
•Contents of /opt/svplus/log/CMSCclient.log file.
Possible alternative workaround—None.
J.10.8 Connection Management Problems
This section includes the following information:
•Configuration Center GUI—Framework Issues
•Configuration Center GUI—CM Server-Reported Errors
•Cmsvr Errors
•CMGRD
J.10.8.1 Configuration Center GUI—Framework Issues
The Configuration Center management functions are divided into the following categories:
•Connections—Manages the connections.
Note For connection management, the Configuration Center communicates with the Connection Management Server process.
•NE—Manages the nodes and their components.
Note For network element management, the Configuration Center communicates with the Config Server process.
The Configuration Center's Connection tab is the main window for creating, modifying, and viewing network connections. The Connection tab content pane provides additional tabs for creating connections (Advanced Mode and Quick Mode tabs) and retrieving a list of existing connections (Connection tab).
The Configuration Center uses the CTM framework and workflow mechanism to launch applications and to create and display connection information.
•Connections can be selected in CTM applications, and the Configuration Center (Connection Tab) can be launched to view/modify the selected connection.
•Connections can be dragged and dropped from other application to the Configuration Center's Connection tab for modification.
The following sections describe guidelines for troubleshooting issues related to creating, modifying, and viewing network connection using the Configuration Center.
J.10.8.1.1 Connections Tab—Double-Click Operation on Tree View Does Not Launch Connection's Internal Frame
The double-click operation on a network element in the tree view on the Connection tab (a) fails to open an internal frame, (b) fails to recycle the contents of an existing internal frame to display the double-clicked connections attributes, or (c) results in the Operation Not Supported message dialog box.
Step 1 Determine if the double-click operation is supported for the selected object.
The Connection tab supports connection management for the Node, Card, Line, and Port objects. The drag-and-drop functionality of Folder, IMA, and IMA link objects is not supported.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file from the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.2 Connections Tab—Drag and Drop Does Not Launch Connection's Internal Frame
The drag-and-drop functionality of a network element from the Configuration Center tree view to the Connections tab's content pane (a) fails to open an internal frame, (b) fails to recycle the contents of an existing frame to display the dropped object's attributes, or (c) results in the 'Operation Not Supported' message dialog box.
Step 1 Determine if the drag-and-drop functionality is supported for the dropped object.
The Connection tab supports the drag-and-drop functionality for the Node, Card, Line, and Port objects. The drag-and-drop functionality of Folder, IMA, and IMA link objects is not supported.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.3 Connection List Tab—Cannot Launch Internal Frame to Modify an Existing Connection
The Details button fails to launch an internal frame to display the connection details for a retrieved connection.
Step 1 Check that the connection status is not in the Incomplete state.
Only connection in the OK and Fail states can be launched using the Details button on the Connection tab.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.4 Advanced Mode Tab—Cannot Launch the Connection Details Dialog Box Using the Details Button
The Details button on the Advanced Mode tab fails to launch the Connection Details dialog box.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.5 Advanced Mode Tab—Cannot Launch the Template Details Dialog Box Using the Template Details Button
The Template Details button on the Advanced Mode tab fails to launch the Template Configuration dialog box.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.6 Cross Application—Cannot Launch Other Applications from the Connection List
For a retrieved connection on the Connection tab, the right-click popup menu does not launch the selected target application for the connection or it results in the Operation Not Supported message dialog box.
Step 1 Determine if the operation is supported by the target application.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.7 Cross Application—Connection List as Drag Source
As drag source, the drag-and-drop functionality of a connection from the Configuration Center's connection list to another application fails to display the selected connection in the target application.
Step 1 Determine if the target application supports connection object drop operations.
Step 2 Determine if the target application supports the drag-and-drop functionality for connection objects.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.8 Cross Application—Connection Tab's Content Pane as Drop Target
The drag-and-drop functionality of a connection object from another application to the Configuration Center's connection tab content pane (a) fails to open an internal frame, (b) fails to recycle the contents of an existing frame to display the newly dropped connection object, or (c) results in the 'Operation Not Supported' message dialog box.
Step 1 Determine if the source and the target application belong to the same instance of the Java virtual machine (VM). CTM framework does not support drag-and-drop functionality across applications belonging to different Java VMs.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.1.9 Configuration Center's Connection Tab Does Not Respond
The Configuration Center's Connection tab does not respond and the GUI is disabled.
Step 1 Try to reproduce the problem in order to collect additional Java-related data using the Java DOS window.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.2 Configuration Center GUI—CM Server-Reported Errors
For connection management, the Configuration Center communicates with the Connection Management (CM) Server process to (a) create, modify, delete, or retrieve connections and (b) create, modify, delete, or retrieve a connection template. The CM Server throws application exceptions if it cannot successfully complete the requested operation. The CM Server exception contains an error message that describes the error condition. The Configuration Center displays the reported error message for further analysis. In such cases, the problem must be researched on the CM Server side. The CM Server troubleshooting sections describe the steps required to further analyze the CM Server-related issues. See Connection Management Problems. In this section, the required validation and data collection procedures related to the Configuration Center are presented.
J.10.8.2.1 Connection Creation, Modification, Deletion, and Retrieval Errors
When performing any connection creation, modification, deletion, and retrieval, the CM Server could encounter an error and notify the Configuration Center by throwing an application exception error that contains information describing the error. As a result, the Configuration Center displays the error message. The following lists some CM Server-related exceptions.
DATABASE_ERROR:"Database error [%s]."
UNKNOWN_EXCEPTION: "Program Error. Unknown Exception is caught in [%s]."
SUBSYSTEM_EXCEPTION:"Program Error. Subsystem exception [%s] caught in [%s]"
INTERNAL_ERROR: "Program or System Error. Function [%s] reports: [%s]"
CONNECTION_LOST: "Connection from host [%s] application [%s] lost during request."
UNKNOWN_PARAM_VALUE_TYPE: "Unknown parameter value type [%d] is processed in [%s]."
PARAM_TYPE_VALUE_MISMATCH: "Parameter value [%s] is does not match type [%d]"
VALUE_OUT_OF_RANGE: "Value [%d] is outside of the valid [%d] - [%d] range"
NO_SUCH_CONNECTION_TYPE_ENUM: "No such connection type enum [%d]"
NO_SUCH_SERVICE_TYPE_ENUM: "No such service type enum [%d]"
NO_SUCH_ENDPOINT_TYPE_ENUM: "No such endpoint type enum [%d]"
NO_PORT_TABLE_INFO_FOR_CARD: "No port table information for card family [%d]"
NO_LINE_TABLE_INFO_FOR_CARD: "No line table information for card family [%d]"
FILTER_REQUIRED:"Filter [%s] is required"
UNEXPECTED_PARAMSET: "CM Server does not expect parameter set [%d]."
UNSUPPORTED_ENDPOINT: "CM Server does not support end point with card type [%s], vs[%s],
interface type [%s], node platform [%s], controller type [%s] for connection type [%s]
service type [%s], endtoend type [%s]."
UNSUPPORTED_PARAMETER_FOR_VISM: "Unsupported parameter setting on the [%s] end. CM Server
does not support [%s] + [%s] PVC's for VISM cards with firmware revision less than [%s]
and card mode set to [%s]."
PROTECTED_CONNECTION: "[%s] end of the connection is protected. Protected connections
cannot be deleted."
UNSUPPORTED_NODE_PLATFORM: "CM Server does not support node platform [%d]."
UNSUPPORTED_CONN_TYPE: "CM Server does not support Connection of type [%d]."
UNSUPPORTED_SERV_TYPE: "CM Server does not support Connection type [%s] , Service type
[%s] and Endtoend type [%s] between Card type [%s] and [%s]."
UNSUPPORTED_SERV_TYPE_ONE_END: "CM Server does not support Connection type [%s] , Service
type [%s] and Endtoend type [%s] on Card type [%s]."
UNSUPPORTED_CARD_PAIR: "Connection between card type [%s] and card type [%s] is not
supported."
NO_SUCH_NODE_ID_IN_DB: "Node id [%d] not found in table [node]."
NO_SUCH_NODE_NAME_IN_DB: "Node name [%s] not found in table [node]."
NO_SUCH_CARD_IN_DB: "Card : Node[%d], Slot[%d] not found in table [card]."
NO_SUCH_LOG_PORT_IN_DB: "Card : Node[%d], Slot[%d], LogPort[%d] (%d - in database), not
found in table [%s]."
NO_SUCH_PHY_PORT_IN_DB: "Card : Node[%d], Slot[%d], Line[%d], PhyPort[%d] (%d - in
database), not found in table [%s]."
NO_SUCH_LINE_IN_DB: "Card : Node[%d], Slot[%d], Line[%d] not found in table [%s]."
NO_SUCH_CONNECTION_IN_DB: "Connection [%s] not found in database"
UP_CONN_FAILED: "Upping Connection Failed. Test Result from switch [%s]"
DOWN_CONN_FAILED: "Downing Connection Failed. Test Result from switch [%s]"
NO_ONE_SEGMENT_HYBRID: "Cannot create one segment hybrid connection."
INCORRECT_LINE_SUBTYPE: "Incorrect Line SubType. Line: Node[%d],slot[%d],ifindex[%d] is
[%d] not [%d]."
INVALID_LINE_NUMBER: "func [%s]:Invalid Line Number(s)."
NO_CONTROLLER_IN_PORT: "node[%s] slot[%d] port[%d] does not have controller of type [%s]";
INVALID_FDN: "The given FDN format or value is invalid. [%s]"
INVALID_PORT: "Provisioning operations are not allowed on feeder trunk/routing trunk and
XLMI trunk ports. [%s]"
Step 1 Verify that the CM Server and sdbroker are up and running by entering psg sdbroker and psg cmserver on the command line.
Step 2 Verify if the CmServer restarted while the operation was being done by checking the cmsvr.log and searching for the string "START OF THE PROCESS". Verify that the time stamp of the server process was restarted while the operation was being performed. Also check for any cmsvr coredumps under the /opt/svplus/corefilesdir/ directory.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.2.2 Connection Template Creation, Modification, Deletion, and Retrieval Errors
When performing any connection template creation, modification, deletion, and retrieval, the CM Server could encounter an error and notify the Configuration Center by throwing an application exception error that contains information describing the error. As a result, the Configuration Center displays the error message. The following lists some CM Server-related connection template exceptions:
•CONN_TEMPLATE_ALREADY_PRESENT: "The Connection Template[%s] is already present in the conn_template table."
•CONN_TEMPLATE_NOT_PRESENT: "The Connection Template[%s] is not present in the conn_template table."
•CONN_TEMPLATE_TABLE_INCONSISTENT: "The Connection Template table contains more than one entry for the template[%s]."
•CONN_TEMPLATE_PARAMS_NOT_PRESENT: "The Connection Template[%s] is not present in the conn_templ_param table."
•DATABASE_ERROR:"Database error [%s]."
•UNKNOWN_EXCEPTION: "Program Error. Unknown Exception is caught in [%s]."
•SUBSYSTEM_EXCEPTION: "Program Error. Subsystem exception [%s] caught in [%s]."
•INTERNAL_ERROR: "Program or System Error. Function [%s] reports: [%s]."
Step 1 Verify that the CM Server, sdbroker, and xdbroker are up and running by entering psg sdbroker, psg xdbroker, and psg cmserver on the command line.
Step 2 Verify if the CM Server restarted while the operation was being done by checking the cmsvr.log and searching for the string "START OF THE PROCESS". Verify that the time stamp of the server process was restarted while the operation was being performed. Also check for any cmsvr coredumps under the /opt/svplus/corefilesdir/ directory.
Defect Information—Collect the following information for further analysis:
•CMSCclient.log file under the D:\Documents and Settings\<username>\log directory.
•Java Console information; in particular, any Java-raised exceptions.
•Screen snapshots; in particular, error/information messages dialog boxes.
•Cmsvr.log file under the /opt/svplus/log directory.
•Configserver.log file under the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.3 Cmsvr Errors
The following are some of the most common errors seen when a provisioning request is issued:
•End-to-End Type Cannot Be PVC for Connections with PNNI Segment
•Unsupported Service Type for the Foresight Connections for PNNI/SPVC Endpoints
•CE-CE Connection Port Speed Should Be the Same on Both Endpoints
•Cannot Create One-Segment Hybrid
•Cannot Add Connection Between -byte Port Header and -byte Port Header for chan type NIW and NIW-replace
•Unsupported Service Type—General node-slot-port-version
•Endpoint Is Not Present in One of the Connection Tables During a Modify Operation
•Cmsvr—Connection Diagnostic Issues
•TestConn or TestDelay Failed with Switch Error
•Connection Up/Down/Reroute Failed
•Connection Trace Failures
J.10.8.3.1 End-to-End Type Cannot Be PVC for Connections with PNNI Segment
When an add connection is attempted, cmsvr checks its the endpoints and the end-to-end type to make sure it is supported. The error message states "EndtoEndType cannot be PVC for connections with a PNNI segment."
The problem must be fixed on the CM GUI while adding the connection.
Step 1 Verify that the endpoints that are being used to add the connection are not on PNNI nodes or feeders to PNNI nodes.
Step 2 Change the end-to-end type in the CM GUI to either SPVC or Hybrid.
Defect Information—Collect the cmsvr logs from the /opt/svplus/log directory.
Possible alternative workaround—None.
J.10.8.3.2 Unsupported Service Type for the Foresight Connections for PNNI/SPVC Endpoints
During connection addition, the error message could state "CM Server does not support Connection type[], Service type[], and Endtoend type []between Card type and []." The parentheses are filled with the appropriate values.
Step 1 Determine if the service type is applicable to the endpoints.
Step 2
