Cisco Prime Optical User Guide, 10.6

Prime Optical Installation Fails Before the Database Is Created

If the Prime Optical installation fails before the database is created, do the following before reinstalling:

• Use the ps -ef | grep setup command to verify that the previous installation is not running. The installation script is setup.sh. If the installation process is still running even after you exit InstallAnywhere, terminate the installation with the kill -9 pid command, where pid is the process identifier that is returned by the ps -ef | grep setup command.
• Verify that the /tmp directory is not full. During the Prime Optical installation, the /tmp directory is used to store a temporary copy of the installation scripts and the JRE used for installation. Use the df -k command to check if there is enough space on the other file systems (for example, in the /opt directory).
• Make sure that you follow the installation procedures explicitly. Even minor steps, such as creating the /tftpboot directory, must be completed. If you deviate from the documented installation procedures, you will encounter problems during installation.
• If you are not installing Prime Optical from the CD—for example, you are installing Prime Optical from your hard drive—make sure that all of the required scripts and files are available and have the correct ownership and permissions.

Note It is not recommended that you install Prime Optical from CD files copied to your hard drive.

You can use the /temp/dbinit.log for debugging when a new Prime Optical installation fails.

Prime Optical Installer Hangs or Quits

Check the disk space in the root and user installation directories. Also, check the disk space in the /tmp and /temp directories. If disk space is not a problem, check for available RAM.

Default Group Name Does Not Appear During New Installation

Verify the following when the default group name (root/other) does not appear in the Group Info panel during a new installation, or, if supplied, an “Invalid Groupname” error message appears.

• The /tmp folder is created in the correct directory.
• The installation scripts are executable. If not, use the chmod a+r* and chmod a+x* commands.

Insufficient RAM when Specifying the Destination Directory

Use the prtconf command to check your RAM and validate it against the specifications listed in the Cisco Prime Optical Installation Guide .

SQL Errors

If you encounter SQL errors, check the log or err files in the /temp or /tmp directories and do one of the following:

• Check whether the /opt/Cisco*Server/cfg/CTMServer.cfg file exists.
• Make sure that the database password is correct. Use the password to connect to Oracle.

SQL problems occur when there is no connection to Oracle or there are Oracle command syntax violations in SQL.

Uninstaller Quits While Uninstalling the Prime Optical Server

While uninstalling the Prime Optical server, the uninstallation process might quit with the following error message: “CTM server is running.” Use the opticalctl status command to check whether the Prime Optical server is running. If it is running, use the opticalctl stop command to stop the server and relaunch the uninstaller.

Things to Keep in Mind when Installing or Uninstalling Prime Optical

• After the main installation phase in the GUI, wait for several minutes while certain files and scripts are copied to the installation directory. Before proceeding, wait for the message “Please reboot your system.”
• Stop the Prime Optical server before starting the migration process.
• Stop the Prime Optical server before uninstalling it.
• The upgrade network size configuration is allowed only in medium-to-large, medium-to-high end, and large-to-high end configurations.
• The Prime Optical GateWay/CORBA option is disabled during migration. It will be reinstalled if it was previously installed.
• Use the /opt/Cisco*Server/UninstallerData/installvariables.properties file to make sure that all of your specifications during the preinstallation process are correct.
• All installation files are available in the /tmp or /temp directories.

PM Data, Audit Log Data, and Error Log Data Are Not Collected Correctly—CTM R7.0 and Earlier

Collect the following information:

• Error logs in the /opt/CiscoTransportManagerServer/logs directory
• Prime Optical database size

The following cases indicate that the problem is related to the partitioning of the database tables:

• The size of the Prime Optical database is medium, large, or high end.
• The log files contain ORA-01440 errors.

If the problem is related to the Audit Log or Error Log , collect the output of the following query:

If PM data is not collected correctly, collect the output of the following query:

An example of the last line of the query result is P01012007. This means that the latest partition was added on January 1, 2007. You have to manually add the partitions if the date when you performed the query is not January 1, 2007.

Database Crashes

If the database crashes, obtain a copy of all of the log files from the /oracle/admin/CTM/bdump and /oracle/admin/CTM/udump directories. The .trc file located in the udump directory contains the session status logs. The alert_CTM.log file located in the bdump directory contains the issues related to the different database processes and gives the process description with a time stamp.

Database Troubleshooting Tools

Checkdbinfo.sh Script

To debug Oracle performance issues, use the checkdbinfo.sh script located in the /opt/CiscoTransportManagerServer/bin directory. The /oracle/dbcheck.log file contains the output of the script.

Toad

Run a procedure using Toad and you can check the values of the procedure variables.

Prime Optical Database Functions

The displayphyloc and displayip functions that are defined in the Prime Optical database can be used to view data in a readable format. For example, the following query displays the list of installed nodes and the related IP address in readable format:

See Chapter 3 of the Cisco Prime Optical Database Schema for more information.

Tnsping Utility

For remote databases, performance issues might cause process timeouts. Use the tnsping utility to confirm that process timeouts occur.

Conditions that Affect Prime Optical Server Performance

The “System Requirements” chapter in the Cisco Prime Optical Installation Guide lists server, CPU, CPU speed, disk space, and RAM requirements for different types of Prime Optical installations. These requirements are derived from guidelines based on scalability simulation testing.

Actual Prime Optical server performance varies for each customer and is affected by:

• The total number of NEs actively managed by Prime Optical
• The number of NE-related services (including northbound services) running on the server
• The rate of circuit provisioning and the methods used for provisioning
• The rate of network growth (for example, adding 50 NEs to Prime Optical at once)
• The rate of configuration change updates received from the NEs
• The number of circuits provisioned with no signals that generate threshold crossing alerts (TCAs)
• The number of unacknowledged alarms in the Prime Optical database
• The rate of alarm bursts received from the NEs
• The actual number of circuits provisioned in the Prime Optical database

The following conditions might indicate that your Prime Optical installation is near or at capacity:

• Your daily CPU utilization is consistently above 80%
• You experience loss of connectivity (LOC) to 1% to 5% of nodes at random
• Your RAM usage is 100% or your system swaps frequently
• You receive many configuration updates from the NEs
• Your system experiences high Java garbage collection time

Note The Java garbage collection time is determined by engineering after analyzing your CPU utilization data.

If you are experiencing any of these conditions, contact your account representative, who can engage Cisco’s Advanced Services team to audit your Prime Optical server and recommend a higher-capacity server and resources.

Prime Optical Server Does Not Respond

Step 1 Log in as the root or optusr user on the host where the Prime Optical server is installed.

Step 2 Enter the following command to view the status of the Prime Optical server processes:

If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:

If there is a line containing /CTMServer, the Prime Optical server is running.

If there is no line containing /CTMServer, the Prime Optical server is not running. Proceed to Step 3.

Note You can also check the ctmop.log file in /opt/CiscoTransportManagerServer/log to determine whether the server was stopped by another user or if it stopped abnormally. If it stopped abnormally, proceed to Step 3.

Step 3 Start the Prime Optical server by using the opticalctl start script in the /opt/CiscoTransportManagerServer/bin directory.

a. Log in as the root user.

b. Change the directory to /opt/CiscoTransportManagerServer/bin and enter the following command:

If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:

If the preceding procedure does not solve the problem, complete the following steps:

Step 1 Verify whether the /opt/CiscoTransportManagerServer/cfg/CTMServer.cfg file is corrupt. The file must contain the db-config-mode = auto parameter in the [database] section. If the entry is missing, the Prime Optical server configuration file is corrupt. Reinstall the Prime Optical server.

Step 2 Verify whether the first entry in the /var/opt/oracle/oratab file looks similar to CTM8_5:/oracle/product/10.2.0:Y. If this entry is missing, the Oracle database might not be installed. The Oracle database is a prerequisite for installing the Prime Optical server.

Step 3 To improve system performance, regenerate the database statistics, either before or after a significant system workload. See Regenerating Statistics in the Database. Internal Oracle statistics allow Oracle to work efficiently, especially during data query operations. If you experience Prime Optical system performance degradation during normal database activities, the database might be using stale statistics.

Cannot Connect to the Prime Optical Server

Step 1 Ping the server’s IP address from the client PC or workstation.

Step 2 If the ping fails, resolve the IP connectivity problem and try again.

Step 3 Telnet to the server and log in as the root user.

Step 4 Enter the following command to verify that Prime Optical is running:

If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:

Step 5 The server must have at least the following processes running:

Step 6 If you see fewer than four processes running, enter the following command to stop the server manually:

Step 7 If you changed the server IP address, verify that the configuration files have been updated.

Step 8 Verify that the Oracle database is accepting connections:

a. Enter the following command to log in as the Oracle user:

b. Enter the following command to open an SQL*Plus session:

Step 9 Reboot the server if you receive another error message and the SQL prompt does not appear. Wait for the server to boot up and try to run the client. The SQL prompt indicates that the Oracle database is running and accepting connections.

Step 10 Enter the following commands to restart Prime Optical manually:

If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot, enter the following command:

Step 11 Wait for 5 minutes and run the client.

How Do I Restart the Prime Optical Server when the Network Contains Many NEs?

If your network contains a high number of NEs, it often takes a long time for all of the NEs to synchronize after a Prime Optical server restart. To avoid this NE synchronization delay, complete the following steps:

Step 1 In the Domain Explorer , choose Administration > Control Panel .

Step 2 In the Control Panel window, expand the NE Service node and click any NE with a green arrow (which indicates that there are services running).

Step 3 In the Status tab, click the Deactivate button to deactivate the NE service. This ensures that when the Prime Optical server is restarted, no service consumes too much CPU and the NE resynchronization goes smoothly.

Step 4 For CTC-based NEs, expand the NE Service node and click CTC-Based SDH NEs or CTC-Based SONET NEs . In the Status tab, click the Deactivate button to deactivate the network service.

Step 5 Expand the PM Service node and click any NE with a green arrow. In the Status tab, click the Deactivate button to deactivate the PM service.

Step 6 Enter the opticalctl status command. The command output must not show any active NE or PM services. The output must look similar to the following sample:

Step 7 Enter the opticalctl stop command.

Step 8 After the opticalctl stop command is complete, re-enter the opticalctl status command to verify that there are no Prime Optical processes running. The output must look similar to the following sample:

Step 9 If any process is still running, identify the process ID and enter the following command to stop it:

Step 10 After all processes have been stopped, enter the opticalctl start command at the command prompt. This command starts the Prime Optical server but not the NE or PM services.

Step 11 Activate the NE service instances individually after most of the NEs in a partition have started synchronizing. That is, wait until the NE service instance displays an hourglass icon in the Control Panel tree before you activate the next NE service instance. To do this, expand the NE Service node; then, expand the appropriate NE node and in the NE Service Instance properties pane, click the Start Service Instance button.

Step 12 For CTC-based NEs, activate the network service instances individually after most of the NEs in a partition have started synchronizing. To do this, expand the NE Service node; then, expand the CTC-Based SDH NEs or CTC-Based SONET NEs node. In the Network Service Instance properties pane, click the Start Service Instance button.

Step 13 Activate the PM service instances. To do this, expand the PM Service node; then, expand the appropriate NE node and in the PM Service Instance properties pane, click the Start Service Instance button.

NE Connection State Is Listed as Unavailable

If the connection state of an NE is listed as Unavailable in the Domain Explorer window, a connectivity or configuration problem exists. Wait 5 to 10 minutes after adding the NE to the Prime Optical domain; then, complete the following steps:

Step 1 To see the NE IP address, select the NE in the Domain Explorer window. The Address tab of the Network Element Properties pane lists the IP address of the selected NE.

Step 2 From the Prime Optical server, enter the following command to verify connectivity between the Prime Optical server and the NE:

Step 3 If the ping fails, a physical or configuration problem exists in the data communications network (DCN).

a. If connectivity existed earlier:

• If there were no configuration changes made to any routers in the DCN or to the Prime Optical server, the problem is a physical problem.
• If the configuration was changed, the change might have introduced problems. Verify the changed configuration.

b. If connectivity was never established:

• Verify that there are no problems on the physical level.
• Verify the DCN configuration.

Step 4 If the ping succeeds, verify that the NE software version is listed in the Supported NE table ( Administration > Supported NE Table).

Launching Tables Results in Database Errors

If the Oracle database or Oracle listener that Prime Optical is using is down, launching tables will generate database errors. To troubleshoot table launching errors:

Step 1 Log in as the Oracle user and enter the following command:

If the login succeeds, an SQL> prompt appears, which indicates that the Oracle database has been installed and the database server is up and running. If the login fails, either the Oracle database has not been installed or the database server is not running.

Step 2 To start the Oracle database, log in to the Linux workstation as the Oracle software user and enter the following command at the shell prompt:

Step 3 Enter the following command at the shell prompt to start the Oracle listener:

If there are still problems with starting the Oracle database or the Oracle listener, refer to the Oracle documentation or contact Oracle support.

SNMP Traps Are Not Forwarded from NEs

SNMP traps might not be forwarded, either because the trap port is already in use, or because the NE is not configured correctly.

Trap Port Is Unavailable

The Prime Optical server requires exclusive access to the SNMP trap port to receive SNMP traps from the NE.

Step 1 Enter the following command to verify that the standard SNMP trap port (port 162) is not in use by another application running on the same workstation:

If the following line is present, the SNMP trap port is in use by another application:

Step 2 If the trap port is in use by another application, stop the other application.

Performance Monitoring Data Is Not Displayed

Step 1 Use the Domain Explorer > Network Element Properties pane to verify whether PM collection is disabled for the selected NE.

Step 2 Choose Administration > Control Panel and expand PM Service . Select the NE type to verify whether the service status is Active or Not Active.

Step 3 Choose Administration > Audit Log to verify whether PM was ignored for the selected NE.

Step 4 Choose Administration > Control Panel and click PM Service . Look at the PM Data Storage field. Choose Optimized to save only nonzero values in the database. Choosing Optimized requires less space than saving all values, including zero values. If you choose Normal in the PM Data Storage field, the database saves all PM values, including zero values.

Note The Optimized PM data storage feature is available only for CTC-based NEs.

Step 5 Choose Administration > Error Log to verify whether there are any critical, major, or minor errors against the PM service.

Step 6 Verify that the NE is reachable during the duration that PM data is collected. If the NE is not reachable, there is an entry in the Audit Log table ( Administration > Audit Log ).

CTC-Based NE Is Not Discovered

Step 1 Verify that the NE is up and running.

Step 2 Verify that the NE IP address and default route are configured correctly.

Step 3 To verify that the NE is running the correct version of system software, open a CTC session to the NE and check the NE software version.

Step 4 Verify that the NE that acts as the gateway NE (GNE) (through which this node is discovered) is added to Prime Optical and is available.

Step 5 Open CTC from the GNE and verify that CTC can discover the NE.

CTC-Based NE Is Not Reachable

Step 1 Verify that the NE is up and running.

Step 2 Verify that the NE IP address and default route are correctly configured.

Step 3 Wait for five poll cycles while Prime Optical re-establishes connectivity with the NE.

Step 4 To test IP connectivity to the NE from the Prime Optical server, enter the following command from the workstation where Prime Optical is running:

Step 5 To verify that the NE is running the version of system software supported by Prime Optical, open a CTC session to the NE and check the NE software version.

Step 6 Verify that the username and password that Prime Optical uses to reach the NE exist on the NE.

ONS 1530x Communication Problems

Step 1 Make sure that there are no other ONS 1530x NEs with the same name.

Step 2 Test the IP connectivity from the server to the NE.

Step 3 Telnet to the NE from the Prime Optical server. If you are prompted for a password, the NE is running.

Step 4 Connect to the NE using Cisco Edge Craft to verify that SNMP is enabled.

Note Only users connecting from servers that are listed in the SNMP Community table are allowed to log in to the NE.

Step 5 Verify that the NE you are trying to connect to is running a software version supported by Prime Optical.

Step 6 Check whether the product name is one of the following:

• ONS 15305
• AXXEDGE

If the product name does not match any of the recognized names, enter the change-system-name command.

Step 7 For ONS 15305 CTC R3.x NEs, note the following possible connectivity issues and limitations:

• The username and password that Prime Optical uses to reach the ONS 15305 CTC must exist on the NE.
• ONS 15305 CTC NEs support up to two simultaneous client connections, as follows. Other configurations are not supported and might cause connectivity problems:

– One Prime Optical plus one CTC connection

– Two Prime Optical connections

– Two CTC connections

• CTC cannot be launched on the same workstation where the Prime Optical server is running and managing ONS 15305 CTC NEs, or connectivity problems might occur.
• ONS 15305 CTC NEs do not support the ONS 15454 SDH proxy server implementation. If ONS 15305 CTC NEs are connected to ONS 15454 SDH NEs through optical links, enabling the proxy server on ONS 15454 NEs (and then the GNE/ENE configuration) causes connectivity problems on the ONS 15305 CTC NEs. Verify that proxy server is not enabled on any ONS 15454 SDH NEs that are optically connected to ONS 15305 CTC NEs.

Changing the NE Operational State from the UNIX CLI

This procedure is useful when troubleshooting NE connectivity problems, alarm, or configuration synchronization issues and you do not have access to the Prime Optical client GUI.

Step 1 Use the vwdata.sh script to identify the NE database ID number.

Step 2 Run the UNIX CLI tool and connect to the server. The tool name is ctm and is located in the /opt/CiscoTransportManagerServer/bin directory.

Step 3 Use the command set nestate to change the NE operational state.

Circuits Are Not Displayed

If Prime Optical fails to display some circuits, or if the information displayed by CTC and Prime Optical differs, complete the following steps:

Step 1 Wait for 2 minutes while the Prime Optical server synchronizes with the NE. If the Circuit Table is not in autorefresh mode, click the Refresh Data tool when the tool flashes.

Step 2 Verify that the Circuit Table is opened from either the source or destination NE of the circuit to be viewed.

Step 3 Verify that the NEs that are part of the circuit (source, destination, or drop) are available in Prime Optical.

Step 4 Open another CTC session and compare the circuit information in the newly opened CTC session with the circuit information displayed by Prime Optical.

Circuit State for Monitor Circuits Reads “Duplicate ID”

Prime Optical generates a unique number that is appended to the circuit name to make the monitor circuit name unique. On rare occasions, Prime Optical might create two or more monitor circuits with the same name. The circuit state for these circuits reads “Duplicate ID.” This is a known issue that has been tracked as DDTS number CSCdz87566.

When the circuit state reads “Duplicate ID,” you cannot see the actual circuit state. In this case, you must change the duplicate name to a unique name, so that you can see the correct circuit state. Use the Modify Circuit wizard to enter a unique name for each monitor circuit.

NE Model Type Appears as Unknown

If an in-service NE is added to the Domain Explorer but the model type appears as unknown, the software version of the NE might not be prepopulated in the database. In other words, Prime Optical cannot match the NE with a recognizable version.

Use the NE Software Table to add the NE software version string to the Prime Optical database. See Viewing Software Versions and Restarting the NE with a New Software Image.

Cannot Copy the CTC Binary to the Prime Optical Server

If a CTC binary fails to copy to the Prime Optical server, the Prime Optical-install-directory /cms/ directory might be missing or write-protected. If the directory is missing, create a cms directory with write-access permissions. Otherwise, change the permissions of the existing cms directory to allow write access.

Memory Backup, Memory Autobackup, or Memory Restore Fails

Step 1 In the Domain Explorer window, choose Administration > Job Monitor . The Job Monitor Table shows the status of the operation. If a job failure occurs, the reason is displayed in the Additional Information column.

Step 2 Return to the Domain Explorer window and choose Administration > Audit Log . The Audit Log table shows the status of the operation.

Step 3 Return to the Domain Explorer window and choose Administration > Error Log . The Error Log table shows information about the backup, memory restore, or download failure.

Note Sometimes, when a node is busy, a memory backup, memory autobackup, or memory restore job request may fail. In such a scenario, Prime Optical automatically creates a new, similar job and resends the request using the job retry functionality. The new job is triggered within the first 10 minutes of failure detection. Prime Optical attempts to resend the request a maximum of five times. You can monitor a rescheduled job in the Job Monitor table.

Step 4 See Error Messages for the correct action to take in response to the error.

Software Download, Software Commit, or Software Revert Fails

Step 1 In the Domain Explorer window, choose Administration > Job Monitor . The Job Monitor table shows the status of the operation. If a job failure occurs, the reason is displayed in the Additional Information column.

Step 2 Return to the Domain Explorer window and choose Administration > Audit Log . The Audit Log table shows the status of the operation.

Step 3 Return to the Domain Explorer window and choose Administration > Error Log . The Error Log table shows information about the autobackup, software commit, or revert failure.

Note Sometimes, when a node is busy, a software download, software commit, or software revert job request may fail. In such a scenario, Prime Optical automatically creates a new, similar job and resends the request using the job retry functionality. The new job is triggered within the first 10 minutes of failure detection. Prime Optical attempts to resend the request a maximum of five times. You can monitor a rescheduled job in the Job Monitor table.

Step 4 See Error Messages for the correct action to take in response to the error.

Database Is Not Available

If the Prime Optical client cannot connect to the Prime Optical server, verify that the database is available.

Step 1 Log in to the Prime Optical server as the Oracle user.

Step 2 Enter the following command to connect to the database:

Step 3 If the error message “maximum processes exceeded” is received, the maximum number of database connections has been reached. Close several clients or ask the database administrator to increase the maximum number of processes for the database.

Database Timeout Occurs

Step 1 Reduce the scope of the query by selecting a group or an NE and not the entire domain before opening tables such as the Alarm Browser , Alarm Log , Audit Log , and PM tables.

Step 2 Increase the client database query timeout period by editing the ems-client.cfg file in the C:\Cisco\TransportManagerClient\config or /opt/CiscoTransportManagerClient/config directory.

Step 3 Add hardware resources to the Oracle database server.

Step 4 Ping the Oracle database server from the client to verify the response time. Increase the bandwidth if the round-trip response time is inadequate.

Are the Prime Optical Client and the Prime Optical Server Connected?

If the database is available, check connectivity between the client and the server.

Step 1 To see the Prime Optical server IP address, enter the following command on the Linux workstation where the Prime Optical server is running:

The command output looks similar to the following example:

The IP address is the address following the inet field.

Step 2 To verify that the physical connection between the Prime Optical client and the Prime Optical server does not have problems, enter the following command from the Prime Optical client:

where the IP address belongs to the Linux workstation where the Prime Optical server is running.

Step 3 If the ping command fails, fix the physical connectivity; then, log in to the Prime Optical client.

Cannot Log In as Provisioner or Operator

Step 1 Use the default username and password to log in to the Prime Optical client:

Username: SysAdmin

Password: Ctm123!

Step 2 If the SysAdmin login succeeds but the provisioner or operator login fails, verify that the provisioner or operator exists and is not disabled. In the Domain Explorer window, choose Administration > Users to view a table of all configured Prime Optical users.

Step 3 If the provisioner or operator is not in the Cisco Prime Optical Users table, the user is not configured. Configure the provisioner or operator; then, log in as that user.

Step 4 If the provisioner or operator is in the Cisco Prime Optical Users table, select the row corresponding to that user and click the Modify User Properties tool to bring up the Modify User Properties wizard. Verify that the Login State field is set to Enabled. If the login state is disabled, enable it and log in as the provisioner or operator. The user might have been disabled after attempting to log in with an incorrect password.

Step 5 If the password is not correct, set a new password for the user and log in again.

Cannot Authenticate User Message Appears

If the “Cannot authenticate user” error message is received when logging into the Prime Optical client, the Prime Optical server might be initializing. Wait for 5 minutes while the Prime Optical server finishes initializing; then, try to log in again. Alternatively, check your username and password and enter them again. The username and password are case-sensitive.

Socket Over TL1 Issues

Prime Optical supports NEs that are connected in non-IP networks such as Open System Interconnection (OSI) networks. The Cisco NEs will be connected through non-Cisco gateway NEs (NGNEs) and Prime Optical opens a tunnel to connect to the NGNEs. Do the following in case of connectivity issues in a tunnel setup:

Step 1 Ping the NGNE. The NGNE must be connected to the IP network.

Step 2 Check if the NGNE requires login credentials to connect to the network. If it requires login credentials, its TID is listed in the GNE TID text box in the Domain Explorer NE Properties pane. The username and password of the NGNE are specified in the Domain Explorer NE Properties pane and Control Panel .

Step 3 Check that the NGNE encoding mode that is used to connect to the Cisco NEs is correct.

Step 4 (Optional) If you have a very large network (which can slow down the creation of tunnels), do the following to increase the default timeout value of 20 seconds:

a. In the Control Panel window, expand the NE Service and click CTC-Based SONET NEs or CTC-Based SDH NEs to open the NE Service pane.

b. In the Status tab, enter the new timeout value in the TL1 Tunnel Connection Timeout text box.

c. Click Save .

Model Index Is Unknown

If a new NE is added to the Prime Optical domain and marked as In Service, but the model index is unknown, complete the following steps:

Step 1 Verify that the IP address and other parameters are entered correctly.

Step 2 The Prime Optical server might not be able to establish connectivity to the GNE for the NE. Verify that the GNE for the NE is marked as In Service, and that the GNE is correctly populated in the Prime Optical domain.

Step 3 Verify that the GNE is DCC-connected to the NE.

Step 4 The Prime Optical server might not recognize the NEs software version. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity . If the NE is available, contact the Cisco TAC and verify that the version of the software on the NE is compatible with the Prime Optical server. Use the Supported NE table to provide the new NE software version that the Prime Optical server will recognize.

Step 5 The Prime Optical server might be configured to attempt communication with new NEs after a certain delay. To change the health poll frequency:

a. In the Domain Explorer window, choose Administration > Control Panel and click NE Service .

b. In the NE Poller tab, look at the NE Health Poll Interval field. The default is 240 seconds. If the value is too large, there might be a communication delay between Prime Optical and the NEs. Under normal conditions, the Prime Optical server attempts communication with new NEs for no more than twice the number of seconds specified in the NE Health Poll Interva l field.

Cannot Delete an NE

If an NE cannot be deleted, verify that you logged in as a SuperUser or NetworkAdmin, and not as a Provisioner or Operator. Provisioners and Operators cannot delete NEs. If you logged in as a Provisioner or Operator, restart the Prime Optical client session and log in as a SuperUser.

CTC Fails to Start

If CTC cannot be started for a CTC-based NE, complete the following steps:

Step 1 If an error occurs while opening the NE Explorer but the CTC progress screen and login dialog boxes appear, the CTC username or password might be incorrect. Do the following:

a. Choose Administration > Users .

b. In the Cisco Prime Optical Users table, choose Edit > View/Modify . The Modify User Properties wizard opens.

c. Click Next to move to the CTC/Craft User Properties pane and change the CTC username and password to match those configured on the device. Click Finish .

Step 2 If the CTC progress screen idles for a long time before the login dialog box appears, there might be problems with network connectivity to the device.

a. From a DOS command window or a Linux workstation, ping the device.

b. If no response is received, set up routes so that the device is available; then, restart CTC. If CTC does not start, the workstation might have resource constraints.

c. Close some open applications and try again.

d. Close some instances of CTC that are managing another group of CTC-based NEs and try again.

Step 3 If the GNE for the selected NE was not found, there might be a mismatch between the GNE specified for the device and the available GNEs. This means that the data in the database is corrupt. Contact the Cisco TAC for assistance.

Added a New Software Version to the Wrong NE

Step 1 In the Domain Explorer window, choose Administration > Supported NE Table .

Step 2 In the Supported NE table, select the incorrect entry and choose Edit > Delete .

Step 3 Click OK in the confirmation dialog box.

Step 4 Select the correct NE row from the Supported NE Table .

Step 5 Add the correct software version.

Step 6 In the Domain Explorer > Network Element Properties pane, set the operational state of all NEs that are behaving erroneously to Out of Service .

Step 7 Click Save .

Step 8 In the Network Element Properties pane, set the operational state of all the NEs back to In Service .

Step 9 Click Save .

Cannot Delete a Subnetwork

If a subnetwork cannot be deleted from the Subnetwork Explorer , complete the following steps:

Step 1 Verify that the user logged in as a SuperUser—not as a Provisioner or an Operator. Provisioners and Operators cannot delete subnetworks. If the user logged in as a Provisioner or Operator, restart the Prime Optical client session and log in as a SuperUser.

Step 2 Verify that the target subnetwork does not contain NEs. Move all NEs to another subnetwork before deleting the empty target subnetwork.

Cannot Move an NE Between Subnetworks

When automatic grouping of NEs in subnetworks is enabled, you cannot move NEs between subnetworks. To move NEs from one subnetwork to another:

Step 1 In the Domain Explorer window, choose Administration > Control Panel.

Step 2 In the Control Panel , click UI Properties .

Step 3 Under Subnetwork Grouping, uncheck the Automatically Group NEs in Subnetworks check box.

Step 4 Click Save .

NEs Change Subnetworks Rapidly

NEs that are part of a subnetwork might change subnetworks rapidly if the status of the link or of all links connecting subnetworks fluctuates rapidly; this can occur if automatic grouping of subnetworks is enabled in Prime Optical. Disabling the automatic grouping of subnetworks might prevent NEs from changing subnetworks when caused by links going up and down. To disable the automatic grouping of subnetworks:

Step 1 In the Domain Explorer window, choose Administration > Control Panel .

Step 2 In the Control Panel , click UI Properties .

Step 3 Under Subnetwork Grouping, uncheck the Automatically Group NEs in Subnetworks check box.

Step 4 Click Save .

Cannot Schedule Jobs

The Prime Optical client can schedule the following types of administrative tasks:

• Software download
• Memory backup
• Memory restore

Prime Optical maintains an Error Log and Audit Log to track potential problems. To view the Error Log or Audit Log :

Step 1 In the Domain Explorer window, choose Administration > Audit Log or Error Log .

Step 2 Look for errors related to software download, memory backup, or memory restore.

Prime Optical Does Not Receive Autonomous Alarms from ONS 1530x NEs

Prime Optical can collect alarms or receive autonomous notifications from an ONS 1530x NE. Alarm collection occurs when the NE is first discovered and does not rely on spontaneous events sent from the NE.

Step 1 Verify that the Prime Optical server is present in the SNMP Community table and that the trap is set to enable. If not, you can connect to the NE, but Prime Optical will not receive traps.

Step 2 If the Prime Optical server and the NEs are in different subnetworks, make sure that there is no firewall filtering SNMP packets between the subnetworks.

Step 3 Verify that Cisco Edge Craft is receiving autonomous alarms.

Step 4 Verify the following:

• SNMP traps reach the Prime Optical server machine.
• The command snoop udp and port 162 do not show the incoming traps.
• There are no network problems.

Step 5 Check whether the Prime Optical server is receiving traps.

Step 6 Check whether the ONS 1530x NE service is receiving SNMP traps.

Bandwidth-Intensive Operations Are Blocked

Bottlenecks in the DCN affect bandwidth-intensive operations, such as software download. Tune the timeout and retry values to match DCN performance. Log in to the NE and enter the following commands:

• ip tftp min-timeout —Sets the minimum wait time, in seconds, before retrying the read/write request.
• ip tftp max-timeout— Sets the maximum wait time, in seconds, before retrying the read/write request.
• ip tftp backoff-delay —Sets the number of seconds to extend the wait time if the read/write request times out.
• ip tftp write-retries —Sets the maximum number of times to retry the TFTP read/write request before declaring a failure.

NE Displays Incorrect Configuration Management Data

Step 1 In the Domain Explorer window, choose Administration > Service Monitor and verify that the NE Service is running. If it is not running, start the NE Service.

Step 2 Test NE connectivity:

a. In the Domain Explorer tree, select the NE and choose Fault > Test NE Connectivity . The message “ NE-name ( IP-address ) is available” is generated.

b. If the NE is unavailable, establish connectivity to the NE before retrieving configuration data.

Cannot Customize the Network Map

If an image file is not displayed while changing the Network Map background or while changing a node icon, complete the following steps:

Step 1 Choose another image file. The file might be corrupt.

Step 2 Check the size of the image file. The image file might be larger than 100 KB, which is too big to load. If the file is too big, use a smaller image file.

Step 3 Verify that the image file exists in the \images\mapbkgnds\shapefiles directory or the \images\mapicons directory. If the file is missing, it has been deleted. Reinstall the Prime Optical client.

Note The client is bundled primarily with shape files (*.shp) and only a few map background GIF files.

Prime Optical Client Machine Displays Incorrect Colors

When using a UNIX machine as the Prime Optical client host, the machine’s display might show incorrect colors. This problem occurs when UNIX systems use a 256-color palette and the graphics require colors that are not available in the 256-color scheme. Although some UNIX systems can work around the missing colors, there are still some that have problems making adjustments. For these systems, the only solution is to disable any other graphical application (for example, Netscape) that is running and upgrade the system graphics capabilities to a 24-bit color display (for example, upgrade the video card or monitor).

Common Topology Problems

Note After marking NEs as Out of Service, wait until the circuits traversing over these NEs (and not involving any other NE) clear from the Prime Optical Circuit table before marking the NEs as In Service again.

Cannot Load the Barebone Configuration on ML Cards

To load a fresh barebone configuration on an ML card, complete the following steps:

Step 1 In the Domain Explorer tree, select the NE that contains an ML card and choose Configuration > NE Explorer (or click the Open NE Explorer tool).

Step 2 In the NE Explorer tree, select the ML card (or double-click the ML card that is shown in the Shelf View).

Step 3 In the Slot Properties pane, click the Configuration tab.

Step 4 Click the File >> TCC button, specify the configuration file to load, and click OK .

Step 5 Reset the card and wait for it to come up.

Step 6 After the card comes up, enter one of the following commands to telnet to it, depending on the node mode:

• If the node is configured in SSH mode, enter:

• If the node is not configured in SSH mode, enter:

Step 7 Enter the username and password to log in to the ML card. The default values are:

• Username: CISCO15
• Password: CTM123+

Step 8 After logging in, enter the following command:

Topology Goes into Incomplete State

A topology goes into Incomplete state when one or more Layer 1 circuits related to the topology are missing.

Step 1 Launch the Circuit Table and check whether the circuits related to the topology are discovered.

Step 2 If one or more Layer 1 circuits are missing, use the Create Circuit wizard to recreate the circuits. Once the Layer 1 circuits are created and Prime Optical discovers them, the topology goes into Complete state.

Step 3 Check the Circuit Table to verify whether the newly created Layer 1 circuits are discovered.

Step 4 If the topology still fails to go into Complete state, the topology discovery logic has failed. Complete the following substeps to restart the discovery process:

a. Mark the NE service related to the NE as Out of Service and then In Service.

b. Mark the NEs related to the topology as Out of Service and then In Service.

Step 5 If the problem persists, contact the Cisco TAC.

Topology Goes into L2ServiceNotReady State

A topology goes into L2ServiceNotReady state when there is a mismatch between data in the Prime Optical database and data on the ML-series cards.

Step 1 Launch the Circuit Table and check whether all circuits related to the topology are discovered.

Step 2 Launch the Equipment Inventory Table and check whether the NE inventory is collected correctly.

Step 3 Complete one of the following options:

• Launch the L2 Topology Table and enable the L2 service. The Prime Optical database and ML-series cards synchronize. During the synchronization process, the topology goes into In Progress state, and then into Complete state.
• Do a forced resynchronization of the database by launching the NE Explorer for the NEs in the topology and refreshing the data from the NEs. This restarts the synchronization process. Once the NEs are up, the topology goes into Complete state.

Step 4 If the topology still fails to go into Complete state, try the following options to restart the topology discovery process:

• Telnet to the cards and check whether the configuration parameters are corrupt.
• Mark the NE service related to the NE as Out of Service and then In Service.
• Mark the NEs related to the topology as Out of Service and then In Service.
• Delete the topology, reload the barebone config file on the ML-series cards, and reset the cards. To make sure that the barebone config file is reloaded, telnet to the cards and enter the write memory command. Then, recreate the topology.

Step 5 Try one of the following options, depending on when the topology goes into L2ServiceNotReady state:

• If the topology goes into L2ServiceNotReady state immediately after topology creation, choose Administration > Error Log and search for the BaseCardConfigWork file. Check the message associated with that file. If the message says “Base Card Config Work Failed,” open the IosTransportModule.log server file and search from the bottom for the “INVALID” parameter. The command that failed is displayed with an INVALID error. Then, launch the CLI and check whether the card is configured correctly.
• If the topology goes into Complete state but then changes to L2ServiceNotReady after several minutes, choose Administration > Error Log and search for the BaseCardDetectWork file. If the message associated with that file says “BaseCardDetectWork failed,” check the DataService.log file (or the NE Service log for CTM R7.x or later) in the server and search from the bottom for the “BaseCardDetectWork” parameter. Several lines from the bottom, the file shows the exact reason from the base card detection failure.

Step 6 If the problem persists, contact the Cisco TAC with the relevant logs, as well as a configuration snapshot of the cards used in the topology.

Topology Goes into SyncFailed State

A topology can go into SyncFailed state for any of the following reasons:

• A preprovisioned card was used to create the topology. There are two ways to determine whether a card is preprovisioned:

– Launch the NE Explorer and check whether the card is preprovisioned.

– Telnet to the card to verify that it is present.

• The barebone configuration file is not present on all of the cards.
• You have not specified the correct Prime Optical-to-ML-series card login details. To verify the login details, launch the Prime Optical Control Panel and click Security Properties . Click the appropriate NE tab and verify the username and password for Prime Optical server connections to ML-series cards.
• The database and the ML-series card are not synchronized.

Step 1 Launch the Circuit Table and check whether the circuits related to the topology are discovered.

Step 2 Launch the Equipment Inventory Table and check whether the NE inventory is collected correctly.

Step 3 Wait for the next synchronization cycle. The database must synchronize and the topology must go into In Progress state. If the topology goes into L2ServiceNotReady state, see Topology Goes into L2ServiceNotReady State.

Step 4 If the topology still fails to go into Complete state, try the following options:

• Telnet to the cards and check whether the configuration parameters are corrupt.
• Mark the NE service related to the NE as Out of Service and then In Service.
• Mark the NEs related to the topology as Out of Service and then In Service.
• Delete the topology, reload the barebone config file on the ML-series cards, and reset the cards. To make sure that the barebone config file is reloaded, telnet to the cards and enter the write memory command. Then, recreate the topology.

Step 5 If the problem persists, contact the Cisco TAC with the relevant logs, as well as a configuration snapshot of the cards used in the topology. Alternatively, telnet to the NEs and enter the flmDeleteDb command.

Topology Goes into Partially Complete State

A topology goes into Partially Complete state when all of the cards are not synchronized with the Prime Optical database.

This problem also occurs when the data service processes a CARD_CONFIG_DATA_FAILED event for any card in the topology. The topology remains in Partially Complete state until a CARD_CONFIG_DATA_FAILED event is received for the failed card. While the topology is in this state, base card configuration provisioning and detection are performed on other cards in the topology.

Step 1 Launch the Circuit table and check whether all circuits related to the topology are discovered.

Step 2 Launch the Equipment Inventory table and check whether the NE inventory is collected correctly.

Step 3 Wait for the next synchronization cycle. The database must synchronize and the topology must go into Complete state. If the topology goes into L2ServiceNotReady state, see Topology Goes into L2ServiceNotReady State.

Step 4 If the topology still fails to go into Complete state, try the following options:

• Telnet to the cards and check whether the configuration parameters are corrupt.
• Mark the NE service related to the NE as Out of Service and then In Service.
• Mark the NEs related to the topology as Out of Service and then In Service.
• Launch the NE Explorer for the related NE, click the Refresh button, and refresh data from the NE.
• Delete the topology, reload the barebone config file on the cards, and reset the cards. To make sure that the barebone config file is reloaded, telnet to the cards and enter the write memory command. Then, recreate the topology.
• Check whether the IOS Users table contains all of the entries for all cards included in a ring.

Step 5 If the problem persists, contact the Cisco TAC with the relevant logs, as well as a configuration snapshot of the cards used in the topology. Alternatively, telnet to the NEs and enter the flmDeleteDb command.

Prime Optical Cannot Discover the Topology

Step 1 Try the following options to restart the topology discovery process:

• Launch the Circuit Table and check whether all circuits related to the topology are discovered.
• Mark the NE service related to the NE as Out of Service and then In Service.
• Mark the NEs related to the topology as Out of Service and then In Service.

Step 2 If the topology is still not discovered, complete the following substeps to enable the trace level in the data service logs:

a. In the Domain Explorer window, choose Administration > Control Panel .

b. In the Control Panel window, click Logging Properties to open the Logging Properties pane.

c. Click the General tab.

d. In the Preferences area, choose Trace from the Logging Level drop-down list.

e. Click Save .

Step 3 On the Prime Optical server, enter the following command to telnet to the local host. The port number depends on the port to which the NE belongs:

This starts the data log collection on the server. Check whether the data service logs are populating.

Step 4 Check the data service logs to determine whether or not the circuit addition events are being received correctly. You can do this by searching for the string CIRCUIT_ADDED. If the circuit addition events are not being received correctly, there is a problem with Layer 1 circuit provisioning. If the events are received correctly but the topology is not discovered, the circuit deletion logic failed and you must contact the Cisco TAC with the relevant logs.

Prime Optical Cannot Delete the Topology

Step 1 Launch the L2 Topology Table and check whether the topology is in Deleting state. If the topology is not in Deleting state, there is a problem with the deletion logic in Prime Optical.

Step 2 Check the state of the related Layer 1 circuits. They must also be in Deleting state. If they are in Deleting state but are not eventually deleted, there is a problem related to Layer 1 circuit provisioning.

Step 3 If not all of the circuits are deleted and they are not in Deleting state, use CTC to delete the circuits.

Step 4 After the circuits are deleted, if the topology is still not deleted, complete the following substeps to enable the data service trace logs and check whether the circuit deletion events are received correctly:

a. In the Domain Explorer window, choose Administration > Control Panel .

b. In the Control Panel window, click Logging Properties to open the Logging Properties pane.

c. Click the General tab.

d. In the Preferences area, choose Trace from the Logging Level drop-down list.

e. Click Save .

Step 5 On the Prime Optical server, telnet to the local host. This starts the data log collection on the server. Check whether the circuit deletion events are being received correctly. If they are not being received, there is a problem with the Layer 1 circuit provisioning. If they are being received, there is a problem with the topology deletion logic in Prime Optical. Delete the topology from the L2 Topology Table .

Step 6 Open the ML Cards table for the topology. Check the Additional Info column for the reason for the failure.

Step 7 If the problem persists, contact the Cisco TAC with the relevant logs.

How Do I Check Whether All Cards in the Topology Are Synchronized?

Use any of the following options to check whether all cards participating in a topology are synchronized:

• In the Domain Explorer , choose Administration > CTC-Based NEs > IOS Users Table and check whether an entry for the user exists on the cards that will be used. If an entry exists, the card is synchronized with Prime Optical.
• Check the ONS1545_NE_Inventory_table for all of the NEs participating in the ring. The entry for the ML card must be present. This is the same as checking the Equipment Inventory Table from the Prime Optical client and verifying whether all cards are In Service.
• Verify whether you can launch a CLI session on all of the cards:

1. Open the NE Explorer for the NE.

2. Select the card in the tree view.

3. Click the Configuration tab.

4. Click the Launch CLI button.

5. In the Launch CLI dialog box, enter the username and password.

If the CLI session does not launch, the card does not have the barebone configuration or there is a communication failure and Prime Optical cannot read the configuration.

• Verify whether all of the cards have synchronized correctly in the eqpt_info_table. Perform the following database query:

If no rows are found, the card is not synchronized correctly. The card does not have the barebone configuration or there is a communication failure and Prime Optical cannot read the configuration.

• CTM R8.0 and later supports 802.17 RPR, which requires cards to be configured in 802.17 RPR mode. If you include a card in an 802.17 RPR that was used previously as a Cisco RPR, you must reload a fresh barebone configuration, perform a software reset on the card, and then create the topology.

Common VLAN Problems

VLAN Is Not Discovered

This problem occurs when the VLAN discovery process fails. Try the following options to restart the VLAN discovery process:

• Telnet to the cards and check whether the VLAN configuration parameters are corrupt.
• Mark the NE service related to the NE as Out of Service and then In Service.
• Mark the NEs related to the topology as Out of Service and then In Service.
• Launch the NE Explorer for the related NE, click the Refresh button, and refresh data from the NE.

VLAN Is Not Created

If Prime Optical cannot create a VLAN, try the following options:

• If the Prime Optical server is resynchronizing, an error message indicates a VLAN creation failure. Wait for some time before attempting to create the VLAN again.
• If you receive an error message that says you cannot create a bridge group, you might have entered invalid AV pair commands in the CLI. Check the IosTransportModule.log for invalid commands. Enable the Trace error level and check the server logs for exceptions.
• When creating an IP SLA, verify that the associated topology and managed VLAN are synchronized. IP SLA is supported only on NE releases 6.0, 7.0, 7.2, and 8.0. Verify that you are using a supported NE version.

If cards that belong to the topology are not listed in the Manage VLANs dialog box, complete the following steps:

Step 1 Log in to the server and enter the following command:

Step 2 Run the prune_invalid_l2topo.sh script.

Step 3 Restart the NE service.

VLAN Is Not Deleted

If Prime Optical cannot delete a VLAN, try the following options:

• If the Prime Optical server is resynchronizing, an error message indicates a VLAN deletion failure. Wait for some time before attempting to delete the VLAN again.
• Check the IosTransportModule.log for exceptions or invalid commands. Enable the trace error level and check the server logs for exceptions.

PM Data Collection Fails

If PM data collection fails, use a browser to collect PM data directly from the NE. Enter one of the following URLs:

• For real-time PM data:

– For 15-minute collection, use http:// NE-IP-address> /pm/15min/1

– For 1-day collection, use http:// NE-IP-address> /pm/day/1

• For historical PM data:

– For 15-minute collection, use http:// NE-IP-address /pm/15min/ bucket#

– For 1-day collection, use http:// NE-IP-address /pm/day/ bucket#

where bucket# = (current time - missed historical time) / 900000 + 1

Common L1 Circuit Provisioning Problems

The following levels of information are used to debug circuit problems:

• Prime Optical client level

The debug level in the Debug Options window is set to Info by default. Set the debug level to Debug so you can save all the log information into a file that can be used to debug problems while launching circuit-related windows or wizards; for example, Circuit Creation wizard, BLSR Creation wizard, Modify Circuit dialog box, Circuit Trace window, and J1 path trace.

• Prime Optical client level with database interaction

When you perform one of the following operations, the Prime Optical client builds an SQL query and uses the Java Data AcQuisition (JDAQ) system to extract data from the database:

– Find circuit—After performing this operation, the resulting circuits do not meet the criteria or the circuits that meet the criteria are not displayed.

– Launching circuit, link, BLSR, and rolls tables—There are issues with the data displayed in the Circuit Table , Circuit Span Table, Link Table , VLAN Table , Link Utilization Table , Rolls Table , and BLSR Table .

Enable the JDAQ data and JDAQ modules in the Debug Options window to resolve issues resulting from performing the above operations; then, verify that all the following are correct:

– The SQL query that was constructed.

– The parameter values.

– The data set that results from the SQL query.

Logs of the actual data displayed in the tables can be collected by exporting the table data to a text file. Having this data in the form of a text file can be useful when debugging circuit issues.

• Prime Optical client level with Prime Optical server interaction

Almost all the circuit provisioning features involve interaction with the Prime Optical server. To debug the server-side code, collect the NE service logs from the Prime Optical server. The log files are located at /opt/CiscoTransportManagerServer/log. Identify the NE service for which the issue occurred and set the debug level of the NE Service logs in the Control Panel . If you encounter any of the following, collect the NE service logs from the Prime Optical server:

– Issues when launching the Circuit Trace window; or, the circuit trace does not display the correct information.

– Issues related to bridge and roll operations.

– Issues when creating, modifying, or deleting BLSRs.

– Issues when launching the J1 path trace; or, the J1 path trace does not display the correct information.

– Issues when creating, modifying, or deleting circuits.

• Prime Optical level

Ports are available for debugging. Port number 9410 and higher are allocated for debugging Prime Optical-level circuit information. Each NE service or network service has its own port number starting from port 9410. Unfortunately, there is no way to identify which port number is used by which NE service or network service. So while debugging, start with port 9410 and all managed NEs. If the listed NE is the one you need, continue debugging. Otherwise, try port 9411 and so on.

The information that is displayed in the port is cached at the Prime Optical server and is useful when resolving issues such as the following:

– Circuits in Prime Optical and CTC are not synchronized.

– Deleted circuits are removed from CTC but not from Prime Optical.

The following commands can be used for debugging:

– Bye, Exit, Quit—Closes the socket.

– Help—Prints messages.

– History [#]—Prints the command history.

– !<text>—Executes an item in your command history that matches <text>.

– !!—Executes the last item in your command history.

– AllmanagedNEs—Prints the details of all managed NEs.

– Allcircuits—Prints the details for all circuits.

– Logon filename —Logs all the information into a file.

– Logoff—Stops logging all the information into a file.

– Debugon—Enables the debug flag.

– Debugoff—Disables the debug flag.

• NCP level

Complete the following steps to determine the port number and debug circuit issues in the NCP level:

Step 1 Check the NE service log at /tmp/ Service name .log and search for “Debug Telnetd: port is port number .”

Step 2 Note the port number.

Note Each NE service has a different port number.

Step 3 Telnet to the localhost using the debug Telnet port and log in using CISCO15 as the username for CTM R7.2 and earlier and otbu+1 for CTM R8.0 and later.

Note Only one user can use the debug port at a time. If the Telnet command fails and the port closes immediately, check for other open sessions. If there is an open sessions that is not used, close the session immediately.

The debug port contains a lot of information regarding the various cached objects in the NCP. Depending on the faulty feature, you have to decide what logs are required. Use the List command to list all the objects available on a port. When debugging issues related to circuits, the following objects are useful:

• CircuitEnd
• Circuits
• NetCcatCircuit
• NetCircuit
• NetCircuitEnd
• NetCircuitManager
• NetCircuitNode
• NetCircuitSpan
• NetCircuitSplicer
• NetCircuitWatchDog
• Route and network objects—Used for debugging routing or link issues.
• Topology objects—Used for debugging link-related issues.
• BlsrRingManager—Used for debugging BLSR-related issues.
• Bnrpackage, rollmanager, and roll objects—Used for debugging bridge and roll-related issues.

Cannot Route Circuits

Step 1 Make sure that a bandwidth is available for the protected circuit.

Step 2 Enter the following commands to retrieve the dumps before and after the problem occurs:

Step 3 In the Network Control Protocol (NCP) debug window, enter the following commands:

Step 4 Create the circuit and dump all the circuit information into a file.

Step 5 Try manual routing on the circuit that you created and save all information in a file.

Circuit Creation Is Successful in CTC but Fails in Prime Optical

Step 1 Enable the debug logs using the remote debugger.

Step 2 In the Prime Optical Control Panel , set the debug level to Debug.

Step 3 Perform the steps listed in Cannot Route Circuits and collect the NE service logs.

Circuit Table Does Not Display the Correct Information

Step 1 Select the following debug options before opening the Circuit Table :

• Circuit Management
• JDAQ
• JDAQ Data

Step 2 Dump all the log information into a file.

Step 3 Open the Circuit Table .

Step 4 Export all data from the Circuit Table to a file.

Step 5 Identify the remote debug ports of the NEs on which the problem occurs.

Step 6 Log in to the debug port and collect the dumps of all managed circuits.

Circuits Show Up in Incomplete State, Circuits Are Not Discovered, and Circuit Information in Prime Optical Is Not Consistent with Circuit Information in CTC

Perform the following steps when you create a circuit on a set of NEs and the circuit shows as Incomplete in Prime Optical and in CTC, which is launched from Prime Optical. The circuit shows as Active if CTC is launched from a browser.

Step 1 In the NCP debug port window, enter the following commands:

Step 2 Telnet to the localhost using the debug Telnet port and log in using CISCO15 as the username for CTM R7.2 and earlier and otbu+1 for CTM R8.0 and later.

Step 3 Enter the following commands to create a debug log:

Step 4 Create a circuit in Prime Optical and enter the following command to check the circuit name:

Step 5 Enter the following command if you created an STS circuit:

Step 6 Enter the following command if you created a VT circuit:

Note If an NE is moved from one network partition to another, the NCP debug port number changes from one NE service to another.

Missing Links

The following commands are useful when resolving a missing link problem:

Note Enter all commands in the NCP debug port window.

• TopologyX dump—Displays all the nodes and links in the topology. It also displays the current topology host. The X variable represents the topology identifier. Do not include the X variable if you want to display all topologies.
• Network dump—Displays all the nodes and links in the network. It also displays the topology to which the nodes and links belong.
• Network dumpwaitQ—Displays the nodes that have been added to the network but do not have valid node ID.
• NettopologyX dump—Displays all the links in the network topology along with the SPT. The SPT determines the nodes and links that a circuit can be routed to or routed through. The X variable represents the network topology identifier. Do not include the X variable if you want to display all the network topologies.
• Netelement_ xxx-xxx-xxx-xxx dump—Displays the current state of the NE. The xxx-xxx-xxx-xxx variable represents the IP address in reverse order. Do not include the xxx-xxx-xxx-xxx variable to displays all NEs.
• Nldump—Displays the current state of the each net link.
• TopologyX dumpareas—(For multiple OSPF areas only.) Lists all the OSPF areas in the topology and indicates whether the OSPF area is actively monitored by a node.
• TopologyX dumpagents—(For multiple OSPF areas only.) Lists all the topology agents or topology hosts and the OSPF areas that they are responsible for.
• TopologyX dumpareatotopoagentmap—(For multiple OSPF areas only.) Lists all the topology agents or topology hosts that are responsible for maintaining the topology in an OSPF area.

Problems with Manual Links

Sometimes when manual link issues occur in Prime Optical, you have to create the same manual link in CTC before resolving the issue in Prime Optical. These links are called phantom links. Complete the following steps to create a phantom link:

Step 1 Create the phantom link in Prime Optical.

Step 2 Access the CTC Telnet port and run the network dumpnls command to dump all the links with termination points listed in hexadecimal form.

Step 3 Convert the hexidecimal numbers to decimal.

Step 4 In CTC, launch the debug window by holding down the Ctrl key and then the Shift key and press the About key.

Step 5 Enter the following command to create the phantom link between ipaddress1 and ipaddress2, and between the termination points in decimal values:

Debugging Server Trail Links

When debugging server trail link issues, use the same procedure as debugging any other links. Only the size and provision type are different.

If you telnet to an NCP debug port, you can extract server trail link information by using the network dump command.

Debugging Bridge and Roll-Related Problems

When bridge and roll-related problems occur, Telnet to the NCP debug port and enter the rolls dump or rollmanager set command.

If there is a mismatch between the bridge and roll information in Prime Optical and CTC, complete the following steps:

Step 1 On the Prime Optical server, telnet to the local host.

Step 2 Log in using CISCO15 or otbu+1 as the username.

Step 3 Enter the following commands:

Step 4 Create a new roll in Prime Optical.

Step 5 On the Prime Optical server, enter the following command to check for the new roll:

Step 6 Enter the following command to check for the new roll on a circuit:

Step 7 Enter the following command if you created an STS circuit roll:

Step 8 Enter the following command if you created a VT circuit roll:

Step 9 Log off.

Note If NEs are moved from one network partition to another, the NCP debug port number changes from one NE service to another.

If there are any problems while creating a roll or performing any bridge and roll operations, complete the following steps:

Step 1 On the Prime Optical server, telnet to the local host.

Step 2 Log in using CISCO15 or otbu+1 as the username.

Step 3 Enter the following commands:

Common Equipment Provisioning Problems

You can perform equipment provisioning at the following levels:

• Node Level
• Card Level

Node Level

Cannot Create DCC Using OSI or OSI and IP (PPP)

If you cannot create a DCC after choosing to use the OSI or OSI and IP (PPP) Layer 3/Layer 2 configuration, go to the NE Explorer > OSI tab > Routers-Setup tab and enable the router on which you want to create the DCC.

Cannot Modify DCC

If you cannot modify a DCC, check whether the same DCC is created in the SDCC and LDCC.

Card Level

The following are guidelines when configuring pluggable port modules (PPMs), loopback, and alarm profiles on the card level.

PPM Tab

You must provision PPMs before creating pluggable port rates. The following are guidelines when provisioning PPMs on specific cards:

• MRC-12 cards

– You can provision up to twelve PPMs.

– If the MRC-12 card is provisioned on slots 1 to 4 or slots 14 to 17, PPM1 is provisioned with only the OC48 rate.

– If the MRC-12 card is provisioned on slots 5, 6, 12, or 13, you can provision PPM1, PPM4, PPM7, and PPM10 with the OC48 rate.

– The total available bandwidth of the ports is OC48 when the card is provisioned on slots 1 to 4 or slots 14 to 17.

– The total available bandwidth of the ports is OC192 when the card is provisioned on slots 5, 6, 12, or 13.

– The total available bandwidth depends also on the XC card.

• MRC-2.5G-12 cards

– You can provision up to twelve PPMs.

– Only PPM1 allows provisioning of the STM16 rate. If PPM1 is provisioned with STM16, the other PPMs are unprovisionable.

– The total bandwidth of the ports is STM16.

– PPM1, PPM4, PPM7, and PPM10 only allow provisioning of STM4.

• MRC-2.5G-4 cards

– You can provision up to four PPMs.

– Only PPM1 allows provisioning of the OC48 rate. If PPM1 is provisioned with OC48, the other PPMs are unprovisionable.

– The total bandwidth of the ports is OC48.

• STM64 XFP-based cards

– You can only provision one PPM.

– The only allowed port rate is STM64.

Loopback Tab

In the Loopback tab of the NE Explorer , you cannot change the admin state from IS to OOS, DSBLD. You must first change the admin state to OOS, MT or IS, AINS and then OOS, DSBLD.

Alarm Behavior Tab

You can only create alarm profiles at the node level. The Alarm Behavior tab in the card level of the NE Explorer only allows you to apply existing alarm profiles to the card.

How Do I Collect Linux Client Thread Dumps?

Step 1 Open a terminal session.

Step 2 Enter the following command to obtain the client Java process ID:

Step 3 Enter the following command, where pid is the client Java process ID:

How Do I Collect Windows Client Thread Dumps?

To collect thread dumps for CTM R5.0.2 and earlier, complete the following steps:

Step 1 Open a DOS window.

Step 2 Run the ctmc.bat file located in the C:\Cisco\TransportManagerClient directory.

Step 3 Generate the thread dump by holding down the Ctrl key and pressing the Pause ( Break ) key.

Step 4 Copy and paste the output to a text file.

To collect thread dumps for CTM R6.0 and later, complete the following steps:

Step 1 Open a DOS window.

Step 2 Depending on your network size, enter one of the following commands:

Step 3 Generate the thread dump by holding down the Ctrl key and pressing the Pause ( Break ) key.

Step 4 Copy and paste the output to a text file.

How Do I Collect Server Thread Dumps?

Thread dumps are helpful references when debugging the Prime Optical process. To collect thread dumps:

Step 1 Log in to the server workstation as the root user.

Step 2 On the command line, enter the following command:

where:

• Group is the group name for which to collect thread dumps. It can be:

– SM

– SNMPTRAP

– NE

– PM

– GW

• Service is the service name for which the thread dump is required. It can be:

– SMService

– SNMPTrapService

– CORBAGWService

– ONS15216NEService

– ONS15305NEService

– ONS15310NEService

– ONS15327NEService

– ONS15454NEService

– ONS15454SDHNEService

– ONS15600NEService

– ONS15600SDHNEService

– UnmanagedNEService

– ONS15216PMService

– ONS15305PMService

– ONS15310PMService

– ONS15327PMService

– ONS15454PMService

– ONS15454SDHPMService

– ONS15600PMService

– ONS15600SDHPMService

How Do I Enable or Disable the Automatic Refresh Data Feature?

The automatic Refresh Data feature automatically refreshes all data being displayed by Prime Optical. When automatic refresh is enabled, you receive the following prompt:

In an unstable environment where NEs are synchronizing or changing operational states frequently, you might receive the preceding prompt continuously. To disable the prompt:

Step 1 In the Domain Explorer window, choose Edit > User Preferences . The User Preferences dialog box opens.

Step 2 Click the Miscellaneous tab.

Step 3 To disable the prompt, uncheck the Enable Refresh Data Timer check box. (To enable the prompt, leave the check box checked.)

Step 4 Click OK .

How Do I Replace the Alarm Interface Panel?

The Alarm Interface Panel (AIP) provides surge protection for CTC-based NEs. This panel has a nonvolatile memory chip that stores the unique node address known as the MAC address. The MAC address identifies the nodes that support circuits. It allows Prime Optical to determine circuit sources, destinations, and spans.

If an AIP fails, an alarm is generated and the LCD display on the fan-try assemblies of the NEs becomes blank. To perform an in-service replacement of the AIP, you must contact the Cisco TAC.

You can replace the AIP on an in-service system without affecting traffic by using the circuit repair feature. See Repairing Circuits.

Prime Optical GateWay/CORBA Is Installed After Installing the Prime Optical Server

If the Prime Optical server is installed without the Prime Optical GateWay/CORBA option, and the Prime Optical GateWay/CORBA option is installed later, the Control Panel does not refresh automatically after the Prime Optical GateWay/CORBA installation to show that it has been installed.

This problem occurs because the Prime Optical GateWay/CORBA installation is handled by a script that notifies the database that the component has been installed, but does not notify the Prime Optical server.

To work around this problem, click the Refresh Data button in the Control Panel . It might take some time for the Control Panel to show that Prime Optical GateWay/CORBA is installed. You might need to close and reopen the Control Panel to see the change.

Testing Prime Optical GateWay/CORBA

Whenever you encounter Prime Optical GateWay/CORBA problems, complete the following to test Prime Optical GateWay/CORBA:

Step 1 In the Control Panel , make sure that the status of Prime Optical Gate/Way CORBA is Active and that the service action is set to Stop. See Starting or Stopping Prime Optical GateWay/CORBA and Viewing the Prime Optical GateWay/CORBA Service Pane for more information.

Step 2 Check that your profile has been created in the Prime Optical Gateway/CORBA Users table. Use that profile and the correct password to connect the OSS client to the Prime Optical server. See Viewing the Prime Optical GateWay/CORBA Users Table and Adding a Prime Optical GateWay/CORBA User for more information.

Step 3 If you are still encountering problems after performing steps 1 and 2, log in to the server and collect the following logs:

• /tmp/CORBAGWService.log
• /tmo/0.log
• /tmp/jcorbagw/log.bak
• /opt/CiscoTransportManagerServer/log/ CORBAGWService-*.log
• /opt/CiscoTransportManagerServer/openfusion/domains/OpenFusion/localhost/NotificationService/log/NotificationService. log

Step 4 Complete the following steps to change the Prime Optical GateWay/CORBA notification service port to dynamic using the Prime Optical client:

a. In the Domain Explorer window, choose Administration > Control Panel .

b. Click GateWay/CORBA Service to open the GateWay/CORBA Service pane.

c. In the Global tab > Status area, click the Stop button to stop the service.

d. In the GateWay/CORBA Configuration area, set the Notification Service Listening Port Number to zero.

e. Save the changes.