Table Of Contents
Diagnosing Problems with the CiscoWorks2000 Server
Verifying Server Status
Testing Device Connectivity
Troubleshooting the CiscoWorks2000 Server
Frequently Asked Questions
Troubleshooting Suggestions
Diagnosing Problems with the CiscoWorks2000 Server
Use these tools and suggestions to diagnose problems with the CiscoWorks2000 Server:
•
Verifying Server Status
•Testing Device Connectivity
•Troubleshooting the CiscoWorks2000 Server
Verifying Server Status
Server tools enable you to gather and analyze information about your CiscoWorks2000 Server (see Table 5-1).
Table 5-1 Server Tools Tasks
Task
|
Purpose
|
Action
|
Collect server information.
|
Provides system information, environment, configuration, logs, and web server information
|
CiscoWorks2000 Server > Diagnostics >
Collect Server Info
or
Enter the following command:
#/opt/CSCOpx/bin/collect.info
where /opt/CSCOpx/ is the directory where you installed CiscoWorks2000.
|
Perform self test.
|
Runs self-tests and generates a report with the results
|
CiscoWorks2000 Server > Diagnostics >
Self Test
|
Check process failures.
|
Displays the failed process and time failure occurred
|
CiscoWorks2000 Server > Diagnostics >
Process Failures
|
Obtain detailed status of ANI Server.
|
Displays detailed system configuration and settings for ANI Server
|
CiscoWorks2000 Server > Diagnostics >
Analyze ANI Server
|
Check process status.
|
Checks whether back-end processes are in an interim state
|
CiscoWorks2000 Server > Administration >
Process Management > Process Status
|
Query EDS events.
|
Searches for CiscoWorks2000 or third-party events that pass through EDS and matches a specific criteria
|
CiscoWorks2000 Server > Diagnostics >
Event Query
|
Verify installed applications and versions.
|
Lists date and version of currently installed CiscoWorks2000 applications and packages and whether any patches have been applied
|
CiscoWorks2000 Server > About CiscoWorks2000 >
Applications and Versions
|
Testing Device Connectivity
The connectivity tools enable you to test device connectivity and reachability and troubleshoot nonresponsive devices. Some connectivity tools require system administrative-level privileges (see Table 5-2).
Table 5-2 Connectivity Tools Tasks
Task
|
Purpose
|
Action
|
Administrative Tasks
|
Check device connectivity.
|
Verifies that the CiscoWorks2000 server can communicate with the device
|
CiscoWorks2000 Server > Diagnostics > Connectivity Tools > Management Station to Device
|
All Users
|
Look up host or device.
|
Provides device or host information via the name server
|
CiscoWorks2000 Server > Diagnostics > Connectivity Tools > NSLookup
|
Check device traceroute.
|
Detects routing errors between the network management station and a target device
|
CiscoWorks2000 Server > Diagnostics > Connectivity Tools > Traceroute
|
Ping a device.
|
Tests device reachability using an ICMP echo message and its reply
|
CiscoWorks2000 Server > Diagnostics > Connectivity Tools > Ping
|
Troubleshooting the CiscoWorks2000 Server
This section provides information on frequently asked questions (FAQs) and suggestions for troubleshooting the CiscoWorks2000 Server components. If the suggestions do not resolve the error, check the release notes supporting your platform for possible workarounds, or contact the Cisco TAC or your customer support.
Frequently Asked Questions
•I installed CD One and got an error message that EDS wasn't registered with the daemon manager. Did I do anything wrong?
•Why do some CiscoWorks2000 applications not appear in the product?
•Why can't I start my CiscoWorks2000 application?
•What kind of directory structure does CiscoWorks2000 use when backing up data?
•What does "cmf" stand for in the database path?
•I'm locked out of the CiscoWorks2000 Server. Why did this happen, and how do I regain access?
•How can I check the status of the ANI Server?
•How can I verify whether the ANI Server is fully functional?
I installed CD One and got an error message that EDS wasn't registered with the daemon manager. Did I do anything wrong?
EDS is part of the CD One deliverable but is not enabled without Campus Manager or Resource Manager Essentials. If you are going to install either of these application suites, EDS will be automatically enabled after installation.
Why do some CiscoWorks2000 applications not appear in the product?
The CiscoWorks2000 Server represents a common set of management services which are shared by multiple network management applications. These services are enabled when a suite is installed and an application that relies on a particular service enables it.
If a particular suite of applications does not use a particular services, they might not appear on the CiscoWorks2000 Server desktop. Applications and application suites may not use these features at all or to the fullest extent to which they are available. Refer to the user guide for your application suite to determine the extent to which these features are used.
Why can't I start my CiscoWorks2000 application?
If you cannot start your CiscoWorks2000 application and receive error messages complaining that the WebServer or ANIServer might not be running (even though pdshow indicates that those processes are up and running), you might need to check how your machine is resolving its server name and IP address.
The CiscoWorks2000 CORBA applications require name resolution to work properly. Choose Domain Name Service (DNS) or any other form of name resolution supported by both your client and server.
Configure the name resolution mechanism and restart the CiscoWorks2000 Server to access the application correctly.
What kind of directory structure does CiscoWorks2000 use when backing up data?
CiscoWorks2000 uses a standard database structure for backing up all suites and applications. See Table 5-3 for sample directory structure for the CiscoWorks2000 Server (represented by the cmf acronym). The cmf directory has two databases: cmf and ani.
Table 5-3 Sample CMF Backup Directory
Directory path
|
Description
|
Usage Notes
|
/tmp/1
|
Number of backups
|
1, 2, 3...
|
/tmp/2/cmf
|
Application or suite
|
Backs up CiscoWorks2000 Server applications.
|
/tmp/1/cmf/filebackup.tar
|
CiscoWorks2000 server application tar files
|
Application data is stored in the datafiles.txt which are compiled into the tar file.
|
/tmp/1/cmf/database
|
CiscoWorks2000 Server database directory
|
Includes files for each database:
xxx_DbVersion.txt
xxx.db database files
xxx.log database log files
xxx.txt database backup manifest file
|
What does "cmf" stand for in the database path?
The cmf acronym in the database structure represents the Common Management Foundation (CMF). This phrase describes the set of management services provided by the CiscoWorks2000 Server.
I'm locked out of the CiscoWorks2000 Server. Why did this happen, and how do I regain access?
There are several reasons why you might have been locked out. Most likely it is due to changes made using the Select Login Module option (see "Understanding Fall Back Options"in the "Setting Up the CiscoWorks2000 Server" chapter). You will need to replace the incorrect login module with a default configuration, log into CiscoWorks2000, and return to the login module to correct one or more of the following:
•Authentication service that doesn't exist
•Authentication service that isn't reachable
•Incorrect host name
•Incorrect port number
•Incorrect configuration parameters
Note Do not alter the existing technologies in the default configuration file.
If all of the parameters listed above are correct, see the "Troubleshooting Suggestions" section.
How can I check the status of the ANI Server?
Check the status of the ANI Server to determine if it is up-and-running properly:
Step 1 From the CiscoWorks2000 desktop, select CiscoWorks2000 Server > Administration > Process Management > Process Status.
Step 2 Use the information in Table 5-4 to verify the current status of the ANI Server.
Table 5-4 ANI Server Status
State
|
Explanation
|
Solution
|
Administrator has shut down the server.
|
Administrator intentionally shut down the ANI Server.
|
Start the ANI Server to get it up-and-running again.
|
Administrator has shut down the server.
|
ANI Server has failed to come back up.
|
Might occur after stopping and restarting the ANI Server.
|
Failed to Run.
|
ANI Server could not start up initially.
|
Check the log for details:
•Solaris—/var/adm/CSCOpx/logs/daemons.log
•Windows NT— %NMSROOT%\
log\ANIServer.log, where %NMSROOT% is the root install directory.
|
Program started—No mgt msgs received.
|
ANI Server is in the process of starting up.
|
Wait a few minutes, and check the status again.
|
Running but busy flag set.
|
Normal state. ANI Server is up-and-running normally.
|
None. Everything is normal.
|
Can also appear when the ANI Server is hung or not responding.
|
If discovery does not appear to be completing, check the log for details:
•Solaris—/var/adm/CSCOpx/logs/daemons.log
•Windows NT—%NMSROOT%\log\
ANIServer.log, where %NMSROOT% is the root install directory.
|
The ANI Server cannot load the database. Run reinitdb.pl to reinitialize the database.
|
The ANI database is corrupted, and the ANI Server cannot run with a corrupted database.
|
Reinitialize the database:
From the command prompt or shell window, where $NMSROOT is the root directory where you installed CiscoWorks2000, run:
•Solaris—$NMSRROOT/bin/reinitdb.pl
•Windows NT—perl %NMSROOT%\bin\
reinitdb.pl
|
How can I verify whether the ANI Server is fully functional?
The CiscoWorks2000 Server includes several tools to verify varying functional states of the ANI Server. To progressively determine the cause of any problems:
Step 1 Select Ciscoworks2000 Server > About CiscoWorks2000 > About ANI Server > Copyrights and Versions.
•If this page appears, you have verified that the web server and its applets are working properly.
•If this page does not appear, the problem is not related to the ANI Server. The CiscoWorks2000 web server is not running properly.
Step 2 Select Ciscoworks2000 Server > About CiscoWorks2000 > About ANI Server > Devices Support.
•If a list of icons, device names, and device OIDs appears, you have verified that the ANI Server is up-and-running.
•If an error occurs, follow the suggestions appropriate to the reported state of the ANI Server (see Table 5-4).
Step 3 Select Ciscoworks2000 Server > Diagnostics > Analyze ANI Server.
•If this page appears, verify that the name of the ANI Server and the host running it is correct.
•If an error occurs, follow the suggestions appropriate to the reported state of the ANI Server (see Table 5-4).
Troubleshooting Suggestions
Use the suggestions in Table 5-5 to resolve errors or other problems with the CiscoWorks2000 Server.
Table 5-5 Troubleshooting Suggestions
Symptom
|
Probable Cause
|
Possible Solutions
|
Desktop
|
"Authorization required. Please log in with your username and password."
|
Incompatible browser causing cookie failure (unable to retrieve cookie).
|
Refer to the installation documentation for supported Internet Explorer/Navigator software.
|
Database: ODBC error with Essentials. (Windows NT only.)
|
ODBC resource.dll and ODBC driver manager are different versions.
|
Install ODBC from Windows NT CD (selecting SQL server).
|
Display: applet cannot start: class browserServer not found. (Solaris only).
|
The server name is not in the httpd.conf file.
|
Add the server name in the httpd.conf file in the $NMSROOT\lib\web\conf directory.
|
Display: only right side of desktop displayed.
|
Incompatible browser software.
|
Refer to the installation documentation for supported Internet Explorer/Navigator software.
|
Desktop is not registered in DNS.
|
Register desktop in DNS.
|
Setup
|
"Authorization required. Please log in with your username and password."
|
Incompatible browser causing cookie failure (unable to retrieve cookie).
|
Verify that you have Accept all cookies enabled. Refer to the installation documentation for supported Internet Explorer and Netscape Navigator software and setup procedures.
|
"Daemon Manager could not start. The port is in use."
|
The operating system has not yet reallocated the port.
|
Make sure all CiscoWorks2000 processes are terminated (/usr/ucb/ps -auxww | grep CSCO). Wait five to ten minutes, then try to restart the Daemon Manager.
|
User has forgotten his password.
|
CiscoWorks2000 cannot recover forgotten passwords.
|
A system administrator-level user must either change the password or delete and then add the user again.
|
"The ANI Server cannot load the database. Run reinitdb.pl to reinitialize the database."
|
The ANI database is corrupted, and the ANI Server cannot run with a corrupted database.
|
Reinitialize the database:
From the command prompt or shell window, where $NMSROOT is the root directory where you installed CiscoWorks2000, run:
•Solaris—$NMSRROOT/bin/reinitdb.pl
•Windows NT—perl %NMSROOT%\bin\
reinitdb.pl
|
Cannot access the ANI supported devices.
|
ANI may have restarted since you last connected.
|
Log out of CiscoWorks2000 and log in again:
1. Click Logout.
2. Enter your user name and password.
3. Click Connect and attempt to use the Campus Manager application again.
|
Unable to analyze the ANI Server because a message states that you are not authorized.
|
When using a Campus Manager application, a error occurs: "Cannot connect to ANI Server."
|
An error message states that Client Application Manager cannot be registered or unregistered.
|
The web server may be down, preventing Client Application Manager from communicating with the client system.
|
Verify that the CiscoWorks2000 web server is up and running by selecting CiscoWorks2000 Server > Administration > Process Management > Process Status.
|
Administration
|
You are locked out of the CiscoWorks2000 Server.
|
Changes in the login module configuration file might not be correct.
Authentication server might be down and there were no fallback logins set.
|
On UNIX:
1. Log on or su as bin or root.
2. Remove the changed login module override, /opt/CSCOpx/www/classpath/
com/cisco/nm/cmf/security/jaas/
JaasConfigModule.
3. Restart JRunProxyServer:
/opt/CSCOpx/bin/pdterm JRunProxyServer
/opt/CSCOpx/bin//pdexec JRunProxyServer
4. Reload the browser.
|
| |
|
On Windows NT:
1. Remove the changed login module override, C:\Program Files\CSCOpx\www\classpath\
com\cisco\nm\cmf\security\jaas\
JaasConfigModule.
2. Restart JRunProxyServer by either of the following methods:
–Access the Windows NT Service Control dialog and stop and start JRUN Proxy Server for CW2000
or
–In an MS-DOS window, change to C:\Program Files\CSCOpx\bin (this drive and path are the defaults for installation; use the actual installation drive and path). Run the following:
.\pdterm JRunProxyServer
.\pdexec JRunProxyServer
3. Reload the browser.
|
The Log File Status window displays files that exceed their limit.
|
Files need to be backed up so that file size will be reset to zero.
|
1. Stop all processes.
2. Enter the log file maintenance command:
a. On UNIX—$NMSROOT/cgi-bin/admin/
b. On Windows NT— %NMSROOT%\
cgi-bin\admin\
3. Restart all processes.
For detailed instructions refer to the Maintaining Log Files online help (select CiscoWorks2000 Server > Administration > Log File Status, then click Help).
|
Database: inaccessible.
This can occur if processes are not running.
|
Server is unable to connect to the database, which is corrupt or inaccessible.
|
1. Log in to CiscoWorks2000 as admin.
2. Select CiscoWorks2000 Server > Diagnostics >Process Failures to get a list of CiscoWorks2000 back-end processes that have failed.
3. Select CiscoWorks2000 Server > Diagnostics > Self Test.
Click Create to create a report.
Click Display to display the report.
4. Select CiscoWorks2000 Server > Diagnostics > Collect Server Info.
5. Click the Product Database Status link to get detailed database status.
6. Contact the Cisco TAC or your customer support to get the information you need to access the database and find out details about the problem. After you have the required information, perform the following tasks for detecting and fixing database errors.
|
| |
|
Depending upon the degree of corruption, the database engine may or may not start. For certain corruptions, such as bad indexes, the database can function normally until the corrupt index is accessed.
Database corruptions, such as index corruptions, can be detected by the dbvalid utility, which requires the database engine to be running.
To detect database corruption:
1. Log on as root (UNIX) or with administrator privileges (Windows NT).
2. Stop the Daemon manager if it is already running:
–UNIX—/etc/init.d/dmgtd stop
–Windows NT—net stop crmdmgtd (enter command in an MS-DOS window)
3. Make sure no database (dbeng50) processes are running and there is no database log file. For example, if the database file is /opt/CSCOpx/databases/rme/rme.db, the database log file is /opt/CSCOpx/databases/rme/rme.log. This file is not present if the database process shuts down cleanly.
|
| |
|
4. (UNIX only) Check if the database files(s) and the transaction log file (*.log) are owned by user bin. If not, change the ownership of these files to user bin and group bin.
5. (UNIX only) Set environment variables (K-Shell syntax):
export SATMP=/tmp/.SQLAnywhere
export LD_LIBRARY_PATH=/opt/
CSCOpx/lib (Solaris only)
export SQLANY=/opt/CSCOpx/objects/db
|
| |
|
6. Start the database engine:
–UNIX—start it in the foreground:
$SQLANY/bin/dbeng50 -c 16M -m -n validateEng {database file name} -n validateDb
–Windows NT—start it in an MS-DOS window. For example, if CiscoWorks2000 is installed in d:\cw2000:
d:\cw2000\objects\db\win32\dbeng50 -c 16M -n testEng {database file name} -n testDb
If the database engine starts, it displays messages like this (in Windows NT, message appears in a new window):
16384K of memory used for caching
|
| |
|
If this message does not appear, the database has a fatal corruption, but can sometimes be recovered.
7. (UNIX only) If there was no error in step 5, Restart the engine as a daemon:
$SQLANY/bin/dbeng50 -ud -c 16M -m -n validateEng {database file name} -n validateDb
|
| |
|
8. Run dbvalid to detect any other errors such as corrupt indexes:
–UNIX syntax—$SQLANY/bin/dbvalid -c "uid={dba user id for the database};
pwd={dba password for the database};
eng=validateEng;dbn=validateDb"
–NT syntax: —d:\cw2000\objects\db\
win32\dbvalid -c "uid={dba user id for the database};pwd={dba password for the database};eng=validateEng;dbn=validateD"
|
| |
|
The dbvalid command displays a list of tables being validated. The Validation utility scans the entire table, and looks up each record in every index and key defined on the table. If there are errors, the utility displays something like:
run time SQL error -- Foreign key parent_is
has invalid or duplicate index
On UNIX, the command also returns a non-zero return code, that is, $? = 1.
|
| |
|
To fix database errors:
Fatal errors may be due to a corrupt database or a corrupt or missing database transaction log file. If the transaction log is missing or corrupt, the database can be recovered at the cost of a few missing transactions. This can be done by forcibly starting the database without a transaction log; for example:
$SQLANY/bin/dbeng50 -f -n rmeTst /opt/CSCOpx/databases/rme/rme.db
|
| |
|
This starts the database engine in recovery mode, recovers the database, and immediately terminates the engine. Messages similar to the following appear during the recovery process:
2648K of memory used for caching
Database recovery in progress
Last checkpoint on Mon Dec 06 1999 03:06 pm
Transaction log: px.log...
Forcing recovery without transaction log
Recovered to last checkpoint
|
| |
|
If the corruption is due to bad indexes, the indexes must be dropped and then recreated:
1. Start the engine as in step 6 for Windows NT or step 7 for UNIX.
2. Run the isql utility to fix the bad indexes:
–UNIX syntax—$SQLANY/bin/isql -c "uid={dba user id};pwd={dba password};
eng=validateEng;dba=validateDb"
–Windows NT syntax (run in an MS-DOS window). For example, if CiscoWorks2000 is installed in d:\cw2000:
d:\cw2000\objects\db\win32\isql-c "uid={dba user id};pwd={dba password};
eng=validateEng;dbn=validateDb"
|
| |
|
3. Run the following isql commands to delete and add the indexes. This could take a while to run depending upon the number of rows in the table.
–Delete an index with the SQL statement:
DROP INDEX {the index in question}
–Add an index with the SQL statement:
CREATE [UNIQUE] INDEX xyz on {your table pqr} column_name {ASC|DESC}, ...
|
Feedback
