Troubleshooting DCNM-LAN
This chapter describes how to troubleshoot common issues you might experience while using Cisco Data Center Network Manager for LAN (DCNM-LAN).
Note For troubleshooting DCNM-LAN server installation issues, see the Cisco DCNM Installation and Licensing Guide, Release 6.x.
This chapter includes the following sections:
Tips for Using DCNM-LAN
This section includes the following topics:
Events Tabs Show Fewer Events than the Event Browser
The Event Browser feature shows all messages received by DCNM-LAN, even if the message pertains to a feature that is not supported by DCNM-LAN.
An Events tab shows only those messages that reflect the status of the currently selected feature. For some features, this is a subset of the possible messages about the feature.
Event Browser Pie Chart May Be Inaccurate for Small Numbers
The Event Browser pie chart may sometimes show incorrect sizes for wedges that are less than 5 percent of the pie; however, the numbers shown are correct.
Trouble with Starting the DCNM-LAN Server
This section includes the following topics:
DCNM-LAN Server Fails to Start
Check
Table 38-1
for symptoms related to downloading the DCNM-LAN client. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-1 DCNM-LAN Server Fails to Start
|
|
|
DCNM-LAN server fails to start.
|
The Postgres database did not install.
|
For troubleshooting DCNM-LAN server installation issues, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
The Postgres service is not running.
|
Start the Postgres service:
-
In Microsoft Windows Server, choose
Start > All Programs > Postgres 8.2 > Start Service
.
-
In RHEL, use the following command:
$INSTALLDIR/dcm/db/bin/DB start
|
The Postgres user credentials are incorrect.
|
1. Correct the Postgres user credentials. For more information, see the “Updating DCNM-LAN Database Name and Username in pgAdmin III” section.
2. Start the DCNM-LAN server. For more information, see the “Starting DCNM-LAN Servers” section.
|
The ports used by the server are already in use.
|
1. Check the server log for messages such as “Port
port-number
already in use.” The server log is the following file:
Installation_directory
\jboss-4.2.2.GA\server\dcnm\
log\server.log
2. Determine which application is using the port and stop or reconfigure the application.
3. Restart the DCNM-LAN server.
|
Trouble with the DCNM-LAN Database
This section includes the following topics:
Note If the DCNM-LAN database fails or communication to the DCNM-LAN database fails, you must stop the DCNM-LAN server or shut down the cluster of DCNM-LAN servers before addressing the problem. Always verify that the DCNM-LAN database and the communication to the DCNM-LAN database are functioning properly before restarting the DCNM-LAN server or cluster of DCNM-LAN servers.
Trouble with a PostgreSQL Database
Check
Table 38-2
for symptoms related to the pgAdmin III application for administering a postgreSQL database used with DCNM-LAN. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-2 pgAdmin III Errors
|
|
|
Error message states that the DCNM-LAN database does not exist.
|
The DCNM-LAN database name might have changed during an upgrade or reinstallation of the DCNM-LAN server software.
|
In the pgAdmin III application, perform the steps in the “Updating DCNM-LAN Database Name and Username in pgAdmin III” section.
|
Error message states that password authentication failed for the DCNM-LAN database username.
|
The DCNM-LAN database username may have changed during an upgrade or reinstallation of the DCNM-LAN server software.
|
Updating DCNM-LAN Database Name and Username in pgAdmin III
You can update the DCNM-LAN database and username in pgAdmin III.
Step 1 Open the pgAdmin III application.
Step 2 In the Object Browser pane, under Servers, click
PostgreSQL Database Server 8.2
.
In the right-hand pane, the Properties tab appears with several other tabs.
Step 3 On the Properties tab, double-click
Maintenance database
.
A dialog box displays a Properties tab for the server.
Step 4 If you need to change the database name, click the
Maintenance DB
field and type the correct DCNM-LAN database name.
Note The database name should be the name that you specified when you most recently upgraded or reinstalled the DCNM-LAN server software.
Step 5 If you need to change the database username, click the
Username
field and type the correct DCNM-LAN database username.
Note The database username should be the database username that you specified when you most recently upgraded or reinstalled the DCNM-LAN server software.
Step 6 Click
OK
.
Step 7 In the Object Browser pane, double-click
PostgreSQL Database Server 8.2
.
If you changed the username in Step 5, the Connect to Server dialog box appears.
Step 8 If necessary, enter the password for the username that you specified in Step 5 and click
OK
.
The pgAdmin III application connects to the DCNM-LAN database and displays the databases and login roles.
If you need additional assistance, see the Help menu in the pgAdmin III application or see the pgAdmin web site at the following URL:
http://pgadmin.org/docs/1.6/index.html
Trouble with an Oracle Database
If the DCNM-LAN server has trouble using an Oracle database, it logs the error messages in the following file:
Installation_directory
\jboss-4.2.2.GA\server\dcnm\log\server.log
Check
Table 38-3
for symptoms related to using an Oracle database with DCNM-LAN. For each error message, see the possible cause and follow the corresponding solution.
Table 38-3 DCNM-LAN server.log File Errors with an Oracle Database
|
|
|
The following error appears in the server.log file:
java.sql.SQLException: ORA-01653: unable to extend table Cisco DCNMUSER.DCMRAWEVENTTABLE by 1024 in tablespace SYSTEM
|
The tablespace SYSTEM is too small.
|
1. Stop the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
2. Increase the SYSTEM table space. For more information, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
3. Start the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
|
The following error appears in the server.log file:
[org.hibernate.util.JDBCExceptionReporter] Could not create connection; - nested throwable:
(java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
|
The number of available sessions and processes is inadequate.
|
1. Stop the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
2. Increase the number of sessions and processes to 150 each. For more information, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
3. Start the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
|
The following error appears in the server.log file:
2009-04-08 15:53:47,125 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00604: error occurred at recursive SQL level 1
ORA-01000: maximum open cursors exceeded
|
The number of open cursors is inadequate.
|
1. Stop the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
2. Increase the number of open cursors to 1000. For more information, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
3. Start the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
|
Trouble with a PostgreSQL Database
When the PostgreSQL Database causes high CPU utilization on the DCNM server, perform the following:
Step 1 Stop the LAN and SAN server, by using the following command:
$INSTALLDIR/dcm/dcnm/bin/stopLANSANserver.sh
Step 2 Stop the PostgreSQL Database, by using the following command:
$INSTALLDIR/dcm/db/bin/DB stop
Step 3 Start the PostgreSQL Database, by using the following command:
$INSTALLDIR/dcm/db/bin/DB start
Step 4 Start LAN and SAN server, by using the following command:
$INSTALLDIR/dcm/dcnm/bin/startLANSANserver.sh
After the restart, CPU utilization will begin to increase while DCNM runs a device discovery job.
Note The CPU utilization must decrease after the discovery is complete.
Trouble with the DCNM-LAN Client
This section includes the following topics:
Cannot Download the DCNM-LAN Client from the Server
Check
Table 38-4
for symptoms related to downloading the DCNM-LAN client. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-4 Cannot Download the DCNM-LAN Client from the Server
|
|
|
Cannot download the DCNM-LAN client from the server.
|
You are using the wrong URL or web server port.
|
Verify that you are using the correct URL, including the port number.
|
The TCP port is blocked by a gateway device.
|
Open the TCP port in your firewall. For information about ports used by DCNM-LAN, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
You are using an unsupported web browser.
|
Use a supported web browser. For more information about supported web browsers, see the
Cisco DCNM Release Notes, Release 7.x
.
|
Cannot Install the DCNM-LAN Client
Check
Table 38-4
for symptoms related to installing the DCNM-LAN client. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-5 Cannot Install the DCNM-LAN Client
|
|
|
Installer attempts to install Java version JRE 1.7.0_55 but fails.
|
The system does not have Internet access.
|
The DCNM-LAN client installer requires Internet access to download the Java version JRE 1.7.0_55. If the system cannot access the Internet, use another system to download the Java installer, copy it to the system that you want to install the DCNM-LAN client on, install Java, and restart the DCNM-LAN client installation.
You can download Java version JRE 1.7.0_55 from the Oracle Technology Network website.
|
Your network environment requires the use of a proxy connection to access the Internet.
|
If your network environment requires a proxy connection to permit the download of the Java installer, ensure that the proxy settings are configured in Internet Options, available from the Control Panel.
|
Cannot Start the DCNM-LAN Client
Check
Table 38-6
for symptoms related to starting the DCNM-LAN client. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-6 Cannot Start the DCNM-LAN Client
|
|
|
Cannot start the DCNM-LAN client.
|
The client installation may be corrupted.
The wrong version of Java may be installed.
|
1. Uninstall the DCNM-LAN client. For more information, see the “Uninstalling the DCNM-LAN Client” section.
2. Download and install the DCNM-LAN client from the DCNM-LAN server.
During the client installation, allow DCNM-LAN to install the supported version of Java on the computer. When you download the client from the DCNM-LAN server, if the supported version of Java is not detected on the computer, DCNM-LAN asks you for permission to install the supported version of Java.
Your browser may notify you that the Java installer was digitally signed by an expired certificate. To continue, confirm the installation.
For more information, see the “Downloading and Launching the DCNM-LAN Client” section.
|
Cannot Log into the DCNM-LAN Client
Check
Table 38-7
for symptoms related to logging into the DCNM-LAN client. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-7 Cannot Log into the DCNM-LAN Client
|
|
|
Cannot log into the DCNM-LAN client.
|
You forgot your password.
|
Ask a DCNM-LAN administrator to reset your password using one of the following scripts:
-
For Microsoft Windows, use
dcnm_root_directory
/dcm/dcnm/bin/pwreset.bat (by default,
dcnm_root_directory
is c:\Program Files\Cisco Systems\dcm\dcnm\bin).
-
For Linux, use
dcnm_root_directory
/dcm/dcnm/bin/pwreset.sh (by default, the
dcnm_root_directory
is /usr/local/cisco).
To reset a password, run the appropriate script for the operating system that you are using, and then enter the user ID to be reset and the password to be used for it.
If no one has administrative access to DCNM-LAN, you can reset the local administrator account or change DCNM-LAN server authentication settings by reinstalling the DCNM-LAN server software. For more information, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
Authentication servers are not configured to authenticate DCNM-LAN users.
|
If DCNM-LAN is configured to use authentication servers, ensure that every authentication server that you have configured DCNM-LAN to use is configured to accept authentication requests from the DCNM-LAN server. If you have deployed DCNM-LAN in a clustered-server environment, ensure that every authentication server is configured to accept requests from each DCNM-LAN server in the cluster.
|
The DCNM-LAN server is down.
|
Restart the DCNM-LAN server. See the “Starting a Single DCNM-LAN Server” section.
|
The DCNM-LAN server is unreachable.
|
Ensure that the computer that runs the DCNM-LAN client meets the network requirements for using the DCNM-LAN client remotely. Any gateway network devices between the DCNM-LAN client and server must allow connections to the DCNM-LAN web server and to the DCNM-LAN server. By default, the DCNM-LAN web server listens to port 8080 and the DCNM-LAN server listens to port 1099; however, you can configure these ports during DCNM-LAN server installation. If you need to change either port, reinstall the server and choose the Full Reinstall option. For information about ports used by DCNM-LAN, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
The DCNM-LAN server IP address changed after you installed the server.
|
Do the following:
1. Ensure that the IP address of the DCNM-LAN server is statically assigned.
2. Reinstall the DCNM-LAN server and choose the Full Reinstall option, which allows you to specify the server IP address. See the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
3. Log into the DCNM-LAN client and specify the new IP address of the DCNM-LAN server in the DCNM-LAN Server field of the login dialog box.
|
Cannot log into the DCNM-LAN client (continued).
|
The wrong DCNM-LAN server port number was used in the login attempt.
|
In the DCNM-LAN client login window, click
More
and, in the Port field, change the port number that your DCNM-LAN server uses. See the “Restarting the DCNM-LAN Client” section.
If you want to change the port that the DCNM-LAN server listens to, reinstall the DCNM-LAN server and choose the Full Reinstall option, which allows you to specify the DCNM-LAN server port. See the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
When you try to log into the DCNM-LAN client, you receive the error message “Can not resolve DCNM-LAN server
hostname
via DNS. Make sure that DCNM-LAN server has a valid DNS entry.”
|
You used a hostname to specify the DCNM-LAN server during the login and DNS does not have an entry for the DCNM-LAN server.
|
Ensure that DNS on your network has an entry for the DCNM-LAN server hostname.
|
Client Loses Connection to the DCNM-LAN Server
Check
Table 38-8
for symptoms related to the DCNM-LAN client losing its connection with the server. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-8 Client Loses Connection to the DCNM-LAN Server
|
|
|
-
Client loses connection to the server.
-
The DCNM-LAN client window is pink.
|
The client had a failure.
|
Restart the DCNM-LAN client.
|
The DCNM-LAN server is down.
|
Restart the DCNM-LAN server. See Chapter27, “Starting and Stopping Cisco DCNM-LAN Servers”
|
The DCNM-LAN server is unreachable.
|
Investigate your network to determine if it meets the network requirements for using the DCNM-LAN client remotely. For information about ports used by DCNM-LAN, see the
Cisco DCNM Installation and Licensing Guide, Release 6.x
.
|
Trouble with Device Discovery or Device Status
Check
Table 38-9
for symptoms related to issues with device discovery or the device status. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-9 Trouble with Device Discovery or Management
|
|
|
-
A device discovery task fails.
-
A device status changes to Unmanaged or Unreachable.
|
Incorrect device credentials were provided.
|
Reenter the username and password, and try discovering the device again.
If you are attempting to discover CDP neighbors of the seed device, ensure that the credentials that you provide are valid on all devices that you want to discover.
|
The SSH server is disabled on the device.
|
Reenable the SSH server on the device and try discovering the device again.
|
The maximum number of SSH sessions that the device can support has been reached.
|
Check the number of user sessions on the device. Free at least one connection and try discovering the device again.
|
CDP is disabled on the device or on the device interface that the DCNM-LAN server connects to.
|
Ensure that CDP is enabled on the device globally and that it is enabled on the specific interface that the DCNM-LAN server connects to.
|
The device interface that the DCNM-LAN server connects to is shut down.
|
Ensure that the device interface that the DCNM-LAN server connects to is up.
|
The device restarted or shut down before discovery could complete.
|
Ensure that the device is running and try discovering the device again.
|
The DCNM-LAN server cannot reach the device.
|
Ensure that the network requirements for device management are met. See the “Verifying the Discovery Readiness of a Cisco NX-OS Device” section.
|
Discrepancy in system log messages.
|
Use the clear logging logfile command to clear the system log in the device and try to manually discover the device.
|
Discrepancy in accounting log messages.
|
Use the clear accounting log command to clear the accounting log messages in the device and try to manually discover the device.
Note When working with a custom VDC, clear the accounting log messages only from the default VDC.
|
Deep Discovery Soft Limit in DCNM 6.2.3
In DCNM 6.2.3, we have enforced a limit of devices that can be managed in the DCNM LAN client. This limit is only applicable for the deep discovery and can be configured. To configure the property, open Jboss-service.xml, the name of the property is cisco.dcbu.dcm.device.managed. The default value of this property is 50 and you can increase it to a maximum of 75. If the specified value is more than 75, then the value will be taken as 75. If the number of managed devices (Calculated as: number of managed VDC (Nexus 7k ) + number of managed devices( Nexus 5k/3k/1k/cat6k) + number of IPaddresses in the deep discovery + number of IP addresses in the deep discovery request) is more than the soft limit then the deep request will be dishonored.
Scenario 1
If you have 51 switches shallow discovered and you perform deep discovery of all the 51 switches.
You will be prompted with the following message at the beginning of the deep discovery and it fails
“Failed to do deep discovery of task. Number of devices already discovered is <Current Number say 40 or 45). Threshold set for number of manageable devices is 50. Reduce the number of devices selected and retry the discovery”.
Scenario 2
If you have already discovered 49 switches and try to discover 2 more switches.
An error message will be displayed stating that you have exceeded the maximum limit of 50 managed devices in DCNM LAN. You must reduce the number of selected devices to ensure that the total count is equal to or below 50.
Scenario 3
If you have upgraded from Cisco DCNM 5.2/6.1 to Cisco DCNM 6.2.3 and discovered 62 switches using Cisco DCNM 5.2/6.1.
All the devices managed in DCNM 5.2/6.1 will be re-discovered. Thus, if you have 62 devices, you will be able to manage 62 devices after upgrade to DCNM 6.2.3. However, if any of these 62 devices are deleted or re-discovered manually, the limit will be enforced.
You will receive the following notification that is displayed during new installation”
“Failed to do deep discovery of task. Number of devices already discovered is <Current Number say 40 or 45) . Threshold set for number of manageable devices is 50. Reduce the number of devices selected and retry the discovery”.
How to increase the threshold limit
Step 1 Go to <DCNM Root> dcm\jboss-4.2.2.GA\server\dcnm\conf.
<DCNM Root> is the directory where DCNM is installed.
-
In Microsoft Windows, the default path of the root directory is C:\Program Files\Cisco Systems.
-
In Linux, the default path of the root directory is /usr/local/cisco.
Step 2 In the Jboss-service.xml file, look for the following property cisco.dcbu.dcm.jms.port.
Step 3 Add the property cisco.dcbu.dcm.device.managed =75 immediately after the property mentioned in Step 2.
<attribute name="Properties"> cisco.dcbu.dcm.dbCfgResource=database/postgresql.cfg.xml cisco.dcbu.dcm.mappingCfgResource=com/cisco/dcbu/dcm/model/model.cfg.xml cisco.dcbu.dcm.notification.syslog.listenerPort=5445 cisco.dcbu.dcm.data.path=C:\\Program Files\\Cisco Systems\\dcm\\dcnm\\data cisco.dcbu.dcm.notification.syslog.thresholdLimit=1000 cisco.dcbu.dcm.cluster.deltaStartingTimer=120 cisco.dcbu.dcm.cluster.deltaChangingTimer=60 cisco.dcnm.remoting.transport=socket cisco.dcbu.dcm.service.discovery.background=true cisco.dcbu.dcm.discovery.pool.cdp.size=15 cisco.dcnm.remoting.port=3873 cisco.dcnm.remoting.ejbport=3873 cisco.dcnm.remoting.sslejbport=3843 cisco.dcnm.remoting.client.invokerDestructionDelay=0 cisco.dcbu.dcm.jms.port=4457 cisco.dcbu.dcm.device.managed =75
Trouble with Device Management
Check
Table 38-6
for symptoms related to device management. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-10 Trouble with Device Management
|
|
|
Clearing the log file or the accounting log on a Cisco NX-OS device does not cause DCNM-LAN to rediscover the device automatically.
|
The device did not generate a system message about the accounting log or the log file being cleared. This problem is particularly likely if the device is a Cisco MDS 9000 Family Multilayer Switch running Cisco SAN-OS Release 3.1 or earlier releases.
|
Rediscover the device.
|
The DCNM-LAN client shows device configuration information that is out of date.
|
The DCNM-LAN server was down.
|
You can do either of the following:
-
Rediscover the device. For more information, see the “Devices” section.
-
Restart the DCNM-LAN server with a clean database. If the server was down for a long time, this action is the recommended solution.
1. Stopping DCNM-LAN Servers
2. Cleaning a DCNM-LAN Database
3. Starting DCNM-LAN Servers
|
Trouble with Topology
Check
Table 38-11
for symptoms related to using the topology feature. For each symptom that describes your problem, determine which possible cause applies and follow the corresponding solution.
Table 38-11 Trouble with Topology
|
|
|
-
Links between Cisco MDS 9000 Family Multilayer Switches continue appear after the link has gone down.
|
Devices are connected by Gigabit Ethernet or Fast Ethernet ports, and are running Cisco SAN-OS Release 3.1 or earlier releases.
|
Rediscover the devices that topology incorrectly shows as linked.
|
Trouble with Device OS Management
Check
Table 38-12
for symptoms related to the Device OS Management feature. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-12 Trouble with Device OS Management
|
|
|
-
During a software installation job, the software image file transfer between a file server and a device takes too much time.
|
The connection between the file server and the device is slow.
|
Use a file server that is on the same LAN as the devices included in the software installation job.
If all of the available file servers transfer software image files too slowly, before you create the software installation job, manually copy the files to the devices that you will include the job and configure the job to use the manually copied files rather than a file server.
For information about configuring a software installation job, see the
System Management Configuration Guide, Cisco DCNM for LAN, Release 6.x
.
|
Trouble with Event Browsing
Check
Table 38-13
for symptoms related to event browsing issues. For each symptom that describes your problem, determine which possible causes apply and follow the corresponding solutions.
Table 38-13 Trouble with Event Browsing
|
|
|
-
Events available on the device command line do not appear in the DCNM-LAN client.
-
Too few events are shown in Event Browser or an Events tab.
|
Logging levels on managed devices are set incorrectly.
|
Check the logging level configuration on managed devices. See the “Cisco NX-OS System-Message Logging Requirements” section.
|
The DCNM-LAN client fetches events that are not old enough.
|
Check the events-related setting in the DCNM-LAN client preferences. For more information, see the “Configuring the Maximum Age of Events Fetched from the Server” section.
|
Too many events are shown in Event Browser or on an Events tab.
|
A managed device has an issue that is generating many system log messages.
|
Temporarily unmanage the device until you resolve the issues on the device.
|
Logging levels on managed devices are set incorrectly.
|
Check the logging level configuration on managed devices. See the “Cisco NX-OS System-Message Logging Requirements” section.
|
A feature Events tab does not show events that appear in the Event Browser.
|
By design, an Events tab shows only messages that apply to the currently selected feature and may show only a subset of the possible messages for the feature. For more information, see the “Events Tabs Show Fewer Events than the Event Browser” section.
|
Use the Event Browser to see status-related system messages received by DCNM-LAN. For more information, see the
System Management Configuration Guide, Cisco DCNM for LAN, Release 6.x
.
|