Appendix 1 Troubleshooting the WEM

Appendix 1
Troubleshooting the WEM
 
 
 
This appendix provides information on troubleshooting the following:
In addition to the above, instructions are also provided for capturing client and server logs. These are provided in the Capturing WEM Client Logs and Capturing WEM Server Logs using Script sections of this appendix.
 
Issues Pertaining to Installation
Determine the status of the /var/tmp directory by entering the df -k command. If it is at or near capacity, choose another directory for the Host Base Directory parameter setting. This parameter can be set via the installation process.
Enter the ps -ef | grep server command to determine if any process instances from previous installations are running. If so, stop them using the instructions in the WEM Server Files and Operation chapter of this guide. Once stopped, start the processes for the current installation using the instructions in the same appendix.
 
Issues Related to the Starting of the WEM
Verify that the Postgres database is running by entering the ps -ef | grep post command. If is not, follow the instructions in WEM Server Files and Operation chapter of this guide to start it.
Determine if the log directory exists in <ems-install-dir>/server (default directory) using the ls command to display the contents of the directory. If it is missing, create it using the mkdir command and stop and restart all WEM processes using the instructions in WEM Server Files and Operation chapter of this guide.
Enter the ps -ef | egrep “server|bulkstatparser|bulkstatserver|scriptsrv” command to determine if WEM server processes are running and, if they are, what directory did they originate from. If they’re different, stop the processes and restart the server from within the desired installation directory using the instructions in WEM Server Files and Operation chapter of this guide.
Determine if the “ServerPort” and/or “ServerIIOPPort” port numbers specified in the nms.cfg file (located in the <ems-install-dir>/server/etc directory by default) are already in use. The default “ServerPort” is 22222, and the default “ServerIIOPPort” is 15000. This can be determined by entering the netstat -a command which displays a list of all the process addresses and ports in use in “ipaddress.port” format. If they are in use, either stop the other processes or configure new values for these parameters.
Determine if a previous Postgres instance is still using system resources by entering the ipcs command. If it is, clear the resources by entering the ipcrm command.
NOTE: The table name in the above message is after the 'FROM' keyword.
 
Issues Related to Login
Verify that server processes are running using the information in WEM Server Files and Operation chapter of this guide.
 
Verify that the .java.policy file is present in your home directory. Refer to Preparing and Using the Client Workstation chapter of this guide for more information.
NOTE: All instances of the browser must be closed and restarted after the policy file has been updated.
Check the configuration of the no limit ConsecutiveFailLogin parameter in the ua.cfg file (located in the <ems-install-dir>/server/etc directory by default). If users are frequently locked out due to reaching the maximum limit, you may consider increasing the limit, or disabling the functionality. You may also consider reducing the amount of time the account is locked out by modifying the configuration of the no locked out LockOutInterval parameter also contained in the ua.cfg file.
 
Issues Related to the Web Browser
For JRE versions greater that 1.5: The Temporary Internet Files group in the General tab of the Java Control Panel should be used to disabled caching.
For JRE versions greater that 1.6: The Temporary Internet Files group in the General tab of the Java Control Panel should be used to disabled caching.
 
Issues Pertaining to CORBA Communication
Ensure ICMP connectivity between the system and the WEM Server using the ping <wem_server_ip_address> command from the chassis’ command prompt. Refer to the Command Line Interface Reference for more information on using this command.
Verify that the ORBEM client identification on the chassis matches that configured on the WEM. The configuration of this parameter on the chassis can be determined by entering the show configuration | grep client command. On the WEM, check for the ASID (Application Server ID), Port, and SSL-enabled flag (IIOP/SIOP) on the Modify IMG screen. Change these settings as needed.
Check the status of the ORBEM client on the chassis by executing the show orbem client id <client_id> command on the chassis. The “State” should be “Enabled”. If the “State” is “Disabled”, execute the activate client id <client_id> command in the ORBEM Configuration Mode and check the status again-- it should now be “Enabled”.
Verify that the configuration of the IIOP port on the chassis matches that configured for the WEM. The configuration of this parameter on the chassis can be determined by entering the show configuration | grep iiop-port. On the WEM, check for the ASID (Application Server ID), Port, and SSL-enabled flag (IIOP/SIOP) on the Modify IMG screen. Change these settings as needed. Change these settings as needed.
Verify that the IIOP transport parameter is enabled on the chassis by entering the show configuration | grep iiop-transport command. If it is not, enable using the instructions found in the System Administration and Configuration Guide.
Check if the SSL is enabled and/or enforced on the WEM. If the SSL is enabled, disable the IIOP transport on the chassis and set the value of IMG Port for the chassis such that it is identical to the SIOP port parameter configured on the chassis.
Edit the img.html file (located in the <ems-install-dir>/client directory (by default) to use fix ports and open the required ports in the firewall. This requires the configuration of the "FIXED_PORT", "FIXED_PORT_RANGE_START" and "FIXED_PORT_RANGE_END".
 
Issues Related to Bulkstatistics
Verify that the FTP server process is running on the server by issuing the ps -ef | grep in.ftpd command. If it is not, start it.
Verify that the latest Solaris operating system patches are installed. Refer to WEM Port and Hardware Information chapter of this guide for more information.
Ensure that the configuration the bulk statistics receiver on the managed system Executing the show bulkstats command on the chassis displays this information. The “Remote File Format” field should contain a valid directory on the WEM Server. (Also verify that this directory exists on the server.) The “Bulkstats Receivers” field should contain the IP address of the WEM Server.
Verify that the Bulkstatistic Server process is running by entering the ps -ef | grep bulkstatserver command. If it is not, execute the ./serv bulkstatserver start command from within the server directory (<ems-install-dir>/server by default).
Verify that the Bulkstatistic Parser process is running by entering the ps -ef | grep bulkstatparser command. If it is not, execute the ./serv parserserver command from within the server directory (<ems-install-dir>/server by default).
Verify that the bulkstatistics format is compatible with the WEM. Refer to the bs.cfg file (located in the <ems-install-dir>/server/etc directory by default) for WEM bulkstatistic formatting.
Important: If XMLFileType is set to 1, it will generate XML files irrespective of the other two mentioned configurables.
Reconfiguration of schema is not done after upgrade. Refer to Reconfiguration of Bulkstat Schemas section of this guide for more information.
 
Issues Pertaining to Configuration Backup
Verify that the FTP server process is running on the server by issuing the ps -ef | grep in.ftpd command. If it is not, start it.
Verify that the latest Solaris operating system patches are installed. Refer to WEM Port and Hardware Information chapter of this guide for more information.
 
Issues Pertaining to Alarms
Verify that the SNMP target IP address and port number configured on the chassis match that of the WEM server. The SNMP target configuration on the chassis can be determined by entering the show snmp transports command. Check this information against the WEM server IP address (“ServerIpAddress”, specified in the nms.cfg file) and the SNMP port number (“SnmpTrapPort”, specified in the fm.cfg file) parameters. (Both of these files are located in the <ems-install-dir>/server/etc directory by default.)
Verify that the E-mail information configured in the Alarm Configuration dialog of the WEM is correct.
Verify that the Script Server is running by entering the ps -ef | grep scriptsrv command. If it is not, execute the ./serv scriptserver command from within the server directory (<ems-install-dir>/server by default).
Verify that the script file is located in the <ems-install-dir>/server/scripts directory (this is the default directory). If it is not, copy the script to that location.
Verify that the script can be executed by entering the ls -al command from within the directory in which the script is located.
 
Issues Pertaining to the Process Monitor (PSMON)
Verify that PSMON is running by entering the ps -ef | grep psmon command. If it is not, start it using the instructions located in WEM Process Monitor chapter of this guide.
 
Issues Pertaining to Java
Library=C:\Program Files\Java\j2re1.4.2_04\bin\jsound.dll
 
Issues Pertaining to WEM Upgrade
 
Capturing WEM Client Logs
 
In the event that an issue exists that could not be solved using the information provided previously in this chapter, you may need to capture client logs for debugging purposes. This section provides information on how to utilize logging for WEM clients.
Step 1
Step 2
Step a
Step b
Step 3
Step 4
The Java Console contains log messages that could be used for debugging the issue.
 
Capturing WEM Server Logs using Script
In the event additional troubleshooting assistance is required, debugging information can be collected using a script called getSupportDetails.pl. This script collects different log files and captures the output of certain system commands that aid in troubleshooting issues. This script is packaged with the WEM Server in the <EMS_INSTALL_DIR>/tools/supportdetails/ directory.
 
This script refers to an XML file to get the list of logs. This XML resides in the same directory as the script. Once executed, the script retrieves the contents of logs, files, folders, and output of certain commands and prepares a zipped file (/tmp/log/emssupportDetails.tar.gz), by default it is placed in /tmp/log directory.
Requirements:
Perl 5.8.5 and above is required for running the script. This is packaged with the WEM Server. Apart from standard Perl modules (which are included in default installation of Perl), some additional modules are required for running the script. The list is as follows:
 
These modules are installed by default by the WEM application. Please ensure that the above mentioned modules are installed when using a different installation of Perl.
To run the script, go to the path where the script is present and enter:
./getSupportDetails.pl [--level=...] [--xmlfile=...] [--help]
For example, ./getSupportDetails --level=4 --xmlfile=/tmp/something.xml
 
WEM IP Address Change Procedure
In the event the customer’s network evolves, the IP address of the Solaris server might be required to change from the existing one. In order to change the WEM server IP address, use the following defined IP planning process:
Step 1
 
./serv stop
Step 2
For example: ifconfig bge0 192.168.1.1 netmask 255.255.255.0 up
Step 3
 
vi nms.cfg
Replace the IP address with the new IP address in the modify serverIpAddress as follows:
For example: ServerIpAddress = 192.168.1.1
Save the file after making the appropriate changes.
Step 4
 
# cat /etc/hosts
<IP address> localhost
<new_IP_address> solaris_hostname
#
Step 5
 
./serv start
Important: /etc/netmasks needs to be modified if the user is subnetting existing address and subsequently using a different network mask than the default one. If the netmask being used for a given IP address is a default one, then there is no need to modify this file.
 

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883