Cisco IPICS Troubleshooting Guide, Release 2.0(1)
Troubleshooting Cisco IPICS Network Processes
Downloads: This chapterpdf (PDF - 264.0KB) The complete bookPDF (PDF - 2.01MB) | Feedback

Troubleshooting Cisco IPICS Network Processes

Table Of Contents

Troubleshooting Cisco IPICS Network Processes

Performing Tomcat Service Procedures

Checking the Status of the Tomcat Service

Stopping the Tomcat Service

Starting the Tomcat Service

Restarting the Tomcat Service

Performing Database Server Procedures

Checking the Status of the Database Server

Restarting the Database Server

Starting the Database Server

Performing License Manager Procedures

Checking the Status of the License Manager

Restarting the License Manager

Starting the License Manager

Performing Dial Engine Procedures

Checking the Status of the Dial Engine

Stopping the Dial Engine

Restarting the Dial Engine

Starting the Dial Engine

Performing CSA Procedures

Viewing CSA Log Messages

Stopping CSA

Starting CSA


Troubleshooting Cisco IPICS Network Processes


When you boot up the Cisco IPICS server, the server software automatically starts the following network processes:

Tomcat service

Database server

License manager

Dial engine (if the policy engine is licensed for your server)

Cisco Security Agent (CSA)

This chapter provides information to help you to troubleshoot these, and includes the following sections:

Performing Tomcat Service Procedures

Performing Database Server Procedures

Performing License Manager Procedures

Performing Dial Engine Procedures

Performing CSA Procedures


Note This chapter provides you with procedures that require you to check the network processes by accessing the server via a terminal console session, logging in with the root user ID, and entering command-line interface (CLI) commands. If the tomcat service and database server are both running, you can check their status without using CLI by logging in to the Administration Console, navigating to the Serviceability > Diagnostics window, and viewing the information in the Diagnostic Summary pane.


Performing Tomcat Service Procedures

The tomcat service contains all of the Cisco IPICS web-based applications. The tomcat service runs processes that are required for the functional operation of Cisco IPICS, and must run continuously for you to access the Administration Console and other web applications.

Cisco IPICS includes a safeguard to make sure that the tomcat service continues to run. This safeguard is a cron job that checks the status of the tomcat service every 60 seconds and is able to restart the service automatically, if the tomcat service stops.

This section includes the following topics:

Checking the Status of the Tomcat Service

Stopping the Tomcat Service

Starting the Tomcat Service

Restarting the Tomcat Service

Checking the Status of the Tomcat Service

You can check the status of the tomcat service by navigating to the Serviceability > Diagnostics window of the Administration Console and viewing the Cisco IPICS Tomcat Web Server Status field.

If the tomcat service or the database server is not running, you cannot check its status in the Administration Console. In this case, you can enter CLI commands to check the status of the tomcat service. To check the status of the tomcat service by using CLI commands, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 Check the status of the tomcat service by entering the following command:

[root]# service ipics_tomcat status

If the tomcat service is running properly, the status command returns a process similar to the following example:

Tomcat process (pid: 24025) is running on the system

If the tomcat service is not running, the response to the status command is similar to the following example:

Tomcat is not running on the system.


If the status command shows that the tomcat service is not running, you can start it manually by entering the service ipics_tomcat start CLI command. For more information, see the "Starting the Tomcat Service" section.

Stopping the Tomcat Service

If you do not want any users to access the Administration Console when you perform system maintenance tasks, such as database-related activities, you can stop the tomcat service.

To stop the tomcat service, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

The console terminal displays.

Step 2 To stop the tomcat service, enter the following command:

[root]# service ipics_tomcat stop

If the tomcat service stops, Cisco IPICS displays the message [OK].

If the tomcat service does not stop, Cisco IPICS displays an error message. If you cannot stop the tomcat service, continue to Step 3.

Step 3 If the tomcat service fails to stop, you can terminate the processes that are running by performing the following procedure:

a. To check which tomcat processes are still running, enter the following grep command, which returns information about the tomcat processes that continue to run:

[root]# ps -ef | grep tomcat

b. Note the Process IDs, which display in the second column of the grep results.

c. To stop the tomcat processes that are still running, enter the following command:

[root]# kill -9 <process-id>

where:

<process-id> specifies the Process IDs that you noted in Step b.

d. Repeat Step c for every tomcat process that is running.

Step 4 Check the status of the tomcat service by entering the following command:

[root]# service ipics_tomcat status

If the tomcat service stops successfully, the following message displays:

Tomcat is not running on the system.

Step 5 If a message displays that indicates that the tomcat service is running on the system, contact your Cisco technical support representative for further assistance.


Starting the Tomcat Service

If the cron job fails to start the tomcat service successfully, or if you stop the tomcat service, you can start the service manually by using CLI commands.

To manually start the tomcat service, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

A terminal window displays.

Step 2 To start the tomcat service, enter the following command:

[root]# service ipics_tomcat start

If you successfully started the tomcat service, Cisco IPICS displays the message [OK].


Note There may be a delay of a few minutes before users can access the Administration Console after the tomcat service starts.


Step 3 If the tomcat service does not successfully start, check the following files to gather information on the nature of the problem:

/root/tomcat/current/logs/catalina.out

/root/tomcat/current/logs/catalina.yyyy-mm-dd.log

where:

yyyy-mm-dd is the date on which the file was created.


Note The catalina logs contain information about the Cisco IPICS web-based processes, including the tomcat service.


Step 4 Attempt to fix the problem based on the information that you obtained in the log files. The logs can provide you with information to find the root cause of a process that could not start, or that terminated unexpectedly.

Step 5 If you cannot resolve the problem with the information in the log files, contact your Cisco technical support representative for further assistance.


Restarting the Tomcat Service

To restart the tomcat service, while it is already running, execute the restart command.

When you restart the tomcat service, the script logs out any users who are logged in to the Administration Console.

To restart the tomcat service, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the tomcat service, enter the following command:

[root]# service ipics_tomcat restart

After restarting the tomcat service, Cisco IPICS displays the message [OK].


Note There may be a delay of a few minutes before users can access the Administration Console after the tomcat service restarts.



Performing Database Server Procedures

The database server performs all database-related activities in Cisco IPICS, such as backup and restore operations and database updates.

This section includes procedures to start, stop, and check the status of the database server and includes the following topics:

Checking the Status of the Database Server

Restarting the Database Server

Starting the Database Server

Checking the Status of the Database Server

You can check the status of the database server via the Administration Console. To do so, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. The database server status is listed in the Cisco IPICS Database Status field.


Note If the database server is stopped, you cannot log in to the Administration Console to check its status.


If you cannot log in to the Administration Console, you can manually check the status of the database server by performing the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the database server, enter the following command:

[root]# service ipics_db status

If the database server is running properly, the status command returns a process that is similar to the following example:

Ipics Database is running...
oninit (pid 21286 21285 21284 21283 21282 21281 21280) is running...

If the database server is not running, the response to the status command is similar to the following example:

Ipics Database is stopped.

If the status command indicates that the database server is not running, start the database server. For more information, see the "Starting the Database Server" sectionl.


Restarting the Database Server

If you are experiencing Cisco IPICS server performance issues, determine whether the database server is the cause of the problem by checking the amount of system resources that the database is consuming. To check system resources, perform one of the following actions:

From the Administration Console, navigate to the Serviceability > Dashboard window and check the memory information that is displayed in the System Dashboard area.

Log in to a console terminal session by using the root user ID; then, enter the top command.

If you determine that Cisco IPICS is using a large amount of memory, you can restart the database server, which might speed up network processes.

To restart the database server, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the database server, enter the following command:

[root]# service ipics_db restart

Cisco IPICS displays the message [OK] when the database server successfully stops, and displays the message [OK] again when the database server successfully restarts.

Step 3 If you receive an error message after you attempt to restart the database server, contact your Cisco technical support representative for further assistance.


Starting the Database Server

Cisco IPICS starts the database server when the server boots up. You can also start the database server manually, if you determine that the database has stopped. To check whether the database is running, see the "Checking the Status of the Database Server" section.

To manually start the database server from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the database server, enter the following command:

[root]# service ipics_db start

If you successfully started the database server, Cisco IPICS displays the message [OK].

Step 3 If the database does not successfully start, check the diagnostics.log file by entering the following command:

[root]# more /opt/cisco/ipics/database/logs/diagnostics.log

Step 4 Press the Spacebar to view all the messages in the log file. To close the message log file, press q.

Step 5 If you cannot resolve the problem with the information in the log file, contact your Cisco technical support representative for further assistance.


Performing License Manager Procedures

The license manager is the network process that manages the Cisco IPICS licenses.

The license manager checks for new licenses every 24 hours. For a new license file to take effect immediately, you must restart the license manager.

This section includes the procedures to start, stop, and check the status of the license manager and includes the following topics:

Checking the Status of the License Manager

Restarting the License Manager

Starting the License Manager

Checking the Status of the License Manager

To check the status of the license manager from the Cisco IPICS Administration Console, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. The database server status is listed in the Cisco IPICS Tomcat Web Server Status field.


Tip Any field that includes the words lmgrd contains information about the license manager.


To manually check the status of the license manager, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the license manager, enter the following command:

[root]# service ipics_lm status

If the license manager is running, the status command displays text that is similar to the following example:

ipics_lm is running (PID 20859).

If the license manager is not running, the status command displays text that is similar to the following example:

ipics_lm is not running.

If the status command indicates that the license manager is not running, start the database. For more information, see the "Starting the Database Server" sectionl.


Restarting the License Manager

If you add files, or change the system date, you must restart the license manager for the license and date changes to take effect.

To restart the license manager from the Administration Console, navigate to the Administration > License Management window and click the Apply button.

To restart the license manager by using CLI commands, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the license manager, enter the following command:

[root]# service ipics_lm restart

Cisco IPICS displays the message [OK] when the license manager successfully stops, and displays the message [OK] again when the license manager successfully restarts.

Step 3 If you receive an error message after you attempt to restart the license manager, contact your Cisco technical support representative for further assistance.


Starting the License Manager

If the license manager has stopped, you should be able to restart it from the Administration Console by navigating to the Administration > License Management window and clicking the Apply button. You can also manually start the license manager from a terminal console session by performing the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the license manager, enter the following command:

[root]# service ipics_lm start

If you successfully started the license manager, Cisco IPICS displays the message [OK].

Step 3 If the license manager does not start, check the status by performing the actions that are documented in the "Checking the Status of the License Manager" section.

Step 4 If you cannot start the license manager, contact your Cisco technical support representative for further assistance.


Performing Dial Engine Procedures

The dial engine controls the dial-in and dial-out functionality for the policy engine. For more information about the policy engine and dial engine, refer to the "Using the Cisco IPICS Policy Engine" chapter in the Cisco IPICS Server Administration Guide, Release 2.0(1).


Note Your Cisco IPICS system must be licensed for the policy engine before you can perform dial engine procedures. To check whether you are licensed for the policy engine, navigate to the Administration > License Management > Summary tab in the Administration Console and check the Policy Engine Base License field. If your system is licensed for the policy engine, the field displays a status of Licensed. For more information about licenses, refer to the "Performing Cisco IPICS System Administrator Tasks" chapter in the Cisco IPICS Server Administration Guide, Release 2.0(1).


This section provides information about starting, stopping, restarting and checking the status of the dial engine and includes the following topics:

Checking the Status of the Dial Engine

Stopping the Dial Engine

Restarting the Dial Engine

Starting the License Manager

Checking the Status of the Dial Engine

To check the status of the dial engine, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the dial engine, enter the following command:

[root]# service ippe_dial_engine status

If the dial engine is running properly, the status command returns a process similar to the following example:

Checking status...
CVD process (pid 11290) is running...
Engine process (pid 11670) is running...

If the dial engine processes are not running, the response to the status command is similar to the following example:

Checking status...
CVD process is NOT running...
Engine process is NOT running...

If the status command indicates that the dial engine is not running, start the database. For more information, see the "Starting the Dial Engine" section.


Stopping the Dial Engine

To stop the dial engine by using CLI commands, perform the following procedure:


Note Cisco IPICS disconnects all active dial-in and dial-out calls when you stop the dial engine.


Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To stop the dial engine, enter the following command:

[root]# service ippe_dial_engine stop

Cisco IPICS displays the message [OK] when the dial engine processes successfully stop.


Restarting the Dial Engine

To restart the dial engine by using CLI commands, perform the following procedure:


Note Cisco IPICS disconnects all active dial-in and dial-out calls when you restart the dial engine.


Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the dial engine, enter the following command:

[root]# service ippe_dial_engine restart

Cisco IPICS displays the message [OK] when the dial engine processes stop and restart.

Step 3 If you cannot restart the dial engine, perform the following steps:

a. Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.

b. Check the status of your license in the Policy Engine Base License field.

The status displays as Licensed or Not Licensed.

c. If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.


Starting the Dial Engine

If the dial engine has stopped, you can manually start it by using CLI commands by performing the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the dial engine, enter the following command:

[root]# service ippe_dial_engine start

If you successfully started the dial engine processes, Cisco IPICS displays the message [OK].

Step 3 If you cannot start the dial engine, perform the following steps:

a. Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.

b. Check the status of your license in the Policy Engine Base License field.

The status displays as Licensed or Not Licensed.

c. If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.


Performing CSA Procedures

CSA provides threat protection for server and desktop computing systems. It also prevents users from performing unauthorized actions on the server. There may be times where stopping CSA is necessary to perform system-level functions, to debug an issue, or to edit protected system files.

This section includes the following topics:

Viewing CSA Log Messages

Stopping CSA

Starting CSA

Viewing CSA Log Messages

If CSA denies a particular action, such as when a user or process attempts to modify or delete a protected file, the process generates a message similar to the following example:

Oct 15 04:02:02 [hostname] CiscoSecurityAgent[3480]: Event: The 
process '/bin/cp' (as user root(0) group root(0)) attempted to access 
'/var/cache/man/whatis'. The attempted access was an open. The 
operation was denied.

You can view the CSA actions in the Security Event Log. To view the Security Event Log, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 Navigate to the /var/log directory on the Cisco IPICS server by entering the following command:

[root]# cd /var/log

Step 3 To list the files that start with the csalog name in the directory, enter the following command:

[root]# ls -l csalog*

All files that begin with csalog display.


Note The Security Event Log file is named csalog. If the csalog file has reached its maximum size, Cisco IPICS creates a new file called csalog.0, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file. If the csalog file again reaches its maximum size, Cisco IPICS renames the csalog.0 file to csalog.1, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file.


Step 4 To view the log file, enter the following command:

[root]# cat csalog.x

where:

x is the numeric extension of the file (if applicable).

A text viewer window displays the contents of the Security Event Log.


Stopping CSA

You can stop CSA by issuing a command in a terminal console session. To stop CSA from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To stop CSA, enter the following command:

[root]# service ciscosec stop

Cisco IPICS displays the message [OK] after CSA stops.


Starting CSA

If you installed CSA with the Cisco IPICS server software, CSA starts automatically when the Cisco IPICS server boots up. If you stop CSA or if CSA stops on its own for any reason, you can restart CSA in the CSA utility or by entering CLI commands.

To start the CSA process from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start CSA, enter the following command:

[root]# service ciscosec start

Cisco IPICS displays the message [OK] after CSA starts.