Cisco IPICS Troubleshooting Guide, Release 2.1(1)
Troubleshooting Cisco IPICS Network Processes
Downloads: This chapterpdf (PDF - 452.0KB) The complete bookPDF (PDF - 2.15MB) | Feedback

Troubleshooting Cisco IPICS Network Processes

Table Of Contents

Troubleshooting Cisco IPICS Network Processes

Performing Tomcat Service Procedures

Checking the Status of the Tomcat Service

Manually Stopping the Tomcat Service

Manually Starting the Tomcat Service

Manually Restarting the Tomcat Service

Performing Database Server Procedures

Checking the Status of the Database Server

Manually Restarting the Database Server

Manually Starting the Database Server

Performing License Manager Procedures

Checking the Status of the License Manager

Restarting the License Manager

Manually Starting the License Manager

Performing Dial Engine Procedures

Checking the Status of the Dial Engine

Manually Stopping the Dial Engine

Manually Restarting the Dial Engine

Manually Starting the Dial Engine

Performing CSA Procedures

Viewing CSA Log Messages

Manually Stopping CSA

Manually Starting CSA


Troubleshooting Cisco IPICS Network Processes


When you boot up the Cisco IPICS server, the server software automatically starts the following network processes:

Tomcat service

Database server

License manager

Dial engine (if the policy engine is licensed for your server)

Cisco Security Agent (CSA)

This chapter provides information to help you to troubleshoot these processes and includes the following sections:

Performing Tomcat Service Procedures

Performing Database Server Procedures

Performing License Manager Procedures

Performing Dial Engine Procedures

Performing CSA Procedures


Note This chapter documents procedures that use command-line interface (CLI) commands. If the tomcat service and database server are both running, you can check their status without using CLI by logging in to the Administration Console, navigating to the Serviceability > Diagnostics window, and viewing the information in the Diagnostic Summary pane.


Performing Tomcat Service Procedures

The tomcat service contains all of the Cisco IPICS web-based applications. The tomcat service runs processes that are required for the functional operation of Cisco IPICS, and must run continuously for you to access the Administration Console and other web applications.

Cisco IPICS includes a cron job that acts as a safeguard to make sure that the tomcat service continues to run. This cron job checks the status of the tomcat service every 60 seconds and can restart the service automatically, if the tomcat service stops.

This section includes information about the tomcat service in the following topics:

Checking the Status of the Tomcat Service

Manually Stopping the Tomcat Service

Manually Starting the Tomcat Service

Manually Restarting the Tomcat Service

Checking the Status of the Tomcat Service

You can check the status of the tomcat service by navigating to the Serviceability > Diagnostics window of the Administration Console and viewing the Cisco IPICS Tomcat Web Server Status field.

If the tomcat service or the database server is not running, you cannot check its status in the Administration Console. In this case, you must enter CLI commands to check its status. To check the status of the tomcat service by using CLI commands, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 Check the status of the tomcat service by entering the following command:

[root]# service ipics_tomcat status

If the tomcat service is running properly, the command returns a process similar to the following example:

Tomcat process (pid: 24025) is running on the system

If the tomcat service is not running, you should see a response that is similar to the following example:

Tomcat is not running on the system.


If the tomcat service is not running, you can start it manually by entering the service ipics_tomcat start CLI command from the root user ID. For more information, see the "Manually Starting the Tomcat Service" section.

Manually Stopping the Tomcat Service

If you do not want any users to access the Administration Console when you perform system maintenance tasks, such as database-related activities, you can stop the tomcat service.

To stop the tomcat service, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

The terminal console displays.

Step 2 To stop the tomcat service, enter the following command:

[root]# service ipics_tomcat stop

If the tomcat service stops, Cisco IPICS displays the message [OK].

If the tomcat service does not stop, an error message displays that indicates that Cisco IPICS cannot stop the service; if you receive this message, continue to Step 3.

Step 3 If the tomcat service fails to stop, you can terminate the processes that are running by performing the following procedure:

a. To check which tomcat processes are still running, enter the following grep command, which returns information about the tomcat processes that continue to run:

[root]# ps -ef | grep tomcat

This command shows you a list of the running tomcat processes.

b. Note the Process IDs, which display in the second column of the grep results.

c. To stop the tomcat processes that are still running, enter the following command:

[root]# kill -9 <process-id>

where:

<process-id> specifies the Process IDs that you noted in Step b.

d. Repeat Step c for every tomcat process that is running.

Step 4 Check the status of the tomcat service by entering the following command:

[root]# service ipics_tomcat status

If the tomcat service stops successfully, the following message displays:

Tomcat is not running on the system.

Step 5 If the tomcat service does not stop, check the ipics.log file to gather information on the nature of the problem by entering the following command:

[root]# tail -75 /root/tomcat/current/logs/ipics.log


Note The ipics.log file contains messages regarding all transactions that occur in the Cisco IPICS server, including tomcat service transactions.


The last 75 lines of the ipics.log file displays.

Step 6 Attempt to fix the problem based on the information that you view in the log file.

The information might include information about a running process that could not stop.

Step 7 If you cannot resolve the problem by using the information in the ipics.log file, contact your Cisco technical support representative for further assistance.


Manually Starting the Tomcat Service

If the cron job fails to start the tomcat service successfully, or if you stop the tomcat service, you can start the service manually by using CLI commands.

To manually start the tomcat service, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

A terminal window displays.

Step 2 To start the tomcat service, enter the following command:

[root]# service ipics_tomcat start

If the tomcat service starts successfully, Cisco IPICS displays the message [OK].


Note After you start the tomcat service, there may be a delay of a few minutes before users can access the Administration Console.


Step 3 If the tomcat service does not successfully start, check the ipics.log file to gather information on the nature of the problem by entering the following command:

[root]# tail -75 /root/tomcat/current/logs/ipics.log


Note The ipics.log file contains messages regarding all transactions that occur in the Cisco IPICS server, including tomcat service transactions.


The last 75 lines of the ipics.log file displays.

Step 4 Attempt to fix the problem based on the information that you view in the log file.

The information might contain a Java exception error or provide information about a process that did not start.

Step 5 If you cannot resolve the problem by using the information in the ipics.log file, contact your Cisco technical support representative for further assistance.


Manually Restarting the Tomcat Service

To restart the tomcat service, perform the following procedure:


Note When you restart the tomcat service, the script logs out any users who are logged in to the Administration Console.


Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the tomcat service, enter the following command:

[root]# service ipics_tomcat restart

After restarting the tomcat service, Cisco IPICS displays the message [OK].


Note After you start the tomcat service, there may be a delay of a few minutes before users can access the Administration Console.



Performing Database Server Procedures

The database server performs all database-related activities in Cisco IPICS, such as backup and restore operations and database updates.

This section includes the procedures to start, stop, and check the status of the database server in the following topics:

Checking the Status of the Database Server

Manually Restarting the Database Server

Manually Starting the Database Server

Checking the Status of the Database Server

You can check the status of the database server via the Administration Console. To do so, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. Navigate to the Cisco IPICS Database Status area in this pane to view the database server status.

If the database server is stopped, you cannot log in to the Administration Console to check its status; however, you can check the status of the database server by entering CLI commands. To check the status of the database server, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the database server, enter the following command:

[root]# service ipics_db status

If the database server is running properly, the server displays a message that is similar to the following example:

Ipics Database is running...
oninit (pid 21286 21285 21284 21283 21282 21281 21280) is running...

If the database server is not running, the server displays a message that is similar to the following example:

Ipics Database is stopped.

If the database server is not running, you can manually start the database server by performing the procedure in the "Manually Starting the Database Server" section.


Manually Restarting the Database Server

If you experience Cisco IPICS server performance issues, determine whether the database server is the cause of the problem by checking the system resources that the database is using. To check system resources, perform one of the following actions:

From the Administration Console, navigate to the Serviceability > Dashboard window and check the memory information that is displayed in the System Dashboard area.

Log in to a terminal console session by using the root user ID; then, enter the top command.


Note The top command displays a listing of the server processes that are using the greatest amount of CPU memory.


If you determine that the Cisco IPICS processes are using a large amount of memory, you can restart the database server, which might speed up network processes.

To restart the database server, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the database server, enter the following command:

[root]# service ipics_db restart

Cisco IPICS displays the message [OK] when the database server successfully stops and displays the message [OK] again when the database server successfully restarts.

Step 3 If you receive an error message after you attempt to restart the database server, contact your Cisco technical support representative for further assistance.


Manually Starting the Database Server

Cisco IPICS starts the database server when the server boots up. You can also start the database server manually if you determine that the database has stopped. To check whether the database is running, see the "Checking the Status of the Database Server" section.

To manually start the database server from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the database server, enter the following command:

[root]# service ipics_db start

If the database server starts successfully, Cisco IPICS displays the message [OK].

Step 3 If the database does not successfully start, contact your Cisco technical support representative for further assistance.


Tip To check the log file for the database, enter the following command:

[root]# more /opt/cisco/ipics/database/logs/diagnostics.log

The diagnostics.log file might provide you with additional details about the reason that the database could not start. Messages such as "Assert failed" or "PANIC" indicate a problem with the database that requires the assistance of your Cisco technical support representative.



Performing License Manager Procedures

The license manager is the network process that manages the Cisco IPICS licenses.

The license manager checks for new licenses every 24 hours. For a new license file to take effect immediately, you must restart the license manager.

This section includes the procedures to start, stop, and check the status of the license manager and includes the following topics:

Checking the Status of the License Manager

Restarting the License Manager

Manually Starting the License Manager

Checking the Status of the License Manager

To check the status of the license manager from the Cisco IPICS Administration Console, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. Navigate to the Cisco IPICS Tomcat Web Server Status area in this pane to view the license manager status.


Tip Any field that includes the words lmgrd contains information about the license manager.


To manually check the status of the license manager, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the license manager, enter the following command:

[root]# service ipics_lm status

If the license manager is running, the server displays a message that is similar to the following example:

ipics_lm is running (PID 20859).

If the license manager is not running, the server displays a message that is similar to the following example:

ipics_lm is not running.

Step 3 If the license manager is not running, you can manually start the license manager by following the procedure in the "Manually Starting the License Manager" sectionl.


Restarting the License Manager

If you add files, or change the system date, you must restart the license manager for the license and date changes to take effect.

To restart the license manager from the Administration Console, navigate to the Administration > License Management window and click the Apply button.

To restart the license manager by using CLI commands, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the license manager, enter the following command:

[root]# service ipics_lm restart

Cisco IPICS displays the message [OK] when the license manager successfully stops and displays the message [OK] again when the license manager successfully restarts.

Step 3 If you receive an error message that indicates that the license manager or lmgrd process could not start, contact your Cisco technical support representative for further assistance.


Manually Starting the License Manager

If the license manager has stopped, you should be able to restart it from the Administration Console by navigating to the Administration > License Management window and clicking the Apply button. You can also manually start the license manager from a terminal console session by performing the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the license manager, enter the following command:

[root]# service ipics_lm start

If the license manager starts successfully, Cisco IPICS displays the message [OK].

Step 3 If the license manager does not start, check the status by performing the actions that are documented in the "Checking the Status of the License Manager" section.

Step 4 If you cannot start the license manager, contact your Cisco technical support representative for further assistance.


Performing Dial Engine Procedures

The dial engine controls the dial-in and dial-out functionality for the policy engine. For more information about the policy engine and dial engine, refer to the "Using the Cisco IPICS Policy Engine" chapter in the Cisco IPICS Server Administration Guide, Release 2.1(1).


Note Your Cisco IPICS system must be licensed for the policy engine before you can perform dial engine procedures. To check whether you are licensed for the policy engine, navigate to the Administration > License Management > Summary tab in the Administration Console and check the Policy Engine Base License field. If your system is licensed for the policy engine, the field displays a status of Licensed. For more information about licenses, refer to the "Performing Cisco IPICS System Administrator Tasks" chapter in the Cisco IPICS Server Administration Guide, Release 2.1(1).


This section provides information about starting, stopping, restarting and checking the status of the dial engine and includes the following topics:

Checking the Status of the Dial Engine

Manually Stopping the Dial Engine

Manually Restarting the Dial Engine

Manually Starting the Dial Engine

Checking the Status of the Dial Engine

To check the status of the dial engine, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To check the status of the dial engine, enter the following command:

[root]# service ippe_dial_engine status

If the dial engine is running properly, the server displays a message that is similar to the following example:

Checking status...
CVD process (pid 11290) is running...
Engine process (pid 11670) is running...

If the dial engine processes are not running, the server displays a message that is similar to the following example:

Checking status...
CVD process is NOT running...
Engine process is NOT running...

Step 3 If the dial engine is not running, you can manually start the dial engine by performing the procedure in the "Manually Starting the Dial Engine" section.


Manually Stopping the Dial Engine

To stop the dial engine by using CLI commands, perform the following procedure:


Note Cisco IPICS disconnects all active dial-in and dial-out calls when you stop the dial engine.


Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To stop the dial engine, enter the following command:

[root]# service ippe_dial_engine stop

Cisco IPICS displays the message [OK] when the dial engine processes successfully stop.


Manually Restarting the Dial Engine

To restart the dial engine by using CLI commands, perform the following procedure:


Note Cisco IPICS disconnects all active dial-in and dial-out calls when you restart the dial engine.


Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To restart the dial engine, enter the following command:

[root]# service ippe_dial_engine restart

Cisco IPICS displays the message [OK] when the dial engine processes stop and restart.

Step 3 If you cannot restart the dial engine, perform the following steps:

a. Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.

b. Check the status of your license in the Policy Engine Base License field.

The status displays as Licensed or Not Licensed.

c. Perform one of the following actions based on the output that displays:

If the Policy Engine Base License field shows a status of Not Licensed, you are not licensed for the policy engine. To purchase a license that includes policy engine functionality, refer to the "Obtaining Your License File" section in the "Installing Cisco IPICS" chapter in the Cisco IPICS Server Installation and Upgrade Guide, Release 2.1(1).

If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.


Manually Starting the Dial Engine

If the dial engine has stopped, you can manually start it by using CLI commands by performing the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start the dial engine, enter the following command:

[root]# service ippe_dial_engine start

If the dial engine processes starts successfully, Cisco IPICS displays the message [OK].

Step 3 If you cannot start the dial engine, perform the following steps:

a. Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.

b. Check the status of your license in the Policy Engine Base License field.

The status displays as Licensed or Not Licensed.

c. Perform one of the following actions based on the output that displays:

If the Policy Engine Base License field shows a status of Not Licensed, you are not licensed for the policy engine. To purchase a license and obtain dial engine and policy engine functionality, refer to the "Obtaining Your License File" "Installing Cisco IPICS" chapter of the Cisco IPICS Server Installation and Upgrade Guide, Release 2.1(1).

If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.


Performing CSA Procedures

CSA provides threat protection for server and desktop computing systems. It also prevents users from performing unauthorized actions on the server. There may be times where stopping CSA is necessary to perform system-level functions, to debug an issue, or to edit protected system files.

This section describes how to perform certain CSA procedures and includes the following topics:

Viewing CSA Log Messages

Manually Stopping CSA

Manually Starting CSA

Viewing CSA Log Messages

If CSA denies a particular action, such as when a user or process attempts to modify or delete a protected file, the process generates a message similar to the following example:

Oct 15 04:02:02 [hostname] CiscoSecurityAgent[3480]: Event: The 
process '/bin/cp' (as user root(0) group root(0)) attempted to access 
'/var/cache/man/whatis'. The attempted access was an open. The 
operation was denied.

You can view the CSA actions in the Security Event Log. To view the Security Event Log, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 Navigate to the /var/log directory on the Cisco IPICS server by entering the following command:

[root]# cd /var/log

Step 3 To list the files that start with the csalog name in the directory, enter the following command:

[root]# ls -l csalog*

All files that begin with csalog display.


Note The Security Event Log file is named csalog. If the csalog file has reached its maximum size, Cisco IPICS creates a new file called csalog.0, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file. If the csalog file again reaches its maximum size, Cisco IPICS renames the csalog.0 file to csalog.1, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file.


Step 4 To view the log file, enter the following command:

[root]# cat csalog.x

where:

x is the numeric extension of the file (if applicable).

A text viewer window displays the contents of the Security Event Log. This log provides a detailed list of security events that CSA has detected and the action, if any, that CSA performed as a result of those events.


Manually Stopping CSA

You can stop CSA by issuing a command in a terminal console session. To stop CSA from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To stop CSA, enter the following command:

[root]# service ciscosec stop

Cisco IPICS displays the message [OK] after CSA stops.


Manually Starting CSA

If you installed CSA with the Cisco IPICS server software, CSA starts automatically when the Cisco IPICS server boots up. If you stop CSA or if CSA stops on its own for any reason, you can restart CSA in the CSA utility or by entering CLI commands.

To start the CSA process from a terminal console session, perform the following procedure:

Procedure


Step 1 Log in to the Cisco IPICS server by using the root user ID.

Step 2 To start CSA, enter the following command:

[root]# service ciscosec start

Cisco IPICS displays the message [OK] after CSA starts.