Troubleshoot Cisco Insight Reporter Issues

Introduction

This document describes how to troubleshoot various issues related to the Insight reporter, which is a software platform integrated with Cisco Service Control Solution and is designed to generate various reports, charts and statistics.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Service Control Solution
• Basic Knowledge on Service Control Engine (SCE), Collection Manager (CM) and Insight reporter

Components Used

The information in this document is based on the tests carried out with Insight version 4.2. However this document is not restricted to any specific software version.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Troubleshoot Insight Issues

If the Insight Reporter is not able to generate any reports, follow these steps to fix the issue.

Step 1. Verify that the required jconn3.jar files are copied in to the Insight Server. The default Insight installation kit is bundled with Java Database Connectivity (JDBC) connector for MySQL 5.x. In order to use Sybase (ASE v 12.x or later) or Oracle (v 9.x or later) as traffic database, you might need to copy the compatible JDBC connector jars (not provided with the Cisco Insight Reporter) for the respective database server to following directory.

 [user home directory]/Insight/apache-tomcat-6.0.20/webapps/insight/WEB-INF/lib

Once the file is copied, the Insight server needs to be restarted for the changes to take effect.

/etc/init.d/tomcat-instance stop
 /etc/init.d/tomcat-instance start

Step 2. Verify that the Insight server is able to communicate with the Traffic Database.

Open the Insight GUI and click on the Settings Management icon on the module launcher. Select the database from the Traffic Databases tab and click on Edit. As displayed, Edit Traffic Database window appears. You have to configure the different fields like DB IP address, DB schema,username,password and so on. After configuration, click on the Test button. If the DB connectivity is successful, it shows Configuration Test Successful as shown in this image.

If the test results in any error like Remote Databse cannot be connected, it points to a communication issue with the database. Follow these steps to fix the issue:

• Verify the Database) parameters are correct
• Verify the DB is up and running
• Verify the IP connectivity between Insight server and the traffic database

Step 3. Verify that the SCE sends the Raw Data Records (RDR) to the CM. This command can be used to verify whether the RDR count is incremented and there are no RDRs queued or thrown in the SCE.

 SCE8000-1#>show RDR-formatter 
 Status:  enabled
 Connection is: down
 Forwarding mode: multicast
 Connection Table:
 ------------------------------------------------------------------------------------------
 | Collector       | Port  |Status| Priority per Category:                                |
 | IP-Address  /   |       |      |-------------------------------------------------------|
 | Host-Name       |       |      | Category1   | Category2   | Category3   | Category4   | 
 |-----------------|-------|------|-------------|-------------|-------------|-------------|
 | 10.106.38.250   | 33000 | down | 100         | 100         | ---         | ---         | 
 ------------------------------------------------------------------------------------------
 RDR:    queued: 46629 ,sent: 28417, thrown:     0, format-mismatch:     0
 UM:     queued:     0 ,sent:     0, thrown:     0
 Logger: queued:     0 ,sent:     0, thrown:     0
 Errors: thrown:     0
 Last time these counters were cleared: 10:19:09  IST  MON  April  11  2016
 SCE8000-1#>

Step 4. Verify that the CM is running, receiving RDRs from the SCE, procesing RDRs and send it to the traffic database.

This script can be used to verify that the CM is functional.

-bash-4.1$ ~scmscm/setup/alive.sh 


STATUS OK. CM is functioning

There is one process for each CM adapter enabled, hence it is useful to identify which all CM adapters are running at the moment.

-bash-4.1$ ps -ef | grep scmscm.adapters
 scmscm    1726 32298  0 00:47 pts/1    00:00:00 grep scmscm.adapters
 scmscm    3910  1663  0 Apr12 ?        00:03:14 /opt/CM510/cm/lib/java/bin/java -DADAPTER=RAGAdapter -Djava.security.egd=file:/dev/./urandom -DconfigFile=cm.conf -server -Xconcurrentio -DAPP=com.cisco.scmscm.adapters.rag.RAGAdapter -Dcom.cisco.scmscm.configPath=/opt/CM510/cm/config/ -Dcom.cisco.scmscm.home=/opt/CM510 -Dcom.cisco.scmscm.logPath=/opt/CM510/cm/logs -Xmx4076M com.cisco.scmscm.adapters.BaseAdapter com.cisco.scmscm.adapters.rag.RAGAdapter /opt/CM510/cm/adapters/RAGAdapter/ /opt/CM510/cm/config/ 10000
 scmscm    3913  1663  0 Apr12 ?        00:02:52 /opt/CM510/cm/lib/java/bin/java -DADAPTER=TAAdapter -Djava.security.egd=file:/dev/./urandom -DconfigFile=cm.conf -server -Xconcurrentio -DAPP=com.cisco.scmscm.adapters.topper.TAAdapter -Dcom.cisco.scmscm.configPath=/opt/CM510/cm/config/ -Dcom.cisco.scmscm.home=/opt/CM510 -Dcom.cisco.scmscm.logPath=/opt/CM510/cm/logs -Xmx512M com.cisco.scmscm.adapters.BaseAdapter com.cisco.scmscm.adapters.topper.TAAdapter /opt/CM510/cm/adapters/TAAdapter/ /opt/CM510/cm/config/ 10000
 scmscm    3919  1663  0 Apr12 ?        00:02:46 /opt/CM510/cm/lib/java/bin/java -DADAPTER=CSVAdapter -Djava.security.egd=file:/dev/./urandom -DconfigFile=cm.conf -server -Xconcurrentio -DAPP=com.cisco.scmscm.adapters.CSVAdapter -Dcom.cisco.scmscm.configPath=/opt/CM510/cm/config/ -Dcom.cisco.scmscm.home=/opt/CM510 -Dcom.cisco.scmscm.logPath=/opt/CM510/cm/logs com.cisco.scmscm.adapters.BaseAdapter com.cisco.scmscm.adapters.CSVAdapter /opt/CM510/cm/adapters/CSVAdapter/ /opt/CM510/cm/config/ 10000
 scmscm    3923  1663  0 Apr12 ?        00:04:15 /opt/CM510/cm/lib/java/bin/java -DADAPTER=JDBCAdapter -Djava.security.egd=file:/dev/./urandom -DconfigFile=cm.conf -server -Xconcurrentio -DAPP=com.cisco.scmscm.adapters.jdbc.JDBCAdapter -Dcom.cisco.scmscm.configPath=/opt/CM510/cm/config/ -Dcom.cisco.scmscm.home=/opt/CM510 -Dcom.cisco.scmscm.logPath=/opt/CM510/cm/logs com.cisco.scmscm.adapters.BaseAdapter com.cisco.scmscm.adapters.jdbc.JDBCAdapter /opt/CM510/cm/adapters/JDBCAdapter/ /opt/CM510/cm/config/ 10000

In order to verify that the CM receives the RDRs from the SCE and the rate at which the various SCEs send the RDRs, use this script.

-bash-4.1$ ~scmscm/scripts/sceconf.sh --op=list
 IP               Rate       Peak      
 ---------------- ---------- ----------
 10.106.37.14     0.04       1001.56   

 Command terminated successfully

This script provides the status of the CM, DB and the status of CM to DB connectivity.

-bash-4.1$ ~scmscm/setup/monitor/monitor.sh -a -d 
 Test: 01db_up.sh. Status: PASS. Message: db is up
 Test: 02cm_up.sh. Status: PASS. Message: CM is running
 Test: 03free_db.sh. Status: PASS. Message: 50% free space in data db
 Test: 04free_log.sh. Status: FAIL. Message: Unsupported for MySQL
 Test: 05cm_persistent_buffers.sh. Status: PASS. Message: All Adapter persistent buffers are in Normal state

This script help to verify that the DB tables are updpted with the lastet data. The script queries the various tables in the traffic database and provide the output with min_time and max_time values. The min_time provides the timestamp information of the first record in the table and the max_time provides the timestamp of the most recent record in the table.

Step 5. Check the CM log files and see whether there are any errors reported in the logs. The CM log files are available in this folder.

~scmscm/cm/logs

Step 6. Verify the INI_VALUES table is updated in the traffic database. The Insight server retrieves the device information from the INI_VALUES table. Whenever the service configuation of the SCE is changed, the INI_VALUES table is updated. Verify that the SCE entries are there in the table with VALUE_TYPE=5 as in this example.

Step 7. Collect the debug logs. In case the Insight is still not able to generate the reports, Cisco Technical Assistance Center (TAC) requires the logs from the Insight server. These are the steps to collect the logs from the Insight server.

• Open the Insight GUI and click on the Settings Management icon on the Module Launcher. Expand the Logging option in the Global Settings tab.
• Set the Event logging threshold to WARN and Application Logging Threshold to DEBUG and save the configuration.
• Run the report from the Insight and immediately collect the logs.log file from the Insight Server. The logs.log file contains the SQL query which the Insight server use to fetch data from the database, for the specific report. The Log File Path in the Logging section of the Global Settings tab provides the exact location of the logs.log file.

Related Information

• Cisco Insight Server User Guide