Cisco PVM Troubleshooting Guide
Tracing
Download this chapter
TracingTracing
Download the complete book
Cisco PVM Troubleshooting Guide, 1.0.2Cisco PVM Troubleshooting Guide, 1.0.2 (PDF - 540 KB)
give_us_feedback

Table Of Contents

Cisco PVM Tracing

Trace File Overview

Trace File Configuration

Trace File Naming and Size Limits

Displaying Trace Files

Using PVM Tracing To Troubleshoot

Choosing the Correct Trace File

Setting the Trace Level To Include Maximum Information

Viewing the Trace File Contents


Cisco PVM Tracing

Cisco PVM provides tracing functions for its collection adapters and server components. These tracing log files may be used to troubleshoot application issues.

Trace File Overview

Cisco PVM trace files are located in the /[CiscoPVM Installation directory]/server/tmp directory. The default location is /opt/CSCOpvm/server/tmp. Each time an adapter or server component is started, a trace file is created. Cisco PVM offers configurable trace levels. The default trace level, 2, provides partial trace information. For troubleshooting purposes, the trace level may be set to any value 0-10 as noted below.

Table 2-1 Trace Files

Trace Level
Result

0

Tracing is off

1

Includes fatal system or application faults

2

Includes information on any error condition that has occurred. This is the default level.

3

Includes information on potential errors, exceptions, or abnormal conditions.

4

Includes major informative tracking messages.

5

Includes minor informative tracking messages.

6-10

Verbose mode. Provides detailed and informative tracking messages


Caution Increasing the trace level beyond the default trace level may impact system performance.

Trace File Configuration

Cisco PVM tracing can be configured using the adapter configuration files located in the [installation directory]/server/etc/agentconfig directory.

Sample output: 

-rw-r--r--  1 pvmadm dba 2293 Dec  6  2005 AlarmOidMap

-rw-r--r--  1 pvmadm dba  331 Jan 20  2006 sp_CiscoART_agent.config

-rw-r--r--  1 pvmadm dba  792 Oct 27  2006 sp_CiscoNAM_826_agent.config

-rw-r--r--  1 pvmadm dba  806 Oct 27  2006 sp_CiscoNAM_827_agent.config

-rw-r--r--  1 pvmadm dba  796 Oct 27  2006 sp_CiscoNAM_828_agent.config

-rw-r--r--  1 pvmadm dba  476 Dec  8  2005 sp_CiscoSwitch_823_agent.config

-rw-r--r--  1 pvmadm dba  476 Dec  8  2005 sp_CiscoSwitch_824_agent.config

-rw-r--r--  1 pvmadm dba  476 Dec  8  2005 sp_CiscoSwitch_825_agent.config

-rw-r--r--  1 pvmadm dba  152 Feb 17  2006 sp_deviceImport.config

-rw-r--r--  1 pvmadm dba   56 Sep 30  2005 sp_hostSync.config

-rw-r--r--  1 pvmadm dba  264 Jan 18  2006 sp_thresholdMonitor.config

-rw-r--r--  1 pvmadm dba  530 Jan 26  2006 SystemHealthMonitor.properties

To view or edit a components trace level, open the corresponding .config file. For example, to view the configuration for the CiscoNAM_826 adapter, execute the following command:

more sp_CiscoNAM_826_agent.config

Sample output: 

LocalSnmpEngID =                  ; Local Snmp Engine ID

interCollectionInterval=60        ; time between collections in seconds

ifErrorResendOneByOne = true      ; If resend get request one by one

numOfActorThread = 8              ; set number of threads in thread pool

numOfSNMPEngine = 24              ; num SNMP engine

traceLevel = 2                    ; trace level for the trace file

maxLoadPerAgent = 24              ; the maximum number of NAMs supported by an a

gent

alarmPollInterval = 60            ; Poll interval in seconds for detecting alarm

 entry changes

enableTimeMark = true             ; Enable data collection by timemark feature

hostBufferSize = 1000         ; Host ID and Address buffer size, range 1000-9000

maxBulkResponseSize = 70000   ; Maximum number of entries in a GETBULK response

Modify the traceLevel parameter and re-start PVM to have the changes take effect.

Trace File Naming and Size Limits

Each time an adapter or server component is started, a trace file is created. The trace files are named for each individual component and use a .trace extension. For example, the ART adapter trace would be named sp_snmp_ciscoART.trace. The following files may be found in the /[CiscoPVM Installation directory]/server/tmp directory. The default location is /opt/CSCOpvm/server/tmp.

Sample output: 

-rw-r--r--  1 pvmadm dba     1254 Sep 25 12:20 sp_deviceImport.out

-rw-r--r--  1 pvmadm dba    12240 Sep 25 12:20 sp_deviceImport.trace

-rw-r--r--  1 pvmadm dba      318 Sep 25 12:20 sp_hostSync.out

-rw-r--r--  1 pvmadm dba     2693 Sep 25 12:20 sp_hostSync.trace

-rw-r--r--  1 pvmadm dba      572 Sep 25 12:20 sp_snmp_ciscoART.out

-rw-r--r--  1 pvmadm dba      756 Sep 25 12:20 sp_snmp_ciscoART.trace

-rw-r--r--  1 pvmadm dba     1006 Sep 24 18:46 sp_snmp_ciscoNAM_826_0.trace

-rw-r--r--  1 pvmadm dba  1520707 Sep 25 11:55 sp_snmp_ciscoNAM_826_1.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_826.out

-rw-r--r--  1 pvmadm dba     1004 Sep 24 18:46 sp_snmp_ciscoNAM_827_0.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_827.out

-rw-r--r--  1 pvmadm dba     1004 Sep 24 18:46 sp_snmp_ciscoNAM_828_0.trace

-rw-r--r--  1 pvmadm dba      913 Sep 24 17:46 sp_snmp_ciscoNAM_828_1.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_828.out

-rw-r--r--  1 pvmadm dba      850 Sep 24 17:46 sp_snmp_ciscoSwitch_823_0.trace

-rw-r--r--  1 pvmadm dba     1036 Sep 25 12:20 sp_snmp_ciscoSwitch_823.out

-rw-r--r--  1 pvmadm dba        0 Sep 20 12:54 sp_snmp_ciscoSwitch_824_0.trace

-rw-r--r--  1 pvmadm dba     1014 Sep 25 12:20 sp_snmp_ciscoSwitch_824.out

-rw-r--r--  1 pvmadm dba      680 Sep 24 17:46 sp_snmp_ciscoSwitch_825_0.trace

-rw-r--r--  1 pvmadm dba     1014 Sep 25 12:20 sp_snmp_ciscoSwitch_825.out

-rw-r--r--  1 pvmadm dba      504 Sep 25 12:20 sp_thresholdMonitor.out

-rw-r--r--  1 pvmadm dba     1032 Sep 25 12:20 sp_thresholdMonitor.trace

-rw-r--r--  1 pvmadm dba     5836 Sep 25 12:22 SystemHealthMonitor.out

Displaying Trace Files

Cisco PVM trace files can be viewed in a user-friendly format using the sp_tracereader tool. The tool provides information on its usage when executed with the -h (Help) option.

cd [install directory]/server/bin

./sp_tracereader -h

Sample output:

Usage: 

sp_tracereader [-l <level>] [-c <category in hex>] [-o <objname>]

                        [-m <options> [-d <delimit string>]]

The following example prints only trace records of level fatal or error

sp_tracereader -l 2

sp_tracereader -m tpm -d " <--> "

Options can be any characters of thplcfonm 

t - time, h - host, p - port, l - level, c - category, f - filename

o - source class, n - line number, m - message

-d prints the info separated by delimit string

This example prints only trace records with component (object) name sp_eventlog 

sp_tracereader -o sp_eventlog

To view the static content of a trace file using the tracereader tool, execute the following command from the /opt/CSCOpvm/server/tmp directory:

cat <trace file name> | sp_tracereader | more

Sample output: 

<------------- < 1>Fri Sep 21 18:34:28 2007 ----------->

 hostname :  safi.trendium.com

 port     :  0

 level    :  2

 category :  1

 filename :  DeviceActor.cpp

 line no  :  936

 obj name :  sp_snmp_ciscoSwitch

 trace msg:  MaxPerPduVarSize is not set in configuration file, default value 5

is used

 <------------- < 2>Mon Sep 24 17:26:04 2007 ----------->

 hostname :  safi.trendium.com

 port     :  0

 level    :  2

 category :  1

 filename :  DeviceActor.cpp

 line no  :  936

 obj name :  sp_snmp_ciscoSwitch

 trace msg:  MaxPerPduVarSize is not set in configuration file, default value 5 is used

To view the changing contents of a trace file as PVM is processing, execute the following command from the /opt/CSCOpvm/server/tmp directory:

tail -f <trace file name> | sp_tracereader

Using PVM Tracing To Troubleshoot

To utilize the tracing functions to troubleshoot application issues, you must:

a. Identify the appropriate trace file(s).

b. Change the trace level to verbose.

c. Restart the component.

d. Review the trace content for information which will aid in troubleshooting.

Choosing the Correct Trace File

Using Table 2-2, select the component/area to monitor. All trace files are located in the [installation directory]/ server/tmp directory.

cd /opt/CSCOpvm/server/tmp

ls -l

e.

Table 2-2 Troubleshooting Using PVM Tracing

Component
Trace File Name
Used for Troubleshooting

NAM Adapters

sp_snmp_ciscoNAM_826_0.trace

sp_snmp_ciscoNAM_827_0.trace

sp_snmp_ciscoNAM_828_0.trace

sp_snmp_ciscoNAM_828_1.trace

NAM adapter collection

ART Adapter

sp_snmp_ciscoART.trace

ART collection

Switch Adapter

sp_snmp_ciscoSwitch_823_0.trace

sp_snmp_ciscoSwitch_824_0.trace

sp_snmp_ciscoSwitch_825_0.trace

Switch adapter collection

Threshold Monitor

sp_thresholdMonitor.trace

Evaluation of thresholds, raising alerts


Sample output: 

-rw-r--r--  1 pvmadm dba     1254 Sep 25 12:20 sp_deviceImport.out

-rw-r--r--  1 pvmadm dba    12240 Sep 25 12:20 sp_deviceImport.trace

-rw-r--r--  1 pvmadm dba      318 Sep 25 12:20 sp_hostSync.out

-rw-r--r--  1 pvmadm dba     2693 Sep 25 12:20 sp_hostSync.trace

-rw-r--r--  1 pvmadm dba      572 Sep 25 12:20 sp_snmp_ciscoART.out

-rw-r--r--  1 pvmadm dba      756 Sep 25 12:20 sp_snmp_ciscoART.trace

-rw-r--r--  1 pvmadm dba     1006 Sep 24 18:46 sp_snmp_ciscoNAM_826_0.trace

-rw-r--r--  1 pvmadm dba  1520707 Sep 25 11:55 sp_snmp_ciscoNAM_826_1.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_826.out

-rw-r--r--  1 pvmadm dba     1004 Sep 24 18:46 sp_snmp_ciscoNAM_827_0.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_827.out

-rw-r--r--  1 pvmadm dba     1004 Sep 24 18:46 sp_snmp_ciscoNAM_828_0.trace

-rw-r--r--  1 pvmadm dba      913 Sep 24 17:46 sp_snmp_ciscoNAM_828_1.trace

-rw-r--r--  1 pvmadm dba      926 Sep 25 12:19 sp_snmp_ciscoNAM_828.out

-rw-r--r--  1 pvmadm dba      850 Sep 24 17:46 sp_snmp_ciscoSwitch_823_0.trace

-rw-r--r--  1 pvmadm dba     1036 Sep 25 12:20 sp_snmp_ciscoSwitch_823.out

-rw-r--r--  1 pvmadm dba        0 Sep 20 12:54 sp_snmp_ciscoSwitch_824_0.trace

-rw-r--r--  1 pvmadm dba     1014 Sep 25 12:20 sp_snmp_ciscoSwitch_824.out

-rw-r--r--  1 pvmadm dba      680 Sep 24 17:46 sp_snmp_ciscoSwitch_825_0.trace

-rw-r--r--  1 pvmadm dba     1014 Sep 25 12:20 sp_snmp_ciscoSwitch_825.out

-rw-r--r--  1 pvmadm dba      504 Sep 25 12:20 sp_thresholdMonitor.out

-rw-r--r--  1 pvmadm dba     1032 Sep 25 12:20 sp_thresholdMonitor.trace

-rw-r--r--  1 pvmadm dba     5836 Sep 25 12:22 SystemHealthMonitor.out

Setting the Trace Level To Include Maximum Information

Set the trace level to 10 to include maximum information.

cd [install directory]/server/etc/agentconfig

vi <trace file name>

Sample output: 

LocalSnmpEngID =                  ; Local Snmp Engine ID

interCollectionInterval=60        ; time between collections in seconds

ifErrorResendOneByOne = true      ; If resend get request one by one

numOfActorThread = 8              ; set number of threads in thread pool

numOfSNMPEngine = 24              ; num SNMP engine

traceLevel = 2                    ; trace level for the trace file

maxLoadPerAgent = 24              ; the maximum number of NAMs supported by an agent

alarmPollInterval = 60            ; Poll interval in seconds for detecting

alarm entry changes

enableTimeMark = true             ; Enable data collection by timemark feature

hostBufferSize = 1000         ; Host ID and Address buffer size, range 1000-9000

maxBulkResponseSize = 70000   ; Maximum number of entries in a GETBULK response

Set the traceLevel parameter to 10 and save the configuration file.

Viewing the Trace File Contents

Re-start the Cisco PVM application and allow the components to run for a few minutes in order to collect some messages in the trace file.