Configuring and Viewing System Logs


Configuring and Viewing System Logs
 
There are five types of logs that can be configured and viewed on the system:
 
Important: Not all Event Logs can be configured on all products. It is dependent on the hardware platform and licenses in use.
 
Event: Event logging can be used to determine system status and capture important information pertaining to protocols and tasks in use by the system. This is a global function in that once it is configured, it will be applied to all contexts, sessions, and processes.
Trace: Trace logging can be used to quickly isolate issues that may arise for a particular connected subscriber session. Traces can be taken for a specific call identification (callid) number, IP address, mobile station identification (MSID) number, or username.
Active: Active logs are event logs that are operator configurable on a CLI instance-by-CLI instance basis (i.e. active logs configured by an administrative user in one CLI instance cannot be viewed by an administrative user in a different CLI instance). Each active log can be configured with filter and display properties that are independent of those configured globally for the system. Active logs are displayed in real time as they are generated.
Monitor: Monitor logging records all activity associated with a particular session. This functionality is available in order to comply with law enforcement agency requirements for monitoring capabilities of particular subscribers. Moniors can be performed based on a subscriber’s MSID or username.
Crash: Crash logging stores useful information pertaining to system software crashes that may be useful in determining the cause of the crash.
This chapter provides information and instructions for configuring parameters related to the various types of logging and for viewing their content.
Configuring Event Logging Parameters
The system can be configured to generate logs based on user-defined filters. The filters specify the facilities (system tasks or protocols) that the system is to monitor and severity levels at which to trigger the generation of the event log.
 
Event logs are stored in system memory and can be viewed via the CLI. There are two memory buffers that store event logging information. The first buffer stores the active log information. The second buffer stores inactive logging information. The inactive buffer is used as a temporary repository to allow you to view logs without having data be overwritten. Logs are copied to the inactive buffer only through manual intervention.
Each buffer can store up to 50,000 events. Once these buffers reach their capacity, the oldest information is removed to make room for the newest.
To prevent the loss of log data, the system can be configured to transmit logs to a syslog server over a network interface.
Configuring Event Log Filters
Follow the example below to configure run time event logging parameters for the system:
 
configure
  logging filter runtime facility <facility> level <report_level>
  logging display
  end
Notes:
Save the configuration as described in the Verifying and Saving Your Configuration chapter.
Configuring Syslog Servers
Information generated by the run time event logging filters can be transmitted to a syslog server for permanent storage.
 
Important: The data transmitted to the syslog server is meant to be used for informational purposes.Therefore, functions such as billing and performance monitoring should not be based on syslogs.
Important: Although the system provides the flexibility to configure syslog servers on a context-by-context basis, it is recommended that all servers be configured in the local context in order to isolate the log traffic from the network traffic.
Use the following example to configure syslog servers:
configure
  context local
     logging syslog <ip_address>
     end
Notes:
A number of keyword options/variables are available for the logging syslog command. Refer to the Command Line Interface Reference for more information.
Save the configuration as described in the Verifying and Saving Your Configuration chapter.
Configuring Trace Logging
Trace logging is useful for quickly resolving issues for specific sessions that are currently active. They are temporary filters that are generated based on a qualifier that is independent of the global event log filter configured using the logging filter command. Like event logs, however, the information generated by the logs is stored in the active memory buffer.
 
All debug level events associated with the selected call are stored.
Important: Trace logs are intrusive to the processing of the session. Therefore, they should be implemented for debug purposes only.
Use the following example to configure trace logs:
logging trace { callid <call_id> | ipaddr <ip_address> | msid <ms_id> | name <username> }
Once all of the necessary information has been gathered, the trace log can be deleted by entering the following command:
no logging trace { callid <call_id> | ipaddr <ip_address> | msid <ms_id> | name <username> }
Configuring Active Logs
Active logs are event logs that are operator configurable on a CLI instance-by-CLI instance basis (i.e. active logs configured by an administrative user in one CLI instance are not displayed to an administrative user in a different CLI instance). Each active log can be configured with filter and display properties that are independent of those configured globally for the system. Active logs are displayed in real time as they are generated.
 
Active logs are not written to the active memory buffer by default. To write active logs to the active memory buffer, in the config mode, use the following command:
logging runtime buffer store all-events
When active logs are written to the active memory buffer, they are available to all users in all CLI instances.
Use the following example to configure active logging:
logging filter active facility <facility> level <report_level>
logging active
Notes:
Refer to the logging filter command in the Command Line Interface Reference to view a list of the supported logging facilities.
A number of keyword options/variables are available for the logging active command. Refer to the Command Line Interface Reference for more information.
Once all of the necessary information has been gathered, the Active log display can be stopped by entering the following command:
no logging active
Configuring Monitor Logs
Monitor logging records all activity associated with all of a particular subscriber’s sessions. This functionality is available in order to comply with law enforcement agency requirements for monitoring capabilities of particular subscribers.
 
Moniors can be performed based on a subscriber’s MSID or username and are only intended to be used for finite periods of time as dictated by the law enforcement agency. Therefore, they should be terminated immediately after the required monitoring period.
This section provides instructions for enabling and disabling monitor logs.
Enabling Monitor Logs
Use the following example to configure monitor log targets:
 
configure
  logging monitor { msid <id> | username <name> | ip_addr | IPv6_addr }
  end
Notes:
Disabling Monitor Logs
Use the following example to disable monitor logs:
 
configure
  no logging monitor { msid <id> | username <name> }
  end
Viewing Logging Configuration and Statistics
Logging configuration and statistics can be verified by entering the following command from the Exec mode:
 
show logging [ active | verbose ]
When no keyword is specified, the global filter configuration is displayed as well as information about any other type of logging that is enabled.
The following table provides information] and descriptions of the statistics that are displayed when the verbose keyword is used.
 
Viewing Event Logs Using the CLI
 
Event logs generated by the system can be viewed in one of the following ways:
 
From the syslog server: If the system is configured to send logs to a syslog server, the logs can be viewed directly on the syslog server.
From the system CLI: Logs stored in the system memory buffers can be viewed directly from the CLI.
From the console port: By default, the system automatically displays events over the console interface to a terminal provided that there is no CLI session active.
This section provides instructions for viewing event logs using the CLI. These instructions assume that you are at the root prompt for the Exec mode.
Step 1
When the active log memory buffer is copied to the inactive log memory buffer existing information in the inactive log memory buffer is deleted.
Both active and inactive event log memory buffers can be viewed using the CLI. However, it is preferable to view the inactive log in order to prevent any data from being over-written. The information from the active log buffer can be copied to the inactive log buffer by entering the following command:
 
logs checkpoint
Step 2
 
show logs
Important: A number of optional keywords/variables are available for the show logs command. Refer to the Command Line Interface Reference for more information.
Configuring and Viewing Software Crash Logging Parameters
In the unlikely even of a software crash, the system stores information that could be useful in determining the reason for the crash. This information can be maintained in system memory or it can be transferred and stored on a network server.
 
The system supports the generation of the following two types of logs:
 
Crash log: Crash logs record all possible information pertaining to a software crash. Due to their size, they can not be stored in system memory. Therefore, these logs are only generated if the system is configured with a Universal Resource Locator (URL) pointing to a local device or a network server where the log can be stored.
Abridged crash log: These logs are automatically generated when a software crash occurs and are stored in system memory. The abridged crash log contains a subset of the possible information that could be generated with a crash log. These logs are generated even if a full crash log is generated and can be viewed using the CLI.
Configuring Software Crash Log Destinations
The system can be configured to store software crash log information to any of the following locations:
 
 
CompactFlash™: Installed on the SPC/SMC
PCMCIA Flash Card: Installed in either the PCMCIA1 or PCMCIA2 slots on the SPC or in the PCMCIA1 slot on the SMC
Network Server: Any workstation or server on the network that the system can access using the Trivial File Transfer Protocol (TFTP), the File Transfer Protocol (FTP), the Secure File Transfer Protocol (SFTP), or the Hyper-Text Transfer Protocol (HTTP); this is recommended for large network deployments in which multiple systems require the same configuration
Crash logs are stored with unique names as they occur to the specified location. The name format is crash-card-cpu-time-core. Where card is the card number, cpu is the number of the CPU on the card, and time is the POSIX timestamp in hexadecimal notation.
Use the following example to configure a software crash log destination:
configure
  crash enable [ encrypted ] url <crash_url>
  end
Notes:
Save the configuration as described in the Verifying and Saving Your Configuration chapter.
 
Viewing Abridged Crash Logs Using the CLI
 
Abridged crash logs are stored on the CompactFlash installed on the SPC/SMC. They are located in the /flash/crash/ directory with file names in the mc-slot-cpu-pid-xxxxxxxx format. Where slot is the card slot in the chassis, cpu is the number of the CPU on the card, pid is the process ID number, and xxxxxxxx is a UNIX date code in hexadecimal notation.
Follow the instructions in this section to view a list of software crashes that have occurred. These instructions assume that you are at the root prompt for the Exec mode.
Step 1
Important: The resulting output may not be the same for all platforms:
 
show crash list
A sample output is displayed below.
 
== ==== ======= =========== ========= ================
# Time Process Card/CPU/ SW HW_SER_NUM
PID VERSION Card
== ==== ======= =========== ========= ================
1 2003-Nov-01+11:04:24 kernel 13/00/NA 3.0(3665) PLX01020114/PLX06020362
The following table provides descriptions for the individual columns displayed in the output.
show crash list Command Output Descriptions
Step 3
 
show crash number <crash_number>
crash_number is the number of the crash for which you wish to view the log as displayed by the show crash list command.
The information contained in the abridged crash log is useful to help identify and diagnose any internal or external factors causing the software to crash. The following displays a sample of the output.
 
********************* CRASH #30 ***********************
Build: 4.0(5800)
Fatal Signal 11: Segmentation fault
PC: [ 0x484650c] strlen()
Faulty address: (nil)
Signal detail: address not mapped to object
Recent events (oldest first):
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x38f0498] xtcp_client_timer_tick()
[ 0x391c630] xtcp_wagg_tick()
[ 0x391c630] xtcp_wagg_tick()
[ 0x391c630] xtcp_wagg_tick()
[ 0x2c77cb0] snreactor_file_cb()
[ 0x2c77cb0] snreactor_file_cb()
[ 0x3932868] sn_epoll_run_events()
[ 0x3932868] sn_epoll_run_events()
[ 0x2c77cb0] snreactor_file_cb()
[ 0x3932868] sn_epoll_run_events()
Process: card=8 cpu=0 pid=917 argv0=orbs
Crash time: 2004-Jun-23+12:53:19
Recent errno: 11 Resource temporarily unavailable
Registers:
zr at v0 v1 a0 a1 a2 a3
00000000 109b20c4 00000000 00000000 00000000 00000000 01010101 80808080
t0 t1 t2 t3 t4 t5 t6 t7
00002050 109dbb1c 00000040 00000007 00000000 2abbe9b0 00000000 00000001
s0 s1 s2 s3 s4 s5 s6 s7
00000000 7fff6c58 7fff6f38 7fff74a8 7fff74a8 7fff6fc8 7fff7168 00000001
t8 t9 k0 k1 gp sp s8 ra
00000000 048464b0 000000f6 00000000 10c1f9f0 7fff6bc0 7fff72d8 048eca80
Stack: 5192 bytes dumped starting from 0x7fff6850
[ 0x484650c] strlen() sp=0x7fff6bc0
[ 0x48eca80] __cxa_bad_typeid() sp=0x7fff6be0
[0x7fff6f40] <trampoline/gdb/stack>() sp=0x7fff6c10
*******************************************************
Saving Log Files
Log files can be saved to a file in a local or remote location specified by a URL. Use the following exec mode command to save log files:
 
save logs { <url> } [ active ] [ inactive ] [ callid <call_id> ] [ event-verbosity <evt_verboseness>] [ facility <facility> ] [ level <severity_level> ] [ pdu-data <pdu_format> ] [ pdu-verbosity <pdu_verboseness> ] [ since <from_date_time> [ until <to_date_time> ] ] [ | { grep <grep_options> | more } ]
For detailed information on the save logs command, refer to the Exec Mode Commands chapter of the Command Line Interface Reference.
Event ID Overview
Important: Not all event IDs are used on all platforms. It depends on the platform type and the license(s) running.
Identification numbers (IDs) are used to reference events as they occur when logging is enabled on the system. As described previously in this chapter, logs are collected on a per facility basis. Each facility possesses its own range of event IDs as indicated in the following table.
System Event Facilities and ID Ranges
Event Severities
The system provides the flexibility to configure the level of information that is displayed when logging is enabled. The following levels are supported:
 
critical: Logs only those events indicating a serious error has occurred that is causing the system tor a system component to cease functioning. This is the highest severity level.
error: Logs events that indicate an error has occurred that is causing the system or a system component to operate in a degraded state. This level also logs events with a higher severity level.
warning: Logs events that may indicate a potential problem. This level also logs events with a higher severity level.
unusual: Logs events that are very unusual and may need to be investigated. This level also logs events with a higher severity level.
info: Logs informational events and events with a higher severity level.
trace: Logs events useful for tracing and events with a higher severity level.
debug: Logs all events regardless of the severity.
Each of the above levels correspond to the “severity” level of the event ID. Therefore, only those event IDs with a “severity” level equal to the logging level are displayed.
Understanding Event ID Information in Logged Output
This section explains the event information that is displayed when logging is enabled.
 
The following displays a sample output for an event that was logged.
2006-Jun-23+12:18:41.993 [cli 30005 info] [8/0/609 <cli:8000609> _commands_cli.c:1290] [software internal system] CLI session ended for Security Administrator admin on device /dev/pts/2
The following table describes the elements of contained in the sample output.
Event Element Descriptions
 
 

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883