> Configuring and Viewing System Logs
> This chapter describes how to configure parameters related to the various types of logging and how to viewing their content. It includes the following sections:

> • | Configuring Event Logging Parameters

> • | Configuring Trace Logging

> • | Configuring Active Logs

> • | Configuring Monitor Logs

> • | Viewing Logging Configuration and Statistics

> • | Viewing Event Logs Using the CLI

> • | Configuring and Viewing Software Crash Logging Parameters

> • | Saving Log Files

> • | Event ID Overview

> System Log Types
> There are five types of logs that can be configured and viewed on the system:
> Important: Not all Event Logs can be configured on all products. Configurability depends on the hardware platform and licenses in use.

> • | Event: Event logging can be used to determine system status and capture important information pertaining to protocols and tasks in use by the system. This is a global function that will be applied to all contexts, sessions, and processes.

> • | Trace: Trace logging can be used to quickly isolate issues that may arise for a particular connected subscriber session. Traces can be taken for a specific call identification (callid) number, IP address, mobile station identification (MSID) number, or username.

> • | Active: Active logs are operator configurable on a CLI instance-by-CLI instance basis. Active logs configured by an administrative user in one CLI instance cannot be viewed by an administrative user in a different CLI instance. Each active log can be configured with filter and display properties that are independent of those configured globally for the system. Active logs are displayed in real time as events are generated.

> • | Monitor: Monitor logging records all activity associated with a particular session. This functionality is available in order to comply with law enforcement agency requirements for monitoring capabilities of particular subscribers. Monitors can be performed based on a subscriber’s MSID or username.

> • | Crash: Crash logging stores useful information pertaining to system software crashes. They are useful in determining the cause of the crash.

> Configuring Event Logging Parameters
> The system can be configured to generate logs based on user-defined filters. The filters specify the facilities (system tasks or protocols) that the system is to monitor and severity levels at which to trigger the generation of the event entries.
> Event logs are stored in system memory and can be viewed via the CLI. There are two memory buffers that store event logging information. The first buffer stores the active log information. The second buffer stores inactive logging information. The inactive buffer is used as a temporary repository to allow you to view logs without having data be overwritten. Logs are copied to the inactive buffer only through manual intervention.
> Each buffer can store up to 50,000 events. Once these buffers reach their capacity, the oldest information is removed to make room for the newest.
> To prevent the loss of log data, the system can be configured to transmit logs to a syslog server over a network interface.
> Configuring Event Log Filters
> Follow the example below to configure run time event logging parameters for the system:
> configure
> logging filter runtime facility <facility> level <report_level>
> logging display
> end
> Notes:

> • | Configure the logging filter that determines which system facilities should be logged and at what levels.

> • | Repeat for every facility that you would like to log.

> • | Optional: Configure event ID restrictions by adding the logging disable eventid command. The system provides the ability to restrict the sending of a specific event ID or a range of event IDs to minimize the amount of data logged to that which is most useful. Repeat to disable logging for additional event IDs or event ID ranges.

> Save the configuration as described in the Verifying and Saving Your Configuration chapter.
> Configuring syslog Servers
> Information generated by the run time event logging filters can be transmitted to a syslog server for permanent storage.
> Important: The data transmitted to the syslog server is meant to be used for informational purposes. Functions such as billing and performance monitoring should not be based on syslogs.
> Important: Although the system provides the flexibility to configure syslog servers on a context-by-context basis, it is recommended that all servers be configured in the local context in order to isolate the log traffic from the network traffic.
> Use the following example to configure syslog servers:
> configure
> context local
> logging syslog <ip_address>
> end
> Notes:

> • | A number of keyword options/variables are available for the logging syslog command. Refer to the Context Configuration Mode Commands chapter in the Command Line Interface Reference for more information.

> • | Repeat as needed to configure additional syslog servers. There is no limit to the number of syslog servers that can be configured.

> Save the configuration as described in the Verifying and Saving Your Configuration chapter.
> Configuring Trace Logging
> Trace logging is useful for quickly resolving issues for specific sessions that are currently active. They are temporary filters that are generated based on a qualifier that is independent of the global event log filter configured using the logging filter command in the Exec mode. Like event logs, however, the information generated by the logs is stored in the active memory buffer.
> All debug level events associated with the selected call are stored.
> Important: Trace logs impact session processing. They should be implemented for debug purposes only.
> Use the following example to configure trace logs in the Exec mode:
> logging trace { callid call_id | ipaddr ip_address | msid ms_id | username username }
> Once all of the necessary information has been gathered, the trace log can be deleted by entering the following command:
> no logging trace { callid call_id | ipaddr ip_address | msid ms_id | username username }
> Configuring Active Logs
> Active logs are event logs that are operator configurable on a CLI instance-by-CLI instance basis. Active logs configured by an administrative user in one CLI instance are not displayed to an administrative user in a different CLI instance. Each active log can be configured with filter and display properties that are independent of those configured globally for the system. Active logs are displayed in real time as they are generated.
> Active logs are not written to the active memory buffer by default. To write active logs to the active memory buffer execute the following command in the Global Configuration mode:
> logging runtime buffer store all-events
> When active logs are written to the active memory buffer, they are available to all users in all CLI instances.
> Use the following example to configure active logging (Global Configuration mode):
> logging filter active facility facility level report_level
> logging active
> Notes:

> • | Refer to the logging filter command in the Global Configuration Mode Commands chapter of the Command Line Interface Reference to view a list of the supported logging facilities.

> • | A number of keyword options/variables are available for the logging active command. Refer to the Exec Mode Commands (D-S) chapter of the Command Line Interface Reference for more information.

> Once all of the necessary information has been gathered, the Active log display can be stopped by entering the following command in the Exec mode:
> no logging active
> Configuring Monitor Logs
> Monitor logging records all activity associated with all of a particular subscriber’s sessions. This functionality is available in compliance with law enforcement agency requirements for monitoring capabilities of particular subscribers.
> Monitors can be performed based on a subscriber’s MSID or username, and are only intended to be used for finite periods of time as dictated by the law enforcement agency. Therefore, they should be terminated immediately after the required monitoring period.
> This section provides instructions for enabling and disabling monitor logs.
> Enabling Monitor Logs
> Use the following example to configure monitor log targets:
> configure
> logging monitor { msid <id> | username <name> | ip_addr | IPv6_addr }
> end
> Repeat to configure additional monitor log targets.
> Disabling Monitor Logs
> Use the following example to disable monitor logs:
> configure
> no logging monitor { msid <id> | username <name> }
> end
> Viewing Logging Configuration and Statistics
> Logging configuration and statistics can be verified by entering the following command from the Exec mode:
> show logging [ active | verbose ]
> When no keyword is specified, the global filter configuration is displayed as well as information about any other type of logging that is enabled.
> The following table provides information] and descriptions of the statistics that are displayed when the verbose keyword is used.

> Field | Description
> General Logging Statistics
> Total events received | Displays the total number of events generated by the system.
> Number of applications receiving events | Displays the number of applications receiving the events.
> Logging Source Statistics
> Event sequence ids by process | Displays a list of system processes that have generated events and the reference identification number of the event that was generated.
> Msg backlog stat with total cnt | Displays the number of event messages that have been back logged in comparison to the total number of events generated.
> LS L2 filter drop rate | Displays the percentage of logging source (LS) layer 2 (L2) event drops.
> Abnormal Log Source Statistics | Displays abnormal logging source (LS) statistics, if any.
> Runtime Logging Buffer Statistics
> Active buffer | Displays the number of events currently logged in the active memory buffer as well as a date/time timestamp for the oldest and most recent entries in the buffer.
> Inactive buffer | Displays the number of events currently logged in the inactive memory buffer.

> ‌
> Viewing Event Logs Using the CLI
> Event logs generated by the system can be viewed in one of the following ways:

> • | From the syslog server: If the system is configured to send logs to a syslog server, the logs can be viewed directly on the syslog server.

> • | From the system CLI: Logs stored in the system memory buffers can be viewed directly from the CLI.

> • | From the console port: By default, the system automatically displays events over the console interface to a terminal provided that there is no CLI session active.

> This section provides instructions for viewing event logs using the CLI. These instructions assume that you are at the root prompt for the Exec mode.

> Step 1 | Copy the active log memory buffer to the inactive log memory buffer.

> When the active log memory buffer is copied to the inactive log memory buffer existing information in the inactive log memory buffer is deleted.
> Both active and inactive event log memory buffers can be viewed using the CLI. However, it is preferable to view the inactive log in order to prevent any data from being over-written. The information from the active log buffer can be copied to the inactive log buffer by entering the following command:
> logs checkpoint

> Step 2 | View the logs by entering the following command:

> show logs
> Important: A number of optional keywords/variables are available for the show logs command. Refer to the Exec Mode Show Commands (H-L) chapter of the Command Line Interface Reference for more information.
> Configuring and Viewing Software Crash Logging Parameters
> In the unlikely even of a software crash, the system stores information that could be useful in determining the reason for the crash. This information can be maintained in system memory or it can be transferred and stored on a network server.
> The system supports the generation of the following two types of logs:

> • | Crash log: Crash logs record all possible information pertaining to a software crash. Due to their size, they can not be stored in system memory. Therefore, these logs are only generated if the system is configured with a Universal Resource Locator (URL) pointing to a local device or a network server where the log can be stored.

> • | Abridged crash log: These logs are automatically generated when a software crash occurs and are stored in system memory. The abridged crash log contains a subset of the possible information that could be generated with a crash log. These logs are generated even if a full crash log is generated and can be viewed using the CLI.

> Configuring Software Crash Log Destinations
> The system can be configured to store software crash log information to any of the following locations:

> • | CompactFlash™: Installed on the SMC

> • | PCMCIA Flash Card: Installed in the PCMCIA1 slot on the SMC

> • | Network Server: Any workstation or server on the network that the system can access using the Trivial File Transfer Protocol (TFTP), the File Transfer Protocol (FTP), the Secure File Transfer Protocol (SFTP), or the Hyper-Text Transfer Protocol (HTTP); this is recommended for large network deployments in which multiple systems require the same configuration

> Crash logs are written with unique names as they occur to the specified location. The name format is crash-card-cpu-time-core. Where card is the card slot, cpu is the number of the CPU on the card, and time is the POSIX timestamp in hexadecimal notation.
> Use the following example to configure a software crash log destination in the Global Configuration mode:
> configure
> crash enable [ encrypted ] url <crash_url>
> end
> Notes:

> • | Keyword and variable options are available for the crash enable command. Refer to the Global Configuration Mode Commands chapter of the Command Line Interface Reference for more information.

> • | Repeat to configure additional software crash log destinations. There is no limit to the number of destinations that can be configured.

> Save the configuration as described in the Verifying and Saving Your Configuration chapter.
> Viewing Abridged Crash Logs Using the CLI
> Abridged crash logs are stored on flash memory on the SMC. They are located in the /flash/crash/ directory with file names in the mc-slot-cpu-pid-xxxxxxxx format. Where slot is the card slot in the chassis, cpu is the number of the CPU on the card, pid is the process ID number, and xxxxxxxx is a UNIX date code in hexadecimal notation.
> Follow the instructions in this section to view a list of software crashes that have occurred. These instructions assume that you are at the root prompt for the Exec mode.

> Step 1 | View a listing of any software crashes that may have occurred on the system by entering the following command

> Important: The resulting output may not be the same for all platforms:
> show crash list
> A sample output is displayed below.
> == ==== ======= =========== ========= ================
> # Time Process Card/CPU/ SW HW_SER_NUM
> PID VERSION Card
> == ==== ======= =========== ========= ================
> 1 2011-Nov-01+11:04:24 kernel 13/00/NA 3.0(3665) PLX01020114/PLX06020362
> The following table provides descriptions for the individual columns displayed in the output.

> Column Title | Description
> # | Displays an internal reference number for this software crash in the log.
> Time | Indicates the date and time that the software crash occurred.
> Process | Indicates the software task that experienced the crash.
> Card | Indicates the card on which the software task was running.
> CPU | Indicates the CPU on which the software task was running.
> PID | Indicates the process identification (PID) number of the software task that experienced the crash.
> SW_Version | Indicates the version of software that experienced the crash.
> HW_SER_NUM Card | The hardware serial numbers of the card where the crash occurred.

> ‌

> Step 2 | View the abridged crash log by entering the following command:

> show crash number crash_number
> crash_number is the number of the crash for which you wish to view the log as displayed by the show crash list command. The information contained in the abridged crash log is useful to help identify and diagnose any internal or external factors causing the software to crash. The following is a sample of the output.
> ********************* CRASH #30 ***********************
> Build: 4.0(5800)
> Fatal Signal 11: Segmentation fault
> PC: [ 0x484650c] strlen()
> Faulty address: (nil)
> Signal detail: address not mapped to object
> Recent events (oldest first):
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x38f0498] xtcp_client_timer_tick()
> [ 0x391c630] xtcp_wagg_tick()
> [ 0x391c630] xtcp_wagg_tick()
> [ 0x391c630] xtcp_wagg_tick()
> [ 0x2c77cb0] snreactor_file_cb()
> [ 0x2c77cb0] snreactor_file_cb()
> [ 0x3932868] sn_epoll_run_events()
> [ 0x3932868] sn_epoll_run_events()
> [ 0x2c77cb0] snreactor_file_cb()
> [ 0x3932868] sn_epoll_run_events()
> Process: card=8 cpu=0 pid=917 argv0=orbs
> Crash time: 2011-Dec-23+12:53:19
> Recent errno: 11 Resource temporarily unavailable
> Registers:
> zr at v0 v1 a0 a1 a2 a3
> 00000000 109b20c4 00000000 00000000 00000000 00000000 01010101 80808080
> t0 t1 t2 t3 t4 t5 t6 t7
> 00002050 109dbb1c 00000040 00000007 00000000 2abbe9b0 00000000 00000001
> s0 s1 s2 s3 s4 s5 s6 s7
> 00000000 7fff6c58 7fff6f38 7fff74a8 7fff74a8 7fff6fc8 7fff7168 00000001
> t8 t9 k0 k1 gp sp s8 ra
> 00000000 048464b0 000000f6 00000000 10c1f9f0 7fff6bc0 7fff72d8 048eca80
> Stack: 5192 bytes dumped starting from 0x7fff6850
> [ 0x484650c] strlen() sp=0x7fff6bc0
> [ 0x48eca80] __cxa_bad_typeid() sp=0x7fff6be0
> [0x7fff6f40] <trampoline/gdb/stack>() sp=0x7fff6c10
> *******************************************************
> Saving Log Files
> Log files can be saved to a file in a local or remote location specified by a URL. Use the following Exec mode command to save log files:
> save logs { url } [active] ] [inactive] [callid call_id] [event-verbosity evt_verboseness] [facility facility] [level severity_level] [pdu-data pdu_format] [pdu-verbosity pdu_verboseness] [since from_date_time[until to_date_time] ] [ | { grep grep_options | more } ]
> For detailed information on the save logs command, refer to the Exec Mode Commands chapter of the Command Line Interface Reference.
> Event ID Overview
> Important: The use of event IDs depends on the platform type and the licenses running on the platform.
> Identification numbers (IDs) are used to reference events as they occur when logging is enabled on the system. As described previously in this chapter, logs are collected on a per facility basis. Each facility possesses its own range of event IDs as indicated in the following table.
> System Event Facilities and ID Ranges

> Facility | Event ID Range
> A10 Protocol Facility Events | 28000-28999
> A11 Protocol Facility Events | 29000-29999
> A11 Manager Facility Events | 9000-9999
> AAA Client Facility Events | 6000-6999
> Active Charging Service (ACS) Controller Events | 90000-90999
> Active Charging Service (ACS) Manager Events | 91000-91999
> Active Charging Service Session Manager (ACSMGR) Signalling Interface Logging Facility | 97000-97099
> Alarm Controller Facility Events | 65000-65999
> ASN Gateway Manager Facility Events | 100000-100499
> ASN Paging/Location-Registry Manager Facility Events | 100500-100999
> Broadcast/Multicast Service (BCMCS) Facility Events | 109000-109999
> BSSAP+ Service Facilities | 131000- 131199
> Call State Control Function (CSCF) | 105000-108999
> CSCF CORE module | 105000 - 105999
> SIPAPP (stack + Proxyapp + B2BUA + Stubs) module | 106000 - 106499
> Proxy CSCF (P-CSCF) module | 106500 - 106549
> Serving CSCF (S-CSCF) module | 106550 - 106999
> CPS | 107000 - 107499
> CSCF Manager (CSCFMGR) | 101000-101999
> Card/Slot/Port (CSP) Facility Events | 7000-7999
> Command Line Interface Facility Events | 30000-30999
> Content Steering Service Facility Events | 77000-77499
> Data Path for IPSec Facility Events | 54000-54999
> Daughter Card Controller Facility Events | 62000-62999
> Daughter Card Manager Facility Events | 57000-57999
> DHCP Facility Events | 53000-53999
> Diameter Accounting Protocol Facility | 112000-112999
> Diameter Authentication Protocol Facility | 111000-111999
> Distributed Host Manager Facility Events | 83000-83999
> Driver Controller Facility Events | 39000-39999
> eGTP-C Facility Events | 141000-141999
> eGTP-U Facility Events | 142000-142999
> eGTP Manager Facility Events | 143000-143999
> Event Log Facility Events | 2000-2999
> Femto Network Gateway Facility Events | 149000-149999
> Firewall Facility | 96000-96999
> Foreign Agent Manager Facility Events | 33000-33999
> GCDR Facility Events | 66000-66999
> GPRS Application Facility (GPRS) | 115100-115399
> GTP-PRIME Protocol Facility Events | 52000-52999
> GTPC Protocol Facility Events | 47000-47999
> GTPC Signaling Demultiplexer Manager Facility Events | 46000-46999
> GTPP Storage Server GCDR Facility Events | 98000-98099
> GTPU Protocol Facility Events | 45000-45999
> H.248 Protocol Facility Events | 42000-42999
> High Availability Task Facility Events | 3000-3999
> Home Agent Manager Facility Events | 34000-34999
> IKEv2 Facility Events | 122000-122999
> IMS Authorization Service Library Facility Events | 98100-98999
> IMSI Manager Facility | 114000-114999
> IMS SH Library Events | 124000-124999
> IMSUE Logging Facility | 135000 -136999
> IP Access Control List (ACL) Log Facility Events | 21000-21999
> IP Address Resolution Protocol (ARP) Facility Events | 19000-19999
> IP Interface Facility Events | 18000-18999
> IP Pool Sharing Protocol (IPSP) Facility Events | 68000-68999
> IP Route Facility Events | 20000-20999
> IPSG Driver Facility | 128000-128999
> IPSec Protocol Facility Events | 55000-55999
> L2TP Control PDU Protocol Facility Events | 50000-50999
> L2TP Data PDU Protocol Facility Events | 49000-49999
> L2TP Demux Facility Events | 63000-63999
> L2TP Manager Facility Events | 48000-48999
> Lawful Intercept Log Facility Events | 69000-69999
> Link Manager Facility | 89500-89999
> MEGADIAM Session Manager service (MEGADIAM) Facility | 121200-121999
> MEGADIAM Manager (MEGADIAMMGR) Facility | 12100 - 121199
> MME App Facility Events | 147000-147999
> MME Demux Manager Facility Events | 154000-154999
> MME-HSS Facility Events | 138000-138999
> MME Miscellaneous Facility Events | 152600-152999
> Mobile Access Gateway Manager Facility Events | 137500-137999
> Mobile IPv6 Facility Events | 129000-129999
> Mobile IP Tunneled Data Facility Events | 27000-27999
> Mobile IP Protocol Facility Events | 26000-26999
> MSEG Application Facility mseg-gtpc Events | 172000-172199
> MSEG Application Facility mseg-gtpu Events | 172200-172299
> MSEG Application Facility mseg-app Events | 172300-172999
> MSEG Manager (MSEGMGR) Facility Events | 171000-171999
> Multicast Proxy Facility Events | 94000-94999
> Network Access Signaling Facility Events | 153000-153999
> Network Storage Events | 78000-78999
> NPU Control Facility Events | 16000-16999
> NPU Manager Facility Events | 17000-17999
> Object Request Broker (ORB) System Facility Events | 15000-15999
> PDG Facility Events | 152010-152999
> PDG TCP Demux Manager (pdgdmgr) Facility Events (this is a customer-specific facility) | 162400-162999
> PDN Gateway Facility Events | 139000-139999
> Point-To-Point Protocol Facility Events | 25000-25999
> P2P Facility | 146000-146999
> RADIUS Accounting Protocol Facility Events | 24000-24999
> RADIUS Authentication Protocol Facility Events | 23000-23999
> Recovery Control Task (RCT) Facility Events | 13000-13999
> Redirector Task (RDT) Facility Events | 67000-67999
> Resource Manager (RM) Facility Events | 14000-14999
> Robust Header Compression Protocol (ROHC) Facility Events | 103000-103999
> RSVP Protocol Facility Events | 93000-93999
> SCTP Protocol Facility | 87300-87499
> Service Redundancy Protocol (SRP) Facility Events | 84000-84999
> Serving Gateway Facility Events | 140000-140999
> Session Controller Facility Events | 8000-8999
> Session Initiation Protocol (SIP) Events | 59000-59999
> Session Manager Facility Events | 10000-12999
> SGTPC Manager Facility | 117000-117999
> Shared Configuration Task (SCT) Facility Events | 32000-32099
> Session Initiation Protocol (SIP) Call Distributor Facility Events | 86000-86999
> Simple Network Management Protocol (SNMP) Protocol Facility Events | 22000-22999
> SSL Facility Events (this is a customer-specific facility) | 156200-157199
> Static Rating Database Facility Events | 102000-102999
> Statistics Facility Events | 31000-31999
> Switch Fabric Task (SFT) Facility Events | 58000-58999
> System Facility Events | 1000-1999
> System Initiation Task (SIT) Main Facility Events | 4000-4999
> TACACS+ Protocol Facility Events | 37000-37999
> Threshold Facility Events | 61000-61999
> TTG Facility Events | 130000-130999
> UDR Facility Events | 79000-79999
> User-Data Facility Events | 51000-51999
> User L3 Tunnel Facility Events | 75000-75999
> Virtual Private Network Facility Events | 5000-5999
> WiMAX R6 Protocol (Signaling) Facility | 104000-104899

> ‌
> Event Severities
> The system provides the flexibility to configure the level of information that is displayed when logging is enabled. The following levels are supported:

> • | critical: Logs only those events indicating a serious error has occurred that is causing the system tor a system component to cease functioning. This is the highest severity level.

> • | error: Logs events that indicate an error has occurred that is causing the system or a system component to operate in a degraded state. This level also logs events with a higher severity level.

> • | warning: Logs events that may indicate a potential problem. This level also logs events with a higher severity level.

> • | unusual: Logs events that are very unusual and may need to be investigated. This level also logs events with a higher severity level.

> • | info: Logs informational events and events with a higher severity level.

> • | trace: Logs events useful for tracing and events with a higher severity level.

> • | debug: Logs all events regardless of the severity.

> Each of the above levels correspond to the “severity” level of the event ID. Therefore, only those event IDs with a “severity” level equal to the logging level are displayed.
> Understanding Event ID Information in Logged Output
> This section explains the event information that is displayed when logging is enabled.
> The following displays a sample output for an event that was logged.
> 2011-Dec-11+5:18:41.993 [cli 30005 info] [8/0/609 <cli:8000609> _commands_cli.c:1290] [software internal system] CLI session ended for Security Administrator admin on device /dev/pts/2
> The following table describes the elements of contained in the sample output.
> Event Element Descriptions

> Element | Description
> 2011-Dec-11+5:18:41.993 | Date/Timestamp indicating when the event was generated
> [cli 30005 info] | Information about the event including:
> • | The facility the event belongs to
> • | The event ID
> • | The event’s severity level
> In this example, the event belongs to the CLI facility, has an ID of 3005, and a severity level of “info”.
> [8/0/609 <cli:8000609> _commands_cli.c:1290] | Information about the specific CLI instance.
> [software internal system] | Indicates that the event was generated because of system operation.
> CLI session ended for Security Administrator admin on device /dev/pts/2 | The event’s details. Event details may, or may not include variables that are specific to the occurrence of the event.

> ‌

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883