Guest

Cisco IOS Software Releases 12.0 S

Onboard Failure Logging

  • Viewing Options

  • PDF (235.8 KB)
  • Feedback
Onboard Failure Logging

Table Of Contents

Onboard Failure Logging

Feature Overview

OBFL MBus Agent

Log Space Usage

Supported Error and Message Types

Baseline Data Collection

Event Driven Data Collection

Additional Security Features

Benefits

Supported Platforms

Supported Standards, MIBs, and RFCs

Configuration Tasks

Monitoring and Maintaining OBFL

Configuration Examples

Command Reference

clear logging onboard

clear logging onboard runtime

show logging onboard

show logging onboard runtime


Onboard Failure Logging


Feature History

Release
Modification

12.0(28)S

OBFL was introduced as an undocumented internal command for the Cisco 12000 Series route processors and line cards.

12.0(31)S

OBFL was marketed as an IOS feature.


Onboard Failure Logging (OBFL) captures and stores hardware failure and environmental information into nonvolatile memory. OBFL permits improved accuracy in hardware troubleshooting and root cause isolation analysis. Stored OBFL data can be retrieved in the event of a line card crash or failure and is accessible even if the line card does not boot.

Feature Overview

This section contains the following sections:

OBFL MBus Agent

Supported Error and Message Types

Baseline Data Collection

Event Driven Data Collection

OBFL MBus Agent

OBFL MBus Agent

OBFL data is stored in a nonvolatile memory location and is preserved if a after a line card resets, becomes nonoperational, or reloads. The nonvolatile memory location used for OBFL data collection is not shared with any other Cisco IOS functions on the Cisco 12000 Series router. OBFL does not impact performance or other system functions.

When OBFL is disabled, troubleshooting information is limited to information that might be lost in the event of a line card reset or reload such as, IOS crash dumps, message logs, and the show diagnostics command output.


Note Cisco has implemented a repair process that ensures stored OBFL data is completely cleared if the line card is returned through an RMA or an EFA.


OBFL MBus Agent

The Maintenance Bus (MBus) agent that includes OBFL is bundled with Cisco IOS. When upgrading to Cisco IOS Release 12.0(28)S or more recent, the earlier line card MBus agents are erased from flash memory and replaced with the MBus agent with OBFL.

If you downgrade to a Cisco IOS release that does not support OBFL, the older MBus agent is downloaded to run from RAM. Customers can then choose to write that earlier agent to flash by using the command upgrade mbus-agent-rom. This will erase all functions provided by the OBFL feature.

Log Space Usage

Data collected by OBFL is logged into non-volatile memory in chronological order. Flash memory used for OBFL log space is divided into two regions, active and buffer. First X is the active region, and then it fills up, the data is screened for important information which is then copied to Y. X is then erased, and new records are appended to Y. When Y fills up, the process repeats.

OBFL log output from the first flash region is displayed in chronological order, and from the second region by type of message.

Supported Error and Message Types

OBFL collects and stores both baseline and event driven information into existing Cisco 12000 Series Route Processor and line card nonvolatile memory.

Baseline Data Collection

Baseline data is stored independent of hardware or software failures occurring. This includes:

Data Type
Details

Installation

Chassis name and slot number are stored at initial boot and for the most recent nine boots.

Temperature

Inlet and hotpoint temperatures recorded 10 minutes after boot

Runtime

Total runtime since initial installation based on the local router clock with a granularity of 30 minutes.


Event Driven Data Collection

Event driven data include line card failure events. Failure events are line card crashes, memory errors, ASIC resets, and similar hardware failure indications.

Data Type
Details

Context

Complete route show context output is stored.

Environmental Errors

Temperature Errors

Inlet and hot point temperature errors

 

Voltage Errors

+5, and MBUS +5, +3.3, and +2.2 voltage errors

An environmental reading is logged when the following temperature or voltage events occur:

they exceed the normal range

change more than 10%

return within range for more than five minutes.

On reboot, these environmental readings are consolidated into a single environmental history record which shows the duration and extent out of normal range for a consecutive set of environmental readings.

Memory Errors

parity, single-bit, and multi-bit ECC errors

Calendar Time

Disabled

The time when OBFL logging was disabled with the no hw-module slot slot-number logging onboard global configuration command.

Cleared

The time when OBFL logging was cleared with the clear logging onboard privileged EXEC command.

Reset to 0

The time when total line card runtime is reset to zero with the clear logging onboard runtime privileged EXEC command.


Additional Security Features

An additional security feature has been added for those customers who want to routinely store OBFL data on external NMS systems, but do not want any of this data returned to Cisco on individual line cards. Customers can use the CLI command "clear logging onboard erase [all | slot <slot#>]" which will erase all OBFL sectors on the specified card.


Caution If OBFL is actively running on a card, issuing the "clear logging onboard erase [all | slot <slot#>]" command can result in a corrupt or incomplete log at a later point in time. OBFL should always be disabled before this command is issued.

Benefits

The benefits of OBFL are increased hardware failure troubleshooting accuracy, and the ability for Cisco to determine the root cause of failure.

Supported Platforms

All Cisco 12000 Series route processors and line cards supported by Cisco IOS releases 12.0(28)S and more recent. Note that this feature was considered internal only until Cisco IOS release 12.0(31)S.

Cisco 12000 Series line cards have different quantities of available non-volatile memory, which can account for slight differences in OBFL behavior among line cards.

OBFL uses 4 kilobytes of non-volatile memory space on most of the Cisco 12000 Series line cards, except for the following line cards, where OBFL requires 128 kilobytes of non-volatile memory:

8-port OC03/STM1 ATM MM LC with SC Connector

PN: 8OC03/ATM/TS-MM-B

8-port OC03/STM1 ATM IR LC with SC Connector

PN: 8OC03/ATM/TS-IR-B

4-port OC-12/STM-4 ATM, multimode with SC connector

PN: OC12/ATM-MM-SC

4-port OC-12/STM-4 ATM, intermediate reach with SC connector

PN: 4OC12/ATM-IR-SC

4-Port OC-12/STM-4 ATM ISE Multimode Line Card

PN: 4OC12X/ATM-MM-SC

4-Port OC-12/STM-4 ATM ISE Single Mode Line Card

PN: 4OC12X/ATM-IR-SC

2xOC-12 ISE SRP

4-port-GE line card for Cisco 12000

PN: 4GE-SFP-LC

All Line cards released after January 2004

Finding Support Information for Platforms and Cisco IOS Software Images

Use Cisco Feature Navigator to find information about platform support and Cisco IOS software image support. Access Cisco Feature Navigator at http://www.cisco.com/go/fn. You must have an account on Cisco.com. If you do not have an account or have forgotten your username or password, click Cancel at the login dialog box and follow the instructions that appear.

Supported Standards, MIBs, and RFCs

There are no standards or MIBs relevant to OBFL, it is an internal command.

To locate and download MIBs for selected platforms, Cisco IOS releases, and feature sets, use Cisco MIB Locator found at the following URL:

http://tools.cisco.com/ITDIT/MIBS/servlet/index

If Cisco MIB Locator does not support the MIB information that you need, you can also obtain a list of supported MIBs and download MIBs from the Cisco MIBs page at the following URL:

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

To access Cisco MIB Locator, you must have an account on Cisco.com. If you have forgotten or lost your account information, send a blank e-mail to cco-locksmith@cisco.com. An automatic check will verify that your e-mail address is registered with Cisco.com. If the check is successful, account details with a new random password will be e-mailed to you. Qualified users can establish an account on Cisco.com by following the directions found at this URL:

http://www.cisco.com/register

Configuration Tasks

OBFL is enabled by default and is active until the no hw-module slot slot-number logging onboard global configuration command is issued. The current status of OBFL is reported in the router's running configuration and can be viewed in the output of the show running-config command.

There are no configuration requirements other than to enable and disable OBFL. The following table lists the Cisco IOS command to enable and disable OBFL.

Command
Purpose

Router(config)# no hw-module slot slot-number logging onboard

Disables OBFL

Router(config)# hw-module slot slot-number logging onboard

Enables OBFL.


Monitoring and Maintaining OBFL

To confirm that the OBFL environmental monitor process is operating use the show processes include obfl privileged EXEC command.

The following table lists the Cisco IOS commands for displaying and clearing OBFL statistics:

Command
Purpose

Router# show logging onboard [slot slot-number]

Displays stored OBFL data for a specific route processor or line card.

Router# clear logging onboard [slot slot-number]

Resets OBFL CLI output, except for total runtime information.

Router# show logging onboard runtime [slot slot-number]

(Need copy of output)

Display line card runtime information stored by OBFL.

Router# clear logging onboard runtime [slot slot-number]

Clears line card runtime information stored by OBFL.


Configuration Examples

Show stored OBFL data for a specific route processor or line card (comments in blue).

router# show logging onboard 
[using 761 of 32768 bytes]
Log cleared at Jan 14 2004 16:51:56 Calendar time of OBFL clearing.
Runtime cleared at Jan 14 2004 16:52:31 Calendar time of OBFL runtime clearing.
Boot location #0: slot 7 in 'router' Initial boot location.
Temperature after last boot in location #0: inlet 31 C, hotpoint 39 C Temperature at 
initial boot.
Inlet temperature range 55 - 68 from Jan 19 2004 18:40:19 to Jan 19 2004 19:33:16
5V range 5250 - 5510 from Jan 20 2004 02:06:47 to Jan 20 2004 02:26:10
<=== Crash at Jan 20 2004 11:05:36 ===>
GS Software (GSR-P-M), Version 12.0(20031217:031315)S IOS release.
Compiled Wed 24-Dec-03 19:06 by apratend
Card uptime was 7 days, 3 hours, 21 minutes. Total runtime since runtime was cleared.
System exception: sig=5, code=0x24, context=0x422C2914
System restarted by a Bus Error exception Bus error indicator.
STACK TRACE:
-Traceback= 50DC091C 5005D028 5006E7A4 500C9898 500C9884 
CONTEXT: Summarized show context output.
$0 : 00000000, AT : 51A30000, v0 : FFFFFFFF, v1 : 00000000
a0 : 00000030, a1 : 52F17F00, a2 : 51A2C900, a3 : BEEFCAFE
t0 : 00008100, t1 : 34008101, t2 : 500FAA88, t3 : FFFF00FF
t4 : 500FAA58, t5 : 00000064, t6 : 00000000, t7 : 00000000
s0 : 00000038, s1 : 00000054, s2 : 00000015, s3 : 51910000
s4 : 53AA5D68, s5 : 00000000, s6 : 00000001, s7 : 00000000
t8 : 5440B538, t9 : 00000000, k0 : 53173160, k1 : 500FD424
gp : 51A343A0, sp : 53AA5D50, s8 : 51910000, ra : 50DC0908
EPC : 0x50DC0918, SREG : 0x34008103, Cause : 0x80000024
ErrorEPC : 0x506F0044, BadVaddr : 0xC9070369
Current/last process: pid 64, name "Exec"
-Process Traceback= No Extra Traceback
<=== End Crash ===>
Temperature after last boot in location #0: inlet 31 C, hotpoint 40 C Temperature error 
information.
Inlet temperature sensor value 72 at Jan 26 2004 11:20:49
Inlet temperature sensor value 63 at Jan 26 2004 11:27:15
Inlet temperature sensor okay at Jan 26 2004 11:33:46
Boot location #1: slot 6 in 'router' Current Boot location.
Temperature after last boot in location #1: inlet 31 C, hotpoint 40 C Temperature at boot 
in current location.
Logging disabled at Jan 27 2004 16:00:02 OBFL time logging was disabled.

The following example clears line card runtime information stored by OBFL:

Router# clear logging onboard runtime

The following example displays runtime data stored by OBFL:

Router# show logging onboard runtime
XXX Missing data XXX

The following example clears line card runtime data stored by OBFL:

Router# clear logging onboard runtime

Command Reference

This section documents new commands. All other commands used with this feature are documented in the Cisco IOS Release 12.0S command reference publications.

clear logging onboard

To reset OBFL CLI output, except for total runtime information, use the clear logging onboard command in privileged exec configuration mode.

clear logging onboard [slot slot-number]

Syntax Description

slot

(Optional) Specifies that only the OBFL information

slot-number

Species the slot number of the line card.


Defaults

No default behavior or values.

Command Modes

Privileged exec

Command History

Release
Modification

12.0(31)S

This command was introduced. Note that this feature was considered internal only until Cisco IOS release 12.0(31)S.


Usage Guidelines

There are no additional usage guidelines.

Examples

The following example clears the OBFL log for the line card in slot 2:

router#: clear logging onboard slot 2 

Related Commands

Command
Description

clear logging onboard runtime

Clears line card OBFL runtime information.

show logging onboard

Displays OBFL information for route processors and line cards.

show logging onboard runtime

Displays OBFL runtime information for route processors and line cards.


clear logging onboard runtime

To clear line card runtime information stored by OBFL, use the clear logging onboard runtime command in privileged exec configuration mode.

clear logging onboard runtime [slot slot-number]

Syntax Description

slot

(Optional). Specifies that only the OBFL information of a single line card will be cleared.

slot-number

Species the slot number of the line card.


Defaults

No default behavior or values.

Command Modes

Privileged exec

Command History

Release
Modification

12.0(31)S

This command was introduced. Note that this feature was considered internal only until Cisco IOS release 12.0(31)S.


Usage Guidelines

There are no additional usage guidelines.

Examples

The following example clears the OBFL runtime log for the line card in slot 2:

router#: clear logging onboard slot 2 

Related Commands:

Command
Description

clear logging onboard

Clears OBFL CLI output log, except for total runtime information.

show logging onboard

Displays OBFL information for route processors and line cards.

show logging onboard runtime

Displays OBFL runtime information for route processors and line cards.


show logging onboard

To display stored OBFL data for a specific route processor or line card, use the show logging onboard command in privileged exec mode.

show logging onboard slot slot-number

Syntax Description

slot

(Optional). Specifies that only the OBFL information of a single line card will be displayed.

slot-number

Species the slot number of the line card.


Defaults

No default behavior or values.

Command Modes

Privileged exec

Command History

Release
Modification

12.0(31)S

This command was introduced. Note that this feature was considered internal only until Cisco IOS release 12.0(31)S.


Usage Guidelines

There are no additional usage guidelines.

Examples

The following example displays OBFL information when the slot keyword is not specified:

Router# show logging onboard 
[using 761 of 32768 bytes]
Log cleared at Jan 14 2004 16:51:56 
Runtime cleared at Jan 14 2004 16:52:31 .
Boot location #0: slot 7 in 'router' 
Temperature after last boot in location #0: inlet 31 C, hotpoint 39 C 
Inlet temperature range 55 - 68 from Jan 19 2004 18:40:19 to Jan 19 2004 19:33:16
5V range 5250 - 5510 from Jan 20 2004 02:06:47 to Jan 20 2004 02:26:10
<=== Crash at Jan 20 2004 11:05:36 ===>
GS Software (GSR-P-M), Version 12.0(20031217:031315)S 
Compiled Wed 24-Dec-03 19:06 by apratend
Card uptime was 7 days, 3 hours, 21 minutes. 
System exception: sig=5, code=0x24, context=0x422C2914
System restarted by a Bus Error exception .
STACK TRACE:
-Traceback= 50DC091C 5005D028 5006E7A4 500C9898 500C9884 
CONTEXT: 
$0 : 00000000, AT : 51A30000, v0 : FFFFFFFF, v1 : 00000000
a0 : 00000030, a1 : 52F17F00, a2 : 51A2C900, a3 : BEEFCAFE
t0 : 00008100, t1 : 34008101, t2 : 500FAA88, t3 : FFFF00FF
t4 : 500FAA58, t5 : 00000064, t6 : 00000000, t7 : 00000000
s0 : 00000038, s1 : 00000054, s2 : 00000015, s3 : 51910000
s4 : 53AA5D68, s5 : 00000000, s6 : 00000001, s7 : 00000000
t8 : 5440B538, t9 : 00000000, k0 : 53173160, k1 : 500FD424
gp : 51A343A0, sp : 53AA5D50, s8 : 51910000, ra : 50DC0908
EPC : 0x50DC0918, SREG : 0x34008103, Cause : 0x80000024
ErrorEPC : 0x506F0044, BadVaddr : 0xC9070369
Current/last process: pid 64, name "Exec"
-Process Traceback= No Extra Traceback
<=== End Crash ===>
Temperature after last boot in location #0: inlet 31 C, hotpoint 40 C 
Inlet temperature sensor value 72 at Jan 26 2004 11:20:49
Inlet temperature sensor value 63 at Jan 26 2004 11:27:15
Inlet temperature sensor okay at Jan 26 2004 11:33:46
Boot location #1: slot 6 in 'router' 
Temperature after last boot in location #1: inlet 31 C, hotpoint 40 C T 
Logging disabled at Jan 27 2004 16:00:02 

Related Commands

Command
Description

clear logging onboard

Clears OBFL CLI output log, except for total runtime information.

clear logging onboard runtime

Clears line card OBFL runtime information.

show logging onboard runtime

Displays OBFL runtime information for route processors and line cards.


show logging onboard runtime

To display line card runtime information stored by OBFL, use the show logging onboard runtime command in privileged exec mode.

show logging onboard runtime [slot slot-number]

Syntax Description

slot

(Optional). Specifies that only the OBFL runtime information of a single line card will be displayed.

slot-number

Species the slot number of the line card.


Defaults

No default behavior or values.

Command Modes

Privileged exec

Command History

Release
Modification

12.0(31)S

This command was introduced. Note that this feature was considered internal only until Cisco IOS release 12.0(31)S.


Usage Guidelines

There are no additional usage guidelines.

Examples

The following example displays OBFL runtime information for a line card in slot 2:

Router# show logging onboard runtime slot 2

Related Commands

Command
Description

clear logging onboard

Clears OBFL CLI output log, except for total runtime information.

clear logging onboard runtime

Clears line card OBFL runtime information.

show logging onboard

Displays OBFL information for route processors and line cards.