Table Of Contents

Overview

Troubleshooting Information in the Cisco UCS Manager GUI

Troubleshooting Information in the Cisco UCS Manager CLI

Additional Troubleshooting Documentation

Faults

Fault Severities

Fault States

Fault Types

Properties of Faults

Lifecycle of Faults

Faults in the Cisco UCS Manager GUI

Faults in the Cisco UCS Manager CLI

Fault Collection Policy

Events

Properties of Events

Events in the Cisco UCS Manager GUI

Events in the Cisco UCS Manager CLI

Core Files

Core Files in the Cisco UCS Manager GUI

Core Files in the Cisco UCS Manager CLI

Core File Exporter

Audit Log

Properties of the Audit Log Entries

Audit Log in the Cisco UCS Manager GUI

Audit Log in the Cisco UCS Manager CLI

System Event Log

SEL File

SEL Policy

Syslog

Syslog Configuration

Syslog Location

Syslog Entry Format

Syslog Entry Severities

Syslog Entry Parameters

Syslog Services

Overview

---

This chapter provides an overview of where to find faults, events, and other information that can help you troubleshoot issues with Cisco Unified Computing System (Cisco UCS) B-Series Servers.

This chapter includes the following sections:

•Troubleshooting Information in the Cisco UCS Manager GUI

•Troubleshooting Information in the Cisco UCS Manager CLI

•Additional Troubleshooting Documentation

•Faults

•Events

•Core Files

•Audit Log

•System Event Log

•Syslog

Troubleshooting Information in the Cisco UCS Manager GUI

The Cisco UCS Manager GUI provides several tabs and other areas that you can use to find troubleshooting information for a Cisco UCS instance. For example, you can view faults and events for specific objects or for all objects in the system.

The Admin tab in the Navigation pane provides access to faults, events, core files, and other information that can help you troubleshoot issues.

If you select Faults, Events and Audit Log in the Filter field on the Admin tab, the Cisco UCS Manager GUI limits the tree browser so that you can only access the following:

•The faults for all components in the system

•The events for all components in the system

•The audit log for the system

•Any core files created by the fabric interconnects in the system

•The fault collection and core file export settings

---

Note Fault thresholds might need to be modified. See the "Statistics Threshold Policy" section in the Cisco UCS Manager GUI Configuration Guide.

---

Troubleshooting Information in the Cisco UCS Manager CLI

The Cisco UCS Manager CLI includes several show commands that you can execute to find troubleshooting information for a Cisco UCS instance. These show commands are scope-aware, which means that if you enter the show fault command from the top scope, it displays all faults in the system. However, if you scope to a specific object, the show fault command displays faults that are related to that object only.

---

Note Fault thresholds might need to be modified. See the "Statistics Threshold Policy" section in the Cisco UCS Manager CLI Configuration Guide.

---

Additional Troubleshooting Documentation

Additional troubleshooting information is available in the following documents:

•Cisco UCS Manager Faults and Error Message Reference—Contains information about Cisco UCS Manager faults and System Event Log messages, including BIOS and CIMC messages.

•Cisco UCS C-Series Servers Integrated Management Controller Troubleshooting Guide—Contains information about how to troubleshoot issues with C-Series rack-mount servers.

Faults

In Cisco UCS, a fault is a mutable object that is managed by Cisco UCS Manager. Each fault represents a failure in the Cisco UCS instance or an alarm threshold that has been raised. During the lifecycle of a fault, it can change from one state or severity to another.

Each fault includes information about the operational state of the affected object at the time the fault was raised. If the fault is transitional and the failure is resolved, the object transitions to a functional state.

A fault remains in Cisco UCS Manager until the fault is cleared and deleted according to the settings in the fault collection policy.

You can view all faults in a Cisco UCS instance from either the Cisco UCS Manager CLI or the Cisco UCS Manager GUI. You can also configure the fault collection policy to determine how a Cisco UCS instance collects and retains faults.

---

Note All Cisco UCS faults can be trapped by SNMP.

---

This section includes the following topics:

•Fault Severities

•Fault States

•Fault Types

•Properties of Faults

•Lifecycle of Faults

•Faults in the Cisco UCS Manager GUI

•Faults in the Cisco UCS Manager CLI

•Fault Collection Policy

Fault Severities

A fault raised in a Cisco UCS instance can transition through more than one severity during its lifecycle. Table 1-1 describes the fault severities that you may encounter.

Table 1-1 Fault Severities in Cisco UCS

Severity	Description
Critical	Service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored.
Major	Service-affecting condition that requires urgent corrective action. For example, this severity could indicate a severe degradation in the capability of the managed object and that its full capability must be restored.
Minor	Nonservice-affecting fault condition that requires corrective action to prevent a more serious fault from occurring. For example, this severity could indicate that the detected alarm condition is not degrading the capacity of the managed object.
Warning	Potential or impending service-affecting fault that has no significant effects in the system. You should take action to further diagnose, if necessary, and correct the problem to prevent it from becoming a more serious service-affecting fault.
Condition	Informational message about a condition, possibly independently insignificant.
Info	Basic notification or informational message, possibly independently insignificant.

Fault States

A fault raised in a Cisco UCS instance transitions through more than one state during its lifecycle. Table 1-2 describes the possible fault states in alphabetical order.

Table 1-2 Fault States in Cisco UCS

State	Description
Cleared	Condition that has been resolved and cleared.
Flapping	Fault that was raised, cleared, and raised again within a short time interval, known as the flap interval.
Soaking	Fault that was raised and cleared within a short time interval, known as the flap interval. Because this state may be a flapping condition, the fault severity remains at its original active value, but this state indicates the condition that raised the fault has cleared.

Fault Types

A fault raised in a Cisco UCS instance can be one of the types described in Table 1-3.

Table 1-3 Types of Faults in Cisco UCS

Type	Description
fsm	FSM task has failed to complete successfully, or Cisco UCS Manager is retrying one of the stages of the FSM.
equipment	Cisco UCS Manager has detected that a physical component is inoperable or has another functional issue.
server	Cisco UCS Manager cannot complete a server task, such as associating a service profile with a server.
configuration	Cisco UCS Manager cannot successfully configure a component.
environment	Cisco UCS Manager has detected a power problem, thermal problem, voltage problem, or loss of CMOS settings.
management	Cisco UCS Manager has detected a serious management issue, such as one of the following: •Critical services could not be started •The primary fabric interconnect could not be identified •Components in the instance include incompatible firmware versions
connectivity	Cisco UCS Manager has detected a connectivity problem, such as an unreachable adapter.
network	Cisco UCS Manager has detected a network issue, such as a link down.
operational	Cisco UCS Manager has detected an operational problem, such as a log capacity issue or a failed server discovery.

Properties of Faults

Cisco UCS Manager provides detailed information about each fault raised in a Cisco UCS instance. Table 1-4 describes the fault properties that you can view in the Cisco UCS Manager CLI or the Cisco UCS Manager GUI.

Table 1-4 Fault Properties 

Property Name	Description
Severity	Current severity level of the fault, which can be any of the severities described in Table 1-1.
Last Transition	Day and time on which the severity for the fault last changed. If the severity has not changed since the fault was raised, this property displays the original creation date.
Affected Object	Component that is affected by the condition that raised the fault.
Description	Description of the fault.
ID	Unique identifier assigned to the fault.
Type	Type of fault that has been raised, which can be any of the types described in Table 1-3.
Cause	Unique identifier associated with the condition that caused the fault.
Created at	Day and time when the fault occurred.
Code	Unique identifier assigned to the fault.
Number of Occurrences	Number of times the event that raised the fault occurred.
Original Severity	Severity assigned to the fault the first time it occurred.
Previous Severity	Previous severity level.
Highest Severity	Highest severity encountered for this issue.

Lifecycle of Faults

Faults in Cisco UCS are stateful. Only one instance of a given fault can exist on each object. If the same fault occurs a second time, Cisco UCS increases the number of occurrences by one.

A fault has the following lifecycle:

1. A condition occurs in the system and Cisco UCS raises a fault. This is the active state.

2. When the fault is alleviated, it enters a flapping or soaking interval that is designed to prevent flapping. Flapping occurs when a fault is raised and cleared several times in rapid succession. During the flapping interval, the fault retains its severity for the length of time specified in the fault collection policy.

3. If the condition reoccurs during the flapping interval, the fault returns to the active state. If the condition does not reoccur during the flapping interval, the fault is cleared.

4. The cleared fault enters the retention interval. This interval ensures that the fault reaches the attention of an administrator even if the condition that caused the fault has been alleviated and the fault has not been deleted prematurely. The retention interval retains the cleared fault for the length of time specified in the fault collection policy.

5. If the condition reoccurs during the retention interval, the fault returns to the active state. If the condition does not reoccur, the fault is deleted.

Faults in the Cisco UCS Manager GUI

If you want to view faults for a single object in the system, navigate to that object in the Cisco UCS Manager GUI and click the Faults tab in the Work pane. If you want to view faults for all objects in the system, navigate to the Faults node on the Admin tab under Faults, Events and Audit Log.

In addition, you can also view a summary of all faults in a Cisco UCS instance in the Fault Summary area in the upper left of the Cisco UCS Manager GUI. This area provides a summary of all faults that have occurred in the Cisco UCS instance.

Each fault severity is represented by a different icon. The number below each icon indicates how many faults of that severity have occurred in the system. If you click an icon, the Cisco UCS Manager GUI opens the Faults tab in the Work area and displays the details of all faults with that severity.

Faults in the Cisco UCS Manager CLI

If you want to view the faults for all objects in the system, enter the show fault command from the top-level scope. If you want to view the faults for a specific object, scope to that object and then execute the show fault command.

If you want to view all available details about a fault, enter the show fault detail command.

Fault Collection Policy

The fault collection policy controls the lifecycle of a fault in the Cisco UCS instance, including the length of time that each fault remains in the flapping and retention intervals.

---

Tip For information on how to configure the fault collection policy, see the Cisco UCS configuration guides, which are accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

---

Events

In Cisco UCS, an event is an immutable object that is managed by Cisco UCS Manager. Each event represents a nonpersistent condition in the Cisco UCS instance. After Cisco UCS Manager creates and logs an event, the event does not change. For example, if you power on a server, Cisco UCS Manager creates and logs an event for the beginning and the end of that request.

You can view events for a single object, or you can view all events in a Cisco UCS instance from either the Cisco UCS Manager CLI or the Cisco UCS Manager GUI. Events remain in the Cisco UCS until the event log fills up. When the log is full, Cisco UCS Manager purges the log and all events in it.

This section includes the following topics:

•Properties of Events

•Events in the Cisco UCS Manager GUI

•Events in the Cisco UCS Manager CLI

Properties of Events

Cisco UCS Manager provides detailed information about each event created and logged in a Cisco UCS instance. Table 1-5 describes the fault properties that you can view in the Cisco UCS Manager CLI or the Cisco UCS Manager GUI.

Table 1-5 Event Properties

Property Name	Description
Affected Object	Component that created the event.
Description	Description of the event.
Cause	Unique identifier associated with the event.
Created at	Day and time when the event was created.
User	Type of user that created the event, such as one of the following: •admin •internal •blank
Code	Unique identifier assigned to the event.

Events in the Cisco UCS Manager GUI

If you want to view events for a single object in the system, navigate to that object in the Cisco UCS Manager GUI and click the Events tab in the Work pane. If you want to view events for all objects in the system, navigate to the Events node on the Admin tab under the Faults, Events and Audit Log.

Events in the Cisco UCS Manager CLI

If you want to view events for all objects in the system, enter the show event command from the top-level scope. If you want to view events for a specific object, scope to that object and then enter the show event command.

If you want to view all available details about an event, enter the show event detail command.

Core Files

Critical failures in Cisco UCS Manager and some of the Cisco UCS components, such as a fabric interconnect or an I/O module, can cause the system to create a core file. Each core file contains a large amount of data about the system and the component at the time of the failure.

Cisco UCS Manager manages the core files from all of the components. You can configure Cisco UCS Manager to export a copy of a core file to a location on an external TFTP server as soon as that core file is created.

This section includes the following topics:

•Core Files in the Cisco UCS Manager GUI

•Core Files in the Cisco UCS Manager CLI

•Core File Exporter

Core Files in the Cisco UCS Manager GUI

You can find out if a component in the Cisco UCS instance generated a core file by navigating to the Core Files node on the Admin tab under the Faults, Events and Audit Log.

Core Files in the Cisco UCS Manager CLI

You can find out if a component in the Cisco UCS instance generated a core file by entering the following commands:

•scope monitoring

•scope sysdebug

•show cores

Core File Exporter

If you enable the Core File Exporter, you can configure Cisco UCS Manager to export the core files as soon as they occur to a specified location on the network through TFTP. This functionality allows you to export the tar file with the contents of the core file to the location specified.

---

Tip For information on how to enable the exporter, see the Cisco UCS configuration guides, which are accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

---

Audit Log

The audit log records actions performed by users in Cisco UCS Manager, including direct and indirect actions. Each entry in the audit log represents a single, non-persistent action. For example, if a user logs in, logs out, or creates, modifies, or deletes an object such as a service profile, Cisco UCS Manager adds an entry to the audit log for that action.

You can view the audit log entries in the Cisco UCS Manager CLI, Cisco UCS Manager GUI, or output from the show tech-support command.

This section includes the following topics:

•Properties of the Audit Log Entries

•Audit Log in the Cisco UCS Manager GUI

•Audit Log in the Cisco UCS Manager CLI

Properties of the Audit Log Entries

Cisco UCS Manager provides detailed information about each entry in the audit log. Table 1-6 describes the fault properties that you can view in the Cisco UCS Manager CLI or the Cisco UCS Manager GUI.

Table 1-6 Audit Log Entry Properties

Property Name	Description
ID	Unique identifier associated with the audit log message.
Affected Object	Component affected by the user action.
Severity	Current severity level of the user action associated with the audit log message. These severities are also used for the faults, as described in Table 1-1.
Trigger	User role associated with the user that raised the message.
User	Type of user that created the event, as follows: •admin •internal •blank
Indication	Action indicated by the audit log message. This can be one of the following: •creation—A component was added to the system. •modification—An existing component was changed.
Description	Description of the user action.

Audit Log in the Cisco UCS Manager GUI

If you want to view the audit log, navigate to the Audit Log node on the Admin tab under Faults, Events and Audit Log.

Audit Log in the Cisco UCS Manager CLI

If you want to view the audit log, enter the following commands:

•scope security

•show audit-logs

System Event Log

The system event log (SEL) resides on the CIMC in NVRAM. It records most of the server-related events, such as overvoltage and undervoltage, temperature events, fan events, events from BIOS, and so on. The SEL is primarily used for troubleshooting purposes.

---

Tip For more information about the SEL, including how to view the SEL for each server and configure the SEL policy, see the Cisco UCS configuration guides, which are accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

---

This section includes the following topics:

•SEL File

•SEL Policy

SEL File

The SEL file is approximately 40 KB. No further events can be recorded when the SEL file is full. It must be cleared before additional events can be recorded.

SEL Policy

You can use the SEL policy to back up the SEL to a remote server and optionally clear the SEL after a backup operation occurs. Backup operations can be triggered, based on specific actions, or they can occur at regular intervals. You can also manually back up or clear the SEL.

Cisco UCS Manager automatically generates the SEL backup file, according to the settings in the SEL policy. The filename format is sel-SystemName-ChassisID-ServerID-ServerSerialNumber-Timestamp

For example, a filename could be sel-UCS-A-ch01-serv01-QCI12522939-20091121160736.

Syslog

The syslog provides a central point for collecting and processing system logs that you can use to troubleshoot and audit the Cisco UCS instance. Cisco UCS Manager relies on the Cisco NX-OS syslog mechanism and API, and on the syslog feature of the primary fabric interconnect to collect and process the syslog entries.

This section includes the following topics:

•Syslog Configuration

•Syslog Location

•Syslog Entry Format

•Syslog Entry Severities

•Syslog Entry Parameters

•Syslog Services

Syslog Configuration

Cisco UCS Manager manages and configures the syslog collectors for the Cisco UCS instance and deploys the configuration to the fabric interconnect or fabric interconnects. This configuration affects all syslog entries generated in the Cisco UCS instance by Cisco NX-OS or by Cisco UCS Manager.

Syslog Location

You can configure Cisco UCS Manager to do one or more of the following with the syslog and syslog entries:

•Display the syslog entries in the console or on the monitor

•Store the syslog entries in a file

•Forward the syslog entries to up to three external log collectors where the syslog for the Cisco UCS instance is stored

Syslog Entry Format

Each syslog entry generated by a Cisco UCS component is formatted as follows:

Year month date hh:mm:ss hostname %facility-severity-MNEMONIC description

For example: 2007 Nov 1 14:07:58 excal-113 %MODULE-5-MOD_OK: Module 1 is online

Syslog Entry Severities

A syslog entry is assigned a Cisco UCS severity by Cisco UCS Manager. Table 1-7 shows how the Cisco UCS severities map to the syslog severities.

Table 1-7 Syslog Entry Severities in Cisco UCS

Cisco UCS Severity	Syslog Severity
Critical	CRIT
Major	ERR
Minor	ERR
Warning	WARNING
Info	INFO

Syslog Entry Parameters

Table 1-8 describes the information contained in each syslog entry.

Table 1-8 Syslog Message Content

Name	Description
Facility	Logging facility that generated and sent the syslog entry. The facilities are broad categories that are represented by integers. These sources can be one of the following standard Linux facilities: •local0 •local1 •local2 •local3 •local4 •local5 •local6 •local7
Severity	Severity of the event, alert, or issue that caused the syslog entry to be generated. The severity can be one of the following: •emergencies •critical •alerts •errors •warnings •information •notifications •debugging
Hostname	Hostname included in the syslog entry that depends upon the component where the entry originated, as follows: •The fabric interconnect, the Cisco UCS Manager, or the hostname of the Cisco UCS instance •For all other components, the hostname associated with the virtual interface
Timestamp	Date and time when the syslog entry was generated.
Message	Description of the event, alert, or issue that caused the syslog entry to be generated.

Syslog Services

The following Cisco UCS components use the Cisco NX-OS syslog services to generate syslog entries for system information and alerts:

•I/O module—All syslog entries are sent by syslogd to the fabric interconnect to which it is connected.

•CIMC—All syslog entries are sent to the primary fabric interconnect in a cluster configuration.

•Adapter—All syslog entries are sent by NIC-Tools/Syslog to both fabric interconnects.

•Cisco UCS Manager—Self-generated syslog entries are logged according to the syslog configuration.