-
Cisco MDS 9000 Family Configuration Guide, Release 2.x
-
New and Changed Information
-
Index
-
Preface
- Part 1 - Getting Started
- Part 2 - Cisco MDS SAN-OS Installation and Switch Management
- Part 3 - Switch Configuration
-
Part 4 - Fabric Configuration
-
Configuring and Managing VSANs
-
Creating Dynamic VSANs
-
Configuring Inter-VSAN Routing
-
Configuring Zones
-
Distributing Device Alias Services
-
Configuring Fibre Channel Routing Services and Protocols
-
Managing FLOGI, Name Server, FDMI, and RSCN Databases
-
Discovering SCSI Targets
-
Configuring FICON
-
Advanced Features and Concepts
-
- Part 5 - Security
- Part 6 - IP Services
- Part 7 - Intelligent Storage Services
- Part 8 - Network and Switch Monitoring
- Part 9 - Traffic Management
- Part 10 - Troubleshooting
-
Feedback
Table Of Contents
Monitoring System Processes and Logs
Online System Health Management
Loopback Test Configuration Frequency
Clearing Previous Error Reports
Interpreting the Current Status
Monitoring System Processes and Logs
This chapter provides details on monitoring the health of the switch. It includes the following sections:
•
Online System Health Management
Displaying System Processes
Use the show processes command to obtain general information about all processes (see Example 50-1 to Example 50-6).
Example 50-1 Displays System Processes
switch# show processesPID State PC Start_cnt TTY Process ----- ----- -------- ----------- ---- -------------868 S 2ae4f33e 1 - snmpd869 S 2acee33e 1 - rscn870 S 2ac36c24 1 - qos871 S 2ac44c24 1 - port-channel872 S 2ac7a33e 1 - ntp- ER - 1 - mdog- NR - 0 - vbuilderWhere:
•
•
–
–
–
–
–
•
•
•
•
•
•
Example 50-2 Displays CPU Utilization Information
switch# show processes cpuPID Runtime(ms) Invoked uSecs 1Sec Process----- ----------- -------- ----- ----- -----------842 3807 137001 27 0.0 sysmgr1112 1220 67974 17 0.0 syslogd1269 220 13568 16 0.0 fcfwd1276 2901 15419 188 0.0 zone1277 738 21010 35 0.0 xbar_client1278 1159 6789 170 0.0 wwn1279 515 67617 7 0.0 vsanWhere:
•
•
•
•
Example 50-3 Displays Process Log Information
switch# show processes logProcess PID Normal-exit Stack-trace Core Log-create-time---------------- ------ ----------- ----------- ------- ---------------fspf 1339 N Y N Jan 5 04:25lcm 1559 N Y N Jan 2 04:49rib 1741 N Y N Jan 1 06:05Where:
•
•
•
•
Example 50-4 Displays Detail Log Information About a Process
switch# show processes log pid 1339Service: fspfDescription: FSPF Routing Protocol ApplicationStarted at Sat Jan 5 03:23:44 1980 (545631 us)Stopped at Sat Jan 5 04:25:57 1980 (819598 us)Uptime: 1 hours 2 minutes 2 secondsStart type: SRV_OPTION_RESTART_STATELESS (23)Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)Exit code: signal 9 (no core)CWD: /var/sysmgr/workVirtual Memory:CODE 08048000 - 0809A100DATA 0809B100 - 0809B65CBRK 0809D988 - 080CD000STACK 7FFFFD20TOTAL 23764 KBRegister Set:EBX 00000005 ECX 7FFFF8CC EDX 00000000ESI 00000000 EDI 7FFFF6CC EBP 7FFFF95CEAX FFFFFDFE XDS 8010002B XES 0000002BEAX 0000008E (orig) EIP 2ACE133E XCS 00000023EFL 00000207 ESP 7FFFF654 XSS 0000002BStack: 1740 bytes. ESP 7FFFF654, TOP 7FFFFD200x7FFFF654: 00000000 00000008 00000003 08051E95 ................0x7FFFF664: 00000005 7FFFF8CC 00000000 00000000 ................0x7FFFF674: 7FFFF6CC 00000001 7FFFF95C 080522CD ........\...."..0x7FFFF684: 7FFFF9A4 00000008 7FFFFC34 2AC1F18C ........4......*Example 50-5 Displays All Process Log Details
switch# show processes log details======================================================Service: snmpdDescription: SNMP AgentStarted at Wed Jan 9 00:14:55 1980 (597263 us)Stopped at Fri Jan 11 10:08:36 1980 (649860 us)Uptime: 2 days 9 hours 53 minutes 53 secondsStart type: SRV_OPTION_RESTART_STATEFUL (24)Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)Exit code: signal 6 (core dumped)CWD: /var/sysmgr/workVirtual Memory:CODE 08048000 - 0804C4A0DATA 0804D4A0 - 0804D770BRK 0804DFC4 - 0818F000STACK 7FFFFCE0TOTAL 26656 KB...Example 50-6 Displays Memory Information About Processes
switch# show processes memoryPID MemAlloc StackBase/Ptr Process----- -------- ----------------- ----------------1277 120632 7ffffcd0/7fffefe4 xbar_client1278 56800 7ffffce0/7ffffb5c wwn1279 1210220 7ffffce0/7ffffbac vsan1293 386144 7ffffcf0/7fffebd4 span1294 1396892 7ffffce0/7fffdff4 snmpd1295 214528 7ffffcf0/7ffff904 rscn1296 42064 7ffffce0/7ffffb5c qosWhere:
•
•
Displaying System Status
Use the show system command to display system-related status information (see Example 50-7 to Example 50-10.
Example 50-7 Displays Default Switch Port States
switch# show system default switchportSystem default port state is downSystem default trunk mode is onExample 50-8 Displays Error Information for a Specified ID
switch# show system error-id 0x401D0019Error Facility: moduleError Description: Failed to stop Linecard Async Notification.Example 50-9 Displays the System Reset Information
switch# Show system reset-reason module 5----- reset reason for module 5 -----1) At 224801 usecs after Fri Nov 21 16:36:40 2003Reason: Reset Requested by CLI command reloadService:Version: 1.3(1)2) At 922828 usecs after Fri Nov 21 16:02:48 2003Reason: Reset Requested by CLI command reloadService:Version: 1.3(1)3) At 318034 usecs after Fri Nov 21 14:03:36 2003Reason: Reset Requested by CLI command reloadService:Version: 1.3(1)4) At 255842 usecs after Wed Nov 19 00:07:49 2003Reason: Reset Requested by CLI command reloadService:Version: 1.3(1)The show system reset-reason command displays the following information:
•
•
•
Use the clear system reset-reason command to clear the reset-reason information stored in NVRAM and volatile persistent storage.
•
•
Example 50-10 Displays System Uptime
switch# show system uptimeStart Time: Sun Oct 13 18:09:23 2030Up Time: 0 days, 9 hours, 46 minutes, 26 secondsUse the show system resources command to display system-related CPU and memory statistics (see Example 50-11).
Example 50-11 Displays System-Related CPU and Memory Information
switch# show system resourcesLoad average: 1 minute: 0.43 5 minutes: 0.17 15 minutes: 0.11Processes : 100 total, 2 runningCPU states : 0.0% user, 0.0% kernel, 100.0% idleMemory usage: 1027628K total, 313424K used, 714204K free3620K buffers, 22278K cacheWhere:
•
•
•
•
Core and Log Files
You can save cores (from the active supervisor module, the standby supervisor module, or any switching module) to an external Flash (slot 0) or to a TFTP server in one of two ways:
•
•
A new scheme overwrites any previously issued scheme. For example, if you perform another core log copy task, the cores are periodically saved to the new location or file.
Tip
To copy the core and log files on demand, follow this step:
•
switch# copy core:133 slot0:fooNo core file found with pid 133•
switch# copy core:7407 slot0:foo12 core files found with pid 7407Only "/isan/tmp/logs/calc_server_log.7407.tar.gz" will be copied to the destination.To copy the core and log files periodically, follow these steps:
Saving the Last Core to Flash
This last core dump is automatically saved to the flash in the /mnt/pss/ partition before the switchover or reboot occurs. Three minutes after the supervisor module reboots, the saved last core is restored from the flash partition (/mnt/pss) back its original RAM location. This restoration is a background process and is not visible to the user.
Tip
To view the last core information, issue the show cores command in EXEC mode.
To view the time of the actual last core dump, issue the show process log command in EXEC mode.
Clearing the Core Directory
Use the clear cores command to clean out the core directory. The software keeps the last few cores per service and per slot and clears all other cores present on the active supervisor module.
switch# clear coresDisplaying Core Status
Use the show system cores command to display the currently configured scheme for copying cores. See Examples 50-12 to 50-14.
Example 50-12 Displays the Status of System Cores
switch# show system coresTransfer of cores is enabledExample 50-13 Displays All Cores Available for Upload from the Active Supervisor Module
switch# show coresModule-num Process-name PID Core-create-time---------- ------------ --- ----------------5 fspf 1524 Nov 9 03:116 fcc 919 Nov 9 03:098 acltcam 285 Nov 9 03:098 fib 283 Nov 9 03:08Where Module-num shows the slot number on which the core was generated. In this example, the fspf core was generated on the active supervisor module (slot 5), fcc was generated on the standby supervisor module (slot 6), and acltcam and fib were generated on the switching module (slot 8).
Example 50-14 Displays Logs on the Local System
switch# show processes logProcess PID Normal-exit Stack Core Log-create-time---------------- ------ ----------- ----- ----- ---------------ExceptionLog 2862 N Y N Wed Aug 6 15:08:34 2003acl 2299 N Y N Tue Oct 28 02:50:01 2003bios_daemon 2227 N Y N Mon Sep 29 15:30:51 2003capability 2373 N Y N Tue Aug 19 13:30:02 2003core-client 2262 N Y N Mon Sep 29 15:30:51 2003fcanalyzer 5623 N Y N Fri Sep 26 20:45:09 2003fcd 12996 N Y N Fri Oct 17 20:35:01 2003fcdomain 2410 N Y N Thu Jun 12 09:30:58 2003ficon 2708 N Y N Wed Nov 12 18:34:02 2003ficonstat 9640 N Y N Tue Sep 30 22:55:03 2003flogi 1300 N Y N Fri Jun 20 08:52:33 2003idehsd 2176 N Y N Tue Jun 24 05:10:56 2003lmgrd 2220 N N N Mon Sep 29 15:30:51 2003platform 2840 N Y N Sat Oct 11 18:29:42 2003port-security 3098 N Y N Sun Sep 14 22:10:28 2003port 11818 N Y N Mon Nov 17 23:13:37 2003rlir 3195 N Y N Fri Jun 27 18:01:05 2003rscn 2319 N Y N Mon Sep 29 21:19:14 2003securityd 2239 N N N Thu Oct 16 18:51:39 2003snmpd 2364 N Y N Mon Nov 17 23:19:39 2003span 2220 N Y N Mon Sep 29 21:19:13 2003syslogd 2076 N Y N Sat Oct 11 18:29:40 2003tcap 2864 N Y N Wed Aug 6 15:09:04 2003tftpd 2021 N Y N Mon Sep 29 15:30:51 2003vpm 2930 N N N Mon Nov 17 19:14:33 2003Kernel Core Dumps
Caution
When a specific module's operating system (OS) crashes, it is sometimes useful to obtain a full copy of the memory image (called a kernel core dump) to identify the cause of the crash. When the module experiences a kernel core dump it triggers the proxy server configured on the supervisor. The supervisor sends the module's OS kernel core dump to the Cisco MDS 9000 System Debug Server. Similarly, if the supervisor OS fails, the supervisor sends its OS kernel core dump to the Cisco MDS 9000 System Debug Server.
Note
Kernel core dumps are only useful to your technical support representative. The kernel core dump file, which is a large binary file, must be transferred to an external server that resides on the same physical LAN as the switch. The core dump is subsequently interpreted by technical personnel who have access to source code and detailed memory maps.
Tip
To configure the external server, follow these steps:
Step 1
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2
switch(config)# kernel core target 10.50.5.5
succeeded
Configures the external server's IP address.
To configure the module information, follow these steps:
All changes made to kernel cores may be viewed using the show running-config command. Alternatively, use the show kernel cores command to view specific configuration changes (see Example 50-15 to Example 50-17).
Example 50-15 Displays the Core Limit
switch# show kernel core limit2Example 50-16 Displays the External Server
switch# show kernel core target10.50.5.5Example 50-17 Displays the Core Settings for the Specified Module
switch# show kernel core module 5module 5 core is enabledlevel is headerdst_ip is 10.50.5.5src_port is 6671dst_port is 6666dump_dev_name is eth1dst_mac_addr is 00:00:0C:07:AC:01Online System Health Management
The Online Health Management System (system health) is a hardware fault detection and recovery feature. It ensures the general health of switching, services, and supervisor modules in any switch in the Cisco MDS 9000 Family.
The system health application runs on all Cisco MDS modules and monitors system hardware in a given MDS switch. The system health application running in the standby supervisor module only monitors the standby supervisor module—if that module is available in the HA standby mode.
See the "HA Switchover Characteristics" section on page 8-2.
The system health application launches a daemon process in all modules and runs multiple tests on each module to test individual module components. The tests run at pre-configured intervals, cover all major fault points, and isolate any failing component in the MDS switch. The system health running on the active supervisor maintains control over all other system health components running on all other modules in the switch.
On detecting a fault, the system health application attempts the following recovery actions:
•
•
•
•
•
•
•
•
•
Each module is configured to run the test relevant to that module. You can change the default parameters of the test in each module as required.
System Health Inititation
By default, the system health feature is enabled in each switch in the Cisco MDS 9000 Family.
To disable or enable this feature in any switch in the Cisco MDS 9000 Family, follow these steps:
Loopback Test Configuration Frequency
Loopback tests are designed to identify hardware errors in the data path in the module(s) and the control path in the supervisors. One loopback frame is sent to each module at a preconfigured frequency—it passes through each configured interface and returns to the supervisor module.
The loopback tests can be run at frequencies ranging from 5 seconds (default) to 255 seconds. If you do not configure the loopback frequency value, the default frequency of 5 seconds is used for all modules in the switch. Loopback test frequencies cannot be altered for each module. The configured value is constant for all modules.
To configure the frequency of loopback tests for all modules in any switch in the Cisco MDS 9000 Family, follow these steps:
Hardware Failure Action
The failure-action command controls the Cisco SAN-OS software from taking any action if a hardware failure is determined while running the tests.
By default, this feature is enabled in all switches in the Cisco MDS 9000 Family—action is taken if a failure is determined and the failed component is isolated from further testing.
Failure action is controlled at individual test levels (per module), at the module level (for all tests), or for the entire switch.
To configure failure action in a switch, follow these steps:
Test Run Requirements
Enabling a test does not guarantee that a test will run.
Tests on a given interface or module only run if you enable system health for all of the following items:
•
•
•
Tip
Tip
Tests for a Specified Module
The system health feature in the SAN-OS software performs tests in the following areas:
•
•
•
•
•
•
•
•
•
To perform the required test on a specific module, follow these steps:
Clearing Previous Error Reports
You can clear the error history for Fibre Channel interfaces, iSCSI interfaces, for an entire module, or one particular test for an entire module. By clearing the history, you are directing the software to retest all failed components that were previously excluded from tests.
If you previously enabled the failure-action option for a period of time (for example, one week) to prevent OHMS from taking any action when a failure is encountered and after that week you are now ready to start receiving these errors again, the you must clear the system health error status for each test.
Tip
Use the EXEC-level system health clear-errors command at the interface or module level to erase any previous error conditions logged by the system health application. The battery-charger, the bootflash, the cache-disk, the eobc, the inband, the loopback, and the mgmt test options can be individually specified for a given module.
The following example clears the error history for the specified Fibre Channel interface:
switch# system health clear-errors interface fc 3/1The following example clears the error history for the specified module:
switch# system health clear-errors module 3The following example clears the management port test error history for the specified module:
switch# system health clear-errors module 1 mgmtPerforming Internal Loopbacks
Internal loopback tests send and receive FC2 frames to/from the same ports and provides the round trip time taken in microseconds. These tests are available for both Fibre Channel and iSCSI interfaces.
Use the EXEC-level system health internal-loopback command to explicitly run this test on demand (when requested by the user) within ports for the entire module.
switch# system health internal-loopback interface iscsi 8/1Internal loopback test on interface iscsi8/1 was successful.Round trip time taken is 79 useconds
Note
External loopback test on interface fc 7/2 failed. Failure reason: Failed to loopback, analysis complete Failed device ID 3 on module 1
Performing External Loopbacks
External loopback tests send and receive FC2 frames to/from the same port. You need to connect a cable (or a plug) to loop the Rx port to the Tx port before running the test. This test is only available for Fibre Channel interfaces.
Use the EXEC-level system health external-loopback command to run this test on demand for external devices connected to a switch that is part of a long-haul network.
switch# system health external-loopback interface fc 3/1This will shut the requested interfaces Do you want to continue (y/n)? [n] yExternal loopback test on interface fc3/1 was successful.Use the system health external-loopback interface force command to shut down the required interface directly without a back out confirmation.
switch# system health external-loopback interface fc 3/1 forceExternal loopback test on interface fc3/1 was successful.
Note
External loopback test on interface fc 7/2 failed. Failure reason: Failed to loopback, analysis complete Failed device ID 3 on module 1
Interpreting the Current Status
The status of each module or test depends on the current configured state of the OHMS test in that particular module (see Table 50-1).
The status of each test in each module is visible when you display any of the show system health commands. See the "Displaying System Health" section.
Displaying System Health
Use the show system health command to display system-related status information (see Example 50-18 to Example 50-23).
Example 50-18 Displays the Current Health of All Modules in the Switch
switch# show system healthCurrent health information for module 2.Test Frequency Status Action-----------------------------------------------------------------Bootflash 5 Sec Running EnabledEOBC 5 Sec Running EnabledLoopback 5 Sec Running Enabled-----------------------------------------------------------------Current health information for module 6.Test Frequency Status Action-----------------------------------------------------------------InBand 5 Sec Running EnabledBootflash 5 Sec Running EnabledEOBC 5 Sec Running EnabledManagement Port 5 Sec Running Enabled-----------------------------------------------------------------Example 50-19 Displays the Current Health of a Specified Module
switch# show system health module 8Current health information for module 8.Test Frequency Status Action-----------------------------------------------------------------Bootflash 5 Sec Running EnabledEOBC 5 Sec Running EnabledLoopback 5 Sec Running Enabled-----------------------------------------------------------------Example 50-20 Displays Health Statistics for All Modules
switch# show system health statisticsTest statistics for module # 1------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------Bootflash Running 5s 12900 12900 0 0 0EOBC Running 5s 12900 12900 0 0 0Loopback Running 5s 12900 12900 0 0 0------------------------------------------------------------------------------Test statistics for module # 3------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------Bootflash Running 5s 12890 12890 0 0 0EOBC Running 5s 12890 12890 0 0 0Loopback Running 5s 12892 12892 0 0 0------------------------------------------------------------------------------Test statistics for module # 5------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------InBand Running 5s 12911 12911 0 0 0Bootflash Running 5s 12911 12911 0 0 0EOBC Running 5s 12911 12911 0 0 0Management Port Running 5s 12911 12911 0 0 0------------------------------------------------------------------------------Test statistics for module # 6------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------InBand Running 5s 12907 12907 0 0 0Bootflash Running 5s 12907 12907 0 0 0EOBC Running 5s 12907 12907 0 0 0------------------------------------------------------------------------------Test statistics for module # 8------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------Bootflash Running 5s 12895 12895 0 0 0EOBC Running 5s 12895 12895 0 0 0Loopback Running 5s 12896 12896 0 0 0------------------------------------------------------------------------------Example 50-21 Displays Statistics for a Specified Module
switch# show system health statistics module 3Test statistics for module # 3------------------------------------------------------------------------------Test Name State Freq(s) Run Pass Fail CFail Errs------------------------------------------------------------------------------Bootflash Running 5s 12932 12932 0 0 0EOBC Running 5s 12932 12932 0 0 0Loopback Running 5s 12934 12934 0 0 0------------------------------------------------------------------------------Example 50-22 Displays Loopback Test Statistics for the Entire Switch
switch# show system health statistics loopback-----------------------------------------------------------------Mod Port Status Run Pass Fail CFail Errs1 16 Running 12953 12953 0 0 03 32 Running 12945 12945 0 0 08 8 Running 12949 12949 0 0 0-----------------------------------------------------------------Example 50-23 Displays Loopback Test Statistics for a Specified Interface
switch# show system health statistics loopback interface fc 3/1-----------------------------------------------------------------Mod Port Status Run Pass Fail CFail Errs3 1 Running 0 0 0 0 0-----------------------------------------------------------------
Note
Example 50-24 Displays the Loopback Test Time Log for All Modules
switch# show system health statistics loopback timelog-----------------------------------------------------------------Mod Samples Min(usecs) Max(usecs) Ave(usecs)1 1872 149 364 2223 1862 415 743 5498 1865 134 455 349-----------------------------------------------------------------Example 50-25 Displays the Loopback Test Time Log for a Specified Module
switch# show system health statistics loopback module 8 timelog-----------------------------------------------------------------Mod Samples Min(usecs) Max(usecs) Ave(usecs)8 1867 134 455 349-----------------------------------------------------------------Default Settings
Table 50-2 lists the default system health and log settings.
Table 50-2 Default System Health and Log Settings
Kernel core generation
One module
System health
Enabled
Loopback frequency
5 seconds
Failure action
Enabled.
