Field Notice: FN - 64211 - MDS9K - Traffic Interruption Due To Fibre Channel Interface Link Up/Down Events - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	21-Oct-16	Initial Release
10.0	11-Oct-17	Migration to new field notice system

Products Affected

Affected OS Type	Affected Release	Affected Release Number	Comments
NON-IOS	6	6.2(15)

Defect Information

Defect ID	Headline
CSCuz90357	'pmon' service crash

Problem Description

In some cases the process (pmon service) responsible for the port-monitoring function on Multilayer Data Switch (MDS) Fibre Channel switching modules terminates unexpectedly. At first, this event might not cause any operational impact and the process normally restarts automatically. However, if the issue recurs multiple times it might eventually lead to severe traffic impact on the entire MDS switch. The trigger for this issue is excessive Fibre Channel interface link up/down events. At least 500 or more events have to occur before the problem occurs. Intially, the effects will be limited to the module(s) where one or more interfaces experience such events, but might later spread to the entire switch. Some common causes of link up/down events are:

- Physical layer issues

- Links that experience heavy congestion

- Host reboots

Background

This problem was initally identified in internal testing and is documented in Cisco bug ID CSCuz90357.

Problem Symptom

Dependent upon the generation of Fibre Channel switching modules used, one or more messages similar to this example might be observed in the switch syslog:

# show logging log

%SYSMGR-SLOT4-2-CORE_SAVE_FAILED: master_core_client_try_spawn: PID 1358 with message Unable to start core client. Cannot allocate memory.

%SYSMGR-SLOT4-3-SYSMGR_CRASHED: Sysmgr (PID 1358) has terminated on receiving signal 6

%PROC_MGR-SLOT1-2-ERR_MSG: ERROR: PID 1216 (pmon) killed with signal (6)

%NTP-2-NTP_SYSLOG_NO_RESP_FROM_LC: from LC1 for Timestamp Update

%PROC_MGR-SLOT1-4-PROC_MGR_APP_NOT_RESP: ERROR: Process (lc_port_cfg) is not responding, it will be terminated

%MODULE-2-MOD_MINORSWFAIL: Module 1 (Serial number: JAF1623APRF) reported a failure in service lc_port_cfg

%PROC_MGR-SLOT1-2-ERR_MSG: ERROR: PID 1181 (lc_port_cfg) killed with signal (6)

One or more cores from processes pmon and/or sysmgr and/or lc_port_cfg might be present on the switch as shown in this example.

# show core

Module Instance Process-name PID Date(Year-Month-Day Time)

------ -------- --------------- -------- -------------------------

4 1 sysmgr 1358 2016-09-27 03:42:40

1 1 pmon 1216 2016-09-28 11:36:40

1 1 lc_port_cfg 1181 2016-09-28 16:32:03

A Fibre Channel switching module might reload or enter into the powered-dn state.

Workaround/Solution

It is recommended that customers upgrade to MDS NX-OS Release 6.2(17) or later. NX-OS Release 6.2(17) was made available on Wednesday, July 13, 2016.

If you want to run NX-OS Release 6.2(15), it is recommended that you monitor these MDS switches and ensure that there are no excessive Fibre Channel interface up/down events (500 or more).

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.