Field Notice: FN72223 - MDS 9148S, MDS 9396S and MDS 9250i Switches That Run on NX-OS Releases 8.4(2), 8.4(2a), 8.4(2b), 9.2(1) or 9.2(2) Might Spontaneously Reload - Software Upgrade Recommended

Available Languages

Updated:October 6, 2023

Document ID:FN72223

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Affected Product Name	Description	Comments
DS-C9148S-12PK9	MDS 9148S 16G FC switch, w/ 12 active ports
DS-C9148S-48PK9	MDS 9148S 16G FC switch, w/ 48 active ports
DS-C9148S-CD12P8K9	MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs
DS-C9148S-CD12PSK9	MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs
DS-C9148S-D12P8K9	MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs
DS-C9148S-D12PSK9	MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs
DS-C9148S-D12S-ENT	MDS 9148S w/ 12 active ports + 16G SFP + Ent License
DS-C9148S-D24P8K9	MDS 9148S 16G FC switch, w/ 24 active ports + 8G SW SFPs
DS-C9148S-D24PSK9	MDS 9148S 16G FC switch, w/ 24 active ports + 16G SW SFPs
DS-C9148S-D48P8K9	MDS 9148S 16G FC switch, w/ 48 active ports + 8G SW SFPs
DS-C9148S-D48PSK9	MDS 9148S 16G FC switch, w/ 48 active ports + 16G SW SFPs
DS-C9148S-D8P8K9	MDS 9148S 16G FC switch, w/ 8 active ports + 8G SW SFPs
DS-C9148S-K9	MDS 9148S 16G FC 48-port switch (w/ 12 ports active)
DS-C9250I-22PK9	MDS 9250i 20-port FC, 2-port FCIP/iSCSI
DS-C9250I-D12S-ENT	MDS 9250i w/ 12 active ports + 16G SFP + Ent License
DS-C9250I-K9	MDS 9250i 50 port switch base config(20xFC, 8xFCoE, 2xFCIP)
DS-C9250I-K9++	MDS 9250i 50 port switch base config(20xFC, 8xFCoE, 2xFCIP)
DS-C9250ID16GSFPK9	MDS 9250i 50 port switch base config bundle,20x16G FC optics
DS-C9250ID8GSFPK9	MDS 9250i 50 port switch base config bundle,20x8G FC optics
DS-C9250IH-K9	MDS 9250i 50 port switch HP config(20xFC, 8xFCoE, 2xFCIP)
DS-C9396S-48E8K9	^MDS 9396S, w/ 48 active ports + 8G SFPs (port-side exhaust)
DS-C9396S-48EK9	^MDS 9396S switch, w/ 48 active ports (port-side exhaust)
DS-C9396S-48ESK9	^MDS 9396S, w/ 48 active ports + 16G SFPs (port-side exhaust)
DS-C9396S-48IK9	^MDS 9396S switch, w/ 48 active ports (port-side intake)
DS-C9396S-96E8K9	^MDS 9396S, w/ 96 active ports + 8G SFPs (port-side exhaust)
DS-C9396S-96EK9	^MDS 9396S switch, w/ 96 active ports (port-side exhaust)
DS-C9396S-96ESK9	^MDS 9396S, w/ 96 active ports + 16G SFPs (port-side exhaust)
DS-C9396S-96IK9	^MDS 9396S switch, w/ 96 active ports (port-side intake)
DS-C9396S-K9	^MDS 9396S HW base (48 ports active)
UCS-EP-MDS9148S-1	MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs
UCS-EP-MDS9148S-16	MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs
UCS-EP-MDS9396S-16	^Cisco MDS 9396S 16G Switch; 48 enabled ports, 48x16G SW SFP

Defect Information

Defect ID	Headline
CSCvw91665	MDS crashes with "Anon_Resident_Mem 0 KB being killed due to lack of memory"

Problem Description

A small memory leak occurs at each log in to a Multilayer Data Switch (MDS) supervisor that runs Cisco MDS NX-OS Releases 8.4(2), 8.4(2a), 8.4(2b), 9.2(1), or 9.2(2) only. The supervisor will reboot when the memory pool is exhausted.

Only MDS 9148S and MDS 9250i platforms have been reported to reach complete memory exhaustion and reboot. These platforms are single supervisor switches and the supervisor reboot causes the whole switch to disruptively reload.

Background

This issue is caused by a small leak in one of the memory pools on the supervisor each time a new process is started. Most services in Cisco NX-OS (such as Fibre Channel Name Server (FCNS), SNMP, and Zone) are started at boot time and run continuously. The most common reason processes are started during switch operation is when a log in occurs. This creates several processes, which triggers this memory leak.

MDS 9148S has the smallest memory pool. An idle MDS 9148S (with only irregular interactive log ins used for occasional switch maintenance) might run for one to two years without rebooting. However, a switch with frequent log ins per hour will be significantly affected. This can occur when a monitoring platform or script automatically logs in to the switch at high frequency. This can lead to exhaustion in as little as two weeks for an MDS 9148S.

Although reboots due to this issue have been reported from the field only for MDS 9148S and MDS 9250i switches, all MDS platforms are subject to this memory leak and theoretically could eventually reboot. Practically, this is not likely since the rate of log ins is limited by NX-OS and the memory pool is many times bigger on the other platforms. This has been verified by lab testing.

Problem Symptom

The switch logs the "Proc ... being killed due to lack of memory" syslog and subsequently reloads. These syslogs from the show logging nvram CLI command can be used to identify this issue:

%KERN-0-SYSTEM_MSG: [18590637.381749] LR = 0xb7c78908 - kernel %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 3609) hasn't caught signal 9 (no core). %KERN-1-SYSTEM_MSG: [18590637.520017] Proc swapper/0 (0) with Total_VM 0 KB Resident_Mem 0 KB Anon_Resident_Mem 0 KB being killed due to lack of memory - kernel

Note: The name of the crashed service and PID ("snmpd" and "PID 3609" in the previous example) are random and not important.

In order to identify if a system is affected, enter the show system internal kernel meminfo CLI command to display the current memory usage.


switch#  show system internal kernel meminfo
MemTotal:        4088568 kB
MemFree:         2732056 kB
MemAvailable:    2834504 kB
Buffers:            6136 kB
Cached:           517872 kB
SwapCached:            0 kB
Active:           654628 kB
Inactive:         250816 kB
Active(anon):     542076 kB
Inactive(anon):   250124 kB
Active(file):     112552 kB
Inactive(file):      692 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       3604476 kB
HighFree:        2384812 kB
LowTotal:         484092 kB
LowFree:          347244 kB
...

On MDS 9148S, MDS 9250i, and MDS 9396S platforms, monitor the ratio of the "LowFree" field to the "LowTotal" field. On all other MDS platforms, monitor the "MemFree" and "MemTotal" fields.

Once the "LowFree" or "MemFree" becomes severely depleted (that is, free memory becomes less than 20% of the total memory), recovery action should be taken. This must be done before the free memory reaches 0% in order to prevent an unscheduled, disruptive reload.

Workaround/Solution

Upgrade the switch software to a fixed version to clear any current symptoms and prevent this issue from recurring.

Note: Some platforms are affected by more than one Field Notice. Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address each individual Field Notice.

Field Notice:	FN72223	FN72237	FN72346
Summary:	Memory leak reload	FCNS issue after ISSU	Machine check reload
Affected MDS Platforms:	9148S 9250i 9396S	9132T 9148S 9148T 9220i 9396S 9396T	9396S 9700 32 Gbps FC module
Cisco MDS NX-OS train	Fixed Cisco MDS NX-OS Versions
8.4 train:	8.4(2c) and later	9220i: None¹ others: 8.4(2d) and later	8.4(2d) and later
8.5 train:	8.5(1)	None	8.5(1)
9.2 train:	9.2(1a)	9.2(1a) and 9.2(2)	None
9.3 train:	9.3(1) and later	9.3(1)and later	9.3(1)and later

¹ Cisco MDS 9220i is not supported in the 8.4 train.

Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address FN72223, FN72237, and FN72346 by platform.

	Minimum Cisco MDS NX-OS Version to Resolve FN72223, FN72237, and FN72346
Cisco MDS NX-OS train	9700 32 Gbps FC module	9396T	9148T	9132T	9396S	9148S	9250i	9220i
8.4 train:	8.4(2d)	8.4(2d)	8.4(2d)	8.4(2d)	8.4(2d)	8.4(2d)	8.4(2c)	None ¹
8.5 train:	8.5(1)	None	None	None	None	None	8.5(1)	None
9.2 train:	None	9.2(1a) and 9.2(2)	9.2(1a) and 9.2(2)	9.2(1a) and 9.2(2)	None	9.2(1a)	9.2(1a)	9.2(1a) and 9.2(2)
9.3 train:	9.3(1)	9.3(1)	9.3(1)	9.3(1)	9.3(1)	9.3(1)	9.3(1)	9.3(1)

¹ Cisco MDS 9220i is not supported in the 8.4 train.

In order to resolve this issue, perform an In Service Software Upgrade (ISSU) or In Service Software Downgrade (ISSD) to a fixed release of Cisco NX-OS. This can be any release of Cisco NX-OS except these affected releases: 8.4(2), 8.4(2a), 8.4(2b), 9.2(1), or 9.2(2). See MDS Software Downloads for a list of software downloads.

In order to temporarily recover from this situation on MDS Fabric switches, proactively execute a nondisruptive reload with the reload system non-disruptive CLI command. On MDS Director switches, perform a nondisruptive supervisor switchover with the system switchover CLI command. Alternatively, an ISSU (or ISSD) to another release of NX-OS on all platform types will also reclaim the leaked memory.

How to Identify Affected Products

In order to verify which release currently runs on Cisco NX-OS, enter the show version CLI command. The relevant release is shown in the "system" field:


switch# show version
...
Software
  BIOS:      version 2.1.17
  loader:    version N/A
  kickstart: version 8.4(2)
  system:    version 8.4(2)

Revision History

Version	Description	Section	Date
1.6	Updated the release numbers for the 9.2 train.	Workaround/Solution	2023-OCT-06
1.5	Updated the Products Affected Section	—	2022-JUL-07
1.4	Updated the Workaround/Solution Section	—	2022-JUN-14
1.3	Updated the Problem Description and Workaround/Solution Sections	—	2022-MAY-20
1.2	Updated the Background Section	—	2021-SEP-02
1.1	Updated the Workaround/Solution Section	—	2021-AUG-10
1.0	Initial Release	—	2021-AUG-05

For More Information

For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:

Receive Email Notification About New Field Notices

To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)