Field Notice: FN74426 - Certain Drives Used in Cisco UCS and HyperFlex Platforms May Experience Performance Issues After 7.5 Years of Run Time - Software Upgrade Recommended

Certain Cisco UCS and HyperFlex platforms may experience performance degradation due to an issue that affects specific serial ATA (SATA) solid-state drives (SSDs). This issue affects SATA SSDs that have reached or exceeded 65,535 power-on hours of operation. Affected units may exhibit a significant decrease in read/write performance, ranging from 50 percent to more than 90 percent, depending on the current workload.

The performance degradation is caused by a firmware limitation involving the internal workload log entry counter on the drive. When the internal LogEntryCnt of the device reaches 65,535, continuous messages are written to the internal debug log during each subsequent update cycle. This background activity consumes controller CPU resources that are otherwise reserved for processing I/O requests, resulting in the observed performance degradation.

Affected SATA SSDs may exhibit a sustained reduction in read/write performance after reaching 65,535 power-on hours (approximately 7.5 years). The observed performance degradation can range from 50 percent to more than 90 percent, depending on the workload.

High read/write latency of affected drives can disrupt Zookeeper transactions in Cisco HyperFlex clusters, potentially leading to cluster outages and data loss.

The SSD remains operational, and no data loss, drive failure, or firmware asserts are observed. The impact is limited to reduced read/write performance.

Workaround

Power cycle the affected drive or system to temporarily restore normal read/write performance.

The performance degradation can recur approximately one hour after the drive is powered back on. A subsequent power cycle can temporarily restore performance again.

Solution

To address this issue, upgrade drives using the appropriate fixed firmware release, as listed in the following table:

The following releases are available in the Cisco Software Download Center:

• UCS B-Series Blade Server Software Release 4.2(3s)
• UCS C220 M5 Rack Server Software Release 4.2(3r)

The following table lists product identifiers (IDs) and the latest firmware version that fixes this issue:

A drive is affected by this issue if all the following conditions are met:

• The drive has a PID that is listed in the Products Affected section of this field notice.
• The drive has accumulated 65,535 or more power-on hours.
• The drive is not running the fixed firmware release — D0MH079, D0MH449, D0MH849, D0MC078, D0MC448 — specified for its PID.

Drives that are running the applicable fixed firmware versions are not affected by this issue, regardless of their accumulated power-on hours.

Use one of the following methods to identify Micron 5100 Series drives and determine the installed firmware version.

Cisco Intersight

1. Sign in to Cisco Intersight.
2. Navigate to Operate > Servers.
3. Choose the applicable server.
4. Choose the Inventory tab.
5. Expand Storage Controllers.
6. Select the applicable storage controller and physical drive.
7. Review the following fields:
   • Model: Confirm that the value begins with Micron_5100_, such as Micron_5100_MTFDDAK240TCB.
   • PID: Confirm that the PID is listed in the Products Affected section.
   • Disk Firmware Version: Compare this value with the applicable fixed firmware version specified for the PID.
8. Repeat these steps for each physical drive that is installed in the server.

Note: The following screenshot was captured in a lab environment and is provided as an example only. Server names, storage controllers, drive PIDs, models, firmware versions, and other displayed values can vary depending on the system configuration. Use the values displayed for the applicable production system.

Cisco UCS Manager CLI

1. Connect to the Cisco UCS Manager CLI.
2. Scope to the applicable server by using the scope server <server-id> command.
3. Display the RAID controllers associated with the server by using the show raid-controller command. The following is an example of the command output:
   

   RAID Controller:
   ID Type PCI Addr
   1 SAS 24:00.0
   7 SATA 00:23.0
   8 PCH 00:17.5

4. From the output, identify the applicable controller ID.
5. Scope to the applicable controller by using the command scope raid-controller <controller-id> <type>.
6. Display the installed drive model by using show local-disk detail command as shown. (Note: show local-disk detail | grep Micron_5100 command can be used to display the Model information only.) 
7. Review the following fields:
   • Model: Confirm that the value begins with Micron_5100_, such as Micron_5100_MTFDDAK240TCB.
   • PID: Confirm that the PID is listed in the Products Affected section.
   • Running-Vers: Compare this value with the applicable fixed firmware version specified for the PID.
8. Repeat these steps for each server and applicable storage controller. The server and controller IDs can vary between systems.

Note : The following command output is from a lab environment and is provided as an example only. Server IDs, RAID controller IDs, drive models, and other displayed values can vary depending on the system configuration. Use the values reported by the applicable production system.

Cisco Integrated Management Controller (CIMC)

1. Sign in to the CIMC web interface for the applicable server.
2. Choose Storage and select the applicable RAID controller.
3. Choose the Physical Drive Info tab.
4. Choose the applicable physical drive, such as PD-1, from the Physical Drives list.
5. Expand Inquiry Data and review the following fields:
   • Product ID: Confirm that the value begins with Micron_5100_, such as Micron_5100_MTFDDAK240TCB.
   • UCS Product ID: Confirm that the PID is listed in the Products Affected section of this field notice.
   • Drive Firmware: Compare this value with the applicable fixed firmware version that is specified for the PID in the Workaround/Solution section of the field notice.
6. The drive is affected if its PID is listed in the Products Affected section and it is not running the applicable fixed firmware version.
7. Repeat these steps for each physical drive installed in the server.

Note: The following screenshot was captured in a lab environment and is provided as an example only. RAID controller names, physical-drive identifiers, drive PIDs, models, firmware versions, power-on hours, and other displayed values can vary depending on the server and storage configuration. Use the values displayed for the applicable production system.

Cisco Hyperflex

1. SSH into the Storage Controller VM (SCVM). 
2. Retrieve the list of Micron_5100 drives by using the following command: grep -ai "Micron_5100*" /var/log/springpath/diskslotmap-v2.txt
3. The format of the output will be as follows: <Stol Number>:<Serial Number>:<Drive Model>:<Serial Number><Firmware Version>:<Type>. The following is an example of the output:
   

   1.0.1:500a0751241c6a0e:MICRON:Micron_5100_MTFDDAK240TCB:1940241C6A0E:D0MH077:SATA: