THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
|Affected Product ID
||i2c bus contention causing network interrupt issue|
Content Delivery Engine 250 (CDE250) servers might encounter Baseboard Management Controller (BMC)/Harmonics chipset I2C bus contention which causes a network interrupt issue.
CDE250 systems utilize an Intelligent Platform Management Interface (IPMI) infrastructure that monitors and manages the health of the system. This is implemented with embedded IPMI firmware that runs within the BMC system chip on the motherboard. The current version of IPMI firmware used on the CDE250 is v3.12. This firmware version has the possibility of a BMC hang due to I2C bus contention. As a result of this contention, interrupts are not generated in time which causes delays in read and write function. This delay leads to numerous disk IOs and eventual depletion of memory which results in a crash.
IPMI firmware v9.15 has been released in order to address this issue.
There are multiple symptoms for this issue, as listed here:
- Too many disk IOs scheduled and the device continuously allocates memory, which eventually causes an ASSERT.
[2285838.309428] WARNING: A disk drive is behind schedule by 9543ms
[2285838.318571] Unreachable Streamer 951(6) (40 Vaults, 25 Streamers and 3 Caches online)
[2285838.415189] CMM::Descriptor::allocateArray tot=5585427 avail=14 count=16
[2285838.498016] CALYPSO ASSERT: ""CMM::Descriptor::allocateArray: cannot free enough blocks"" failed in file "cmm/Descriptor.cpp"
- A primary server conflict or primary server switchover.
[2202534.711317] Detected primary server conflict with server 526-
[2202534.711413] Winner has the most recent setup command
[2202534.711414] The remote server will remain primary
[2202534.721866] Server 525 says it is Available Setup and Control for 10.27.77.1
[2202534.803593] AVSLauncher request to shutdown setup and control applications on 10.27.77.109 succeeded
[2202534.803595] Server 526 says it is Primary Setup and Control for 10.27.77.109
[2202535.041137] Stopped being primary during synchronization to backup for service address 10.27.77.109
- The server becomes unresponsive.
Cisco recommends that these steps be taken in order to update the IPMI for VDS-TV, VQE, and VDS-IS applications and the SDT utility for VDS-TV and VD-IS applications. This update needs to be completed during a maintenance window.
||IPMI Firmware Version
||FW version = 9.15 or later (SDT v2.113 required)
||Unit is good, no action required
||FW version = 1.33 or 2.05 or 3.03, 3.06 or 3.09, 3.12
||Upgrade IPMI FW
- Determine the IPMI firmware version (see the How to Identify Hardware Levels section).
Note: The new IPMI firmware requires SDT version 2.113.
- If the IPMI version is not 9.15, update the IPMI firmware to version 9.15 with the IPMI dedicated Ethernet port and web interface or a DOS boot stick with the IPMIUP.BAT (yafukcs -full X8DAH915.ima).
- Power-cycle the system by removing and re-inserting the power cable after 10 seconds.
- Verify the IPMI firmware is updated successfully.
For further details, refer to the IPMI Firmware v9.15 and SDT 2.113 Upgrade instructions.
For the VDS-IS application, patches have been developed in order to upgrade the SDT version 2.113 as it is required for the new IPMI firmware.
||3.12 and later
||All customers that use version 3.x up to the latest 3.3.1b138 image
||Apply the corresponding patch in VDS-IS to upgrade the SDT version to 2.113
||3.12 and later
||All customers that use version 4.x up to the latest 4.3.2-b28 image
||Apply corresponding patch in VDS-IS to upgrade the SDT version to 2.113
Update the VDS-IS Application with the Script
- Unzip the required IPMI_VDS-IS_3x_SDT-2.113_Patch.zip / IPMI_VDS-IS_4x_SDT-2.113_Patch.zip file.
- Copy all the files to the CDE250. Make sure all the files are copied to the location "local/local1".
- Use these commands in order to check and execute the script:
script check ipmi_script.sh
script execute ipmi_script.sh
- Once the patch is executed properly, reload the box.
Refer to the Release Notes and IPMI Firmware downloads for further details:
How To Identify Affected Products
The hardware information can be obtained either with a CLI command or a physical inspection of the chassis. The CLI command as shown in this screenshot must be used in order to determine the IPMI firmware version.
CDE Product ID Information
In order to identify the affected chassis, check the Product ID (PID) located on the right front corner of the top of the chassis.
This label is located on top right front corner of the chassis.
For More Information
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
Receive Email Notification For New Field Notices
Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.