Field Notice: FN - 70379 - Unified Communications Collaboration Platform(s) - Reboot of Node Results in Fatal Error - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	22-Apr-19	Initial Release
2.0	15-May-19	Updated the Products Affected and Workaround/Solution Sections
3.0	13-Mar-20	Updated the Defect Information and Workaround/Solution Sections

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	Unified Communications Manager / Cisco Unity Connection Updates	UCM v11	11.0(1), 11.0(1a), 11.0(1a)SU1, 11.0(1a)SU2, 11.0(1a)SU3, 11.0(1a)SU3a, 11.0(1a)SU4	Affects Cisco Unified Communications Manager, Cisco Session Management Edition, Cisco Unity Connection, and Cisco Unified Communications Manager IM & Presence
NON-IOS	Unified Communications Manager / Cisco Unity Connection Updates	UCM v10	10.5(1), 10.5(1)SU1, 10.5(1)SU1a, 10.5(2), 10.5(2)SU1, 10.5(2)SU2, 10.5(2)SU2a, 10.5(2)SU3, 10.5(2)SU3a, 10.5(2)SU4, 10.5(2)SU4a, 10.5(2)SU5, 10.5(2)SU6, 10.5(2)SU6a	Affects Cisco Unified Communications Manager, Cisco Session Management Edition, Cisco Unity Connection, and Cisco Unified Communications Manager IM & Presence
NON-IOS	Unified Presence Server (CUP) Updates	11.5(1)	11.5(1), 11.5(1)SU1, 11.5(1)SU2, 11.5(1)SU3, 11.5(1)SU3a, 11.5(1)SU4, 11.5(1)SU5, 11.5(1)SU5a	Affects Cisco Unified Communications Manager, Cisco Session Management Edition, Cisco Unity Connection, and Cisco Unified Communications Manager IM & Presence
NON-IOS	Unified Communications Manager / Cisco Unity Connection Updates	UCM	11.5(1), 11.5(1)SU1, 11.5(1)SU2, 11.5(1)SU3, 11.5(1)SU3a, 11.5(1)SU3b, 11.5(1)SU4, 11.5(1)SU5	Affects Cisco Unified Communications Manager, Cisco Session Management Edition, Cisco Unity Connection, and Cisco Unified Communications Manager IM & Presence
NON-IOS	Emergency Responder Updates	CER 12.0	12.0(1) SU1	Affected release number includes 12.0(1) for Cisco Emergency Responder
NON-IOS	Emergency Responder Updates	CER 11.5	11.5(1), 11.5(2), 11.5(2a), 11.5(3), 11.5(4), 11.5(4)SU2, 11.5(4a)
NON-IOS	Unified Communications Manager Updates	UCM	12.0(1)SU1	Affected release number includes 12.0(1) for Cisco Unified Communications Manager, Cisco Session Management Edition, and Cisco Unified Communications Manager IM & Presence
NON-IOS	Unity Connection Updates	12.0	12.0(1)SU1	Affected release number includes 12.0(1) for Cisco Unity Connection

Defect Information

Defect ID	Headline
CSCvb21486	Reboot of node may result in FATAL error
CSCvh55176	After reboot, server stuck at "Probing EDD (edd=off to disable)... ok" or a black screen with cursor
CSCvm52977	reboot VM console shows FATAL: Could not load /lib/modules/2.6.32-573.18.1.el16.x86_64/modules.dep:
CSCvk55340	CER Reboot of node may result in FATAL error
CSCvq17528	After reboot, server stuck at "Probing EDD (edd=off to disable)... ok" or a black screen with cursor

Problem Description

After a Unified Communications server node is rebooted for any given reason, it will not boot the operating system (OS) back up. A reboot or power cycle of the server node will not boot the server node back up nor restore its services. The server node will remain offline and Applications Services will not be reinitialized.

Background

Cisco has identified an issue where, after a successful VMware Tools upgrade, the initramfs image located in the boot partition is corrupted in varying degrees. This corruption of the initramfs image does not lead to a VMware Tools upgrade failure and the Unified Communications application continues to operate without exhibiting any issues. However, the next time the server node is rebooted for any reason the Linux BootLoader fails to load the initramfs image, which prevents the OS from booting back up.

Problem Symptom

After a planned or unplanned restart of a Cisco Unified Communications server node, its Virtual Machine Console will be stuck on a blank screen or display one or both of these errors:

• Probing EDD (edd=off to disable)… ok
• FATAL: Could not load /lib/modules/2.6.32-573.18.1.el16.x86_64/modules.dep: No such file or directory

Workaround/Solution

Workaround for Cisco Unified Communications Manager, Unity Connection, Cisco Unified Communications Manager IM & Presence, and Session Management Edition

Customers can use these instructions in order to rebuild the initramfs image file. It is recommended that customers perform a full system backup prior to any server reboots.

1. Boot up the server with the applicable Recovery CD.
2. Once the Recovery options appear, press ALT+F2. This takes you to a bash shell.
3. Enter these two commands from the bash shell:
   • ls -la /mnt/part1
   • ls -la /mnt/part2

   The folder with the newest files in it identifies the active partition. Take note of which folder contains the active partition.

4. Change the root folder to the active partition.

   chroot /mnt/part1 or part2 (whichever one is identified as the active partition)

5. Put selinux into permissive mode.

   setenforce 0

6. Have vmware-tools regenerate the initramfs image file.

   /usr/bin/vmware-config-tools.pl -d

   Note: The output of this command shows an error, which is normal and can be ignored.

   Creating a new initrd boot image for the kernel.
Initctl: Unable to connect to Upstart:Failed to connect to socket /com/Ubuntu/upstart: connection refused
Unable to start services for VMware Tools
Execution aborted.
[anaconda root@localhost /]#

7. Put selinux back to enforcing mode.

   setenforce 1

8. Exit the bash shell.

   exit

9. Disconnect the Recovery CD.
10. Reboot the virtual machine.

Workaround for Cisco Emergency Responder

Customers can use these instructions in order to rebuild the initramfs image file. It is recommended that customers perform a full system backup prior to any server reboots.

1. Boot up the server with the applicable Recovery CD.
2. Once the Recovery options appear, press ALT+F2. This takes you to a bash shell.
3. Enter these two commands from the bash shell:
   • ls -la /mnt/part1
   • ls -la /mnt/part2

   The folder with the newest files in it identifies the active partition. Take note of which folder contains the active partition.

4. Change the root folder to the active partition.

   chroot /mnt/part1 or part2 (whichever one is identified as the active partition)

5. Put selinux into permissive mode.

   setenforce 0

6. Have vmware-tools regenerate the initramfs image file.

   /usr/bin/vmware-uninstall-tools.pl

   Note: The output of this command shows an error, which is normal and can be ignored.

   Uninstalling the tar installation of VMware Tools.
Stopping services for vmware-tools
Initctl: Unable to connect to Upstart:Failed to connect to socket /com/Ubuntu/u pstart: connection refused
The removal of VMware Tools 9.10.5 build-2981885 for Linux completed
successfully. Thank you for having tried this software.

7. Put selinux in back in to enforcing mode.

   setenforce 1

8. Exit the bash shell.

   exit

9. Disconnect the Recovery CD.
10. Reboot the virtual machine.

Solution

Customers are recommended to upgrade to the latest version. However, in order to prevent the issue from occuring prior to a recommended upgrade, customers can install ciscocm.CSCvq17528_vmtools_initramfs_v1.2.cop.sgn on each node in the cluster.

Note: Applying this COP will not resolve the issue after it has been encountered, but will prevent its occurrence.

Cisco highly recommends that customers upgrade to these Unified Communications application releases:

• Cisco Emergency Responder Release 12.0(1)SU2 or later
• Cisco Emergency Responder Release 12.5(1a) or later
• Cisco Unified Communications Manager / Session Management Edition Release 10.5(2)SU7 or later
• Cisco Unified Communications Manager / Session Management Edition Release 11.5(1)SU6 or later
• Cisco Unified Communications Manager / Session Management Edition Release 12.0(1)SU2 or later
• Cisco Unified Communications Manager / Session Management Edition Release 12.5(1) or later
• Cisco Unified Communications Manager IM & Presence Release 11.5(1)SU6 or later
• Cisco Unified Communications Manager IM & Presence Release 12.5(1) or later
• Cisco Unity Connection Release 10.5(2)SU7 or later
• Cisco Unity Connection Release 11.5(1)SU6 or later 
• Cisco Unity Connection Release 12.0(1)SU2 or later
• Cisco Unity Connection Release 12.5(1) or later

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.