Field Notice: FN - 64029 - Cisco Business Edition 6000/7000 Series, RAID Settings Cause Severe Performance Issues - Configuration Change Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Products Affected

Defect Information

Problem Description

Due to an incorrect factory script instruction on units shipped between January 1, 2015 and July 29, 2015, the RAID controller configuration settings were not set correctly on these appliances:

• BE6000M based on UCS C220 M4S
• BE6000H based on UCS C220 M4S
• BE7000M based on UCS C240 M4S2
• BE7000H based on UCS C240 M4SX

Note: These appliances are NOT affected by this Field Notice:

• BE6000S based on the Integrated Services Router (ISR) 2900 and Unified Computing System E-Series M2
• BE6000M based on UCS C200 M2 or UCS C220 M3S
• BE6000H based on UCS C220 M3S
• BE7000M based on UCS C240 M3S

Background

Appliances based on the Cisco Unified Computing System C-Series M4 (UCS C-Series M4) use a different RAID controller model than previous hardware generations, which require a different CLI syntax for the configuration settings. Cisco has determined that the correct settings were not properly translated into the factory automation scripts for UCS C-Series M4 hardware setup for affected BE6000 and BE7000 series appliances models.

Problem Symptom

Appliances with incorrect RAID settings might experience slow read and write IO performance to local storage. The results of the performance impact might be seen in Cisco Unified Communications Manager under these conditions:

• Fresh installs, upgrades, backups, or call detail recording that takes several hours longer than expected.
• Dropped calls that occur during high call volumes, upgrades, backups, or call detail recording.

Workaround/Solution

The correct settings are: Read Policy = Read Ahead Always Write Policy = Write Back with BBU Strip Size = 128KB An appliance with settings different from those shown might need to have those settings changed. Cisco recommends all deployments change the Read Ahead Policy and Write Cache Policy settings via Cisco Integrated Management Controller (CIMC), which requires a few minutes to make the changes on a live server and will take effect immediately after you make the change. Servers with older firmware levels will require a graceful shutdown and reboot to make the changes. For the procedure, see Option A in the "Solution" section. As most customers are not at the threshold of performance limits, Option A is expected to resolve issues for the majority of customers. These settings optimize storage performance, but otherwise do not impact server hardware configuration, installed Cisco Collaboration applications, or virtual machine configurations. Cisco only recommends that you change the Strip Size for deployments that continue to experience issues after Option A has been implemented. A change to the Strip Size requires a rebuild of the RAID logical volume and restore of the applications. For the procedure, see Option B in the "Solution" section.

Solution

Option A - for all deployments, complete these steps: (NOTE THAT ALL PROCEDURES MUST BE PERFORMED DURING A MAINTENANCE WINDOW)

1. Ensure system backups are up-to-date (such as from the Disaster Recovery Solution for Unified Communications Manager and ESXi configuration for virtualization software).
2. Change the Write Policy: Managing Storage Adapters,click the Storage tab, click the Virtual Drive Info tab, and for each virtual disk Edit Virtual Drive (as described in the Cisco UCS C-Series Servers Integrated Management Controller GUI Configuration Guide).
3. In order to change the Read Policy, check your Cisco IMC Firmware Running Version (from CIMC, click the Admin tab, Firmware Management).
   • Version 2.0(4c) or later: Managing Storage Adapters, click the Storage tab, Virtual Drive Info tab, and for each virtual disk Edit Virtual Drive (as described in the Cisco UCS C-Series Servers Integrated Management Controller GUI Configuration Guide).
   • Earlier than Version 2.0(4c):
     1. Gracefully shut down applications (such as, for Cisco Unified Communications Manager either enter the utils system shutdown CLI command or power down the VM from ESXi.)
     2. Launch CIMC KVM Console
     3. Either reboot the server from VMware vSphere Client, or Shut Down Server from either the CIMC KVM Console or from CIMC, click the Server tab, Summary, Actions, Shut Down Server.
     4. As host reboots, press Ctrl-R when prompted in order to enter RAID Controller BIOS Configuration Utility.
     5. For each virtual drive, scroll to and choose the drive, choose Advanced, and set the Read Policy to Ahead.
     6. Press the Esc key in order to confirm. When prompted to reboot, choose Warm Boot from the CIMC KVM menu.

Option B - only for deployments that continue to experience issues after you complete Option A, complete these steps: (NOTE THAT ALL PROCEDURES MUST BE PERFORMED DURING A MAINTENANCE WINDOW)

1. Ensure system backups are up-to-date (such as from the Disaster Recovery Solution for Unified Communications Manager and ESXi configuration for virtualization software).
2. Locate and copy/paste your VMware vSphere ESXi license serial number. This can be found in VMware vSphere Client > Configuration > Licensed Features.
3. Locate your VMware vSphere ESXi installer file (filename Vmware-ESXi-5.5.0-2068190-custom-Cisco-5.5.2.3.iso). This can be copied from the appliance's local datastore, or downloaded from vmware.com with a free account.
4. If vMotion is available, move the VM(s) to an alternate VMware host; otherwise power down the VM(s) and copy the entire folder(s) that corresponds to the VM(s) off of the local host datastore.
   1. Rebuild the RAID logical volume with correct Strip Size, per instructions in the "Configure RAID with GUI (M4 servers)" section of Cisco Collaboration on Virtual Servers.
   2. Reinstall the VMware vSphere ESXi and restore ESXi configuration via vSphere CLI.
   3. Reupload the VMware vSphere ESXi license serial number.
   4. Replace the application Virtual Machines (VMs):
      • If vMotion is available, move the VM(s) from the alternate VMware host back onto this server.
      • If VM(s) were powered down and their folder(s)/files copied, then copy them back to the datastore of this host. Add the VM(s) back to the vSphere Inventory. In each folder, right-click the VMX file and choose add to inventory. When you start the VM(s), if the system asks you if the VM was moved or copied, choose moved.
      • If neither vMotion nor copying of VM files was completed, reinstall applications with a restore from backup.

How To Identify Affected Products

In Cisco Integrated Management Controller (CIMC), choose Server > Summary. The serial number of the server is listed in the Server Properties pane of the Server Summary window.

Once you have identified the serial number of the server, use the Serial Number Validation Tool in order to determine if your appliance is impacted.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.