Getting Started Guide for Cisco UCS E-Series Servers, Release 2.x
Recovering from Corrupt CIMC Firmware
Downloads: This chapterpdf (PDF - 1.32MB) The complete bookPDF (PDF - 5.75MB) | The complete bookePub (ePub - 2.02MB) | Feedback

Recovering from Corrupt CIMC Firmware

This chapter includes the following sections:

CIMC Firmware Image Overview

If you have problems booting the E-Series Server, it could be that the CIMC firmware image is corrupted, or the SD card is faulty, or the file system is corrupted, or the CIMC firmware installation did not complete successfully. To recover from a corrupt CIMC firmware image, do one of the following as appropriate:
Important:

Due to security considerations, the boot backup command is disabled.

Recovering from a Corrupted CIMC Firmware Image

Before You Begin
  • Connect the E-Series Server to your PC. Depending on the type of E-Series Server, do one of the following as appropriate:
    • Double-wide—Connect one end of the serial cable to the E-Series Server's serial port and the other end to your PC.
    • Single-wide—First, connect a KVM connector to the E-Series Server's KVM port; and then connect one end of a serial cable to the DB9 port of the KVM connector and the other end to your PC.
  • Attach an Ethernet cable into the Management (dedicated) port of the E-Series Server.
  • To view the serial output, start the Hyper Terminal or Minicom as appropriate. Do one of the following:
    • Microsoft Windows—Start Hyper Terminal.
    • Linux—Start Minicom.
  • Make sure that the communications settings are configured as: 9600 baud, 8 bits, No parity, and 1 stop bit.

Procedure
      Command or Action Purpose
    Step 1 Router# hw-module sm slot oir-stop  

    Shuts down the power to the specified E-Series Server.

    Note   

    The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

     
    Step 2 Router# hw-module sm slot oir-start  

    Restarts the specified E-Series Server.

    Note   

    The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

     
    Step 3 ***
     

    From the Hyper Terminal or Minicom, enter the *** command to enter the bootloader prompt.

     
    Step 4 ucse-cimc > boot current recovery  

    Boots the E-Series Server from the current image.

     
    Step 5 Recovery-shell # dedicated-interface management-interface-ip-address netmask gateway-ip-address  

    Specifies the IP address, subnet mask, and the gateway IP address of the E-Series Server's Management (dedicated) interface.

     
    Step 6 Recovery-shell # ping tftp-ip-address  

    Pings the remote TFTP server in which the CIMC firmware is located to verify network connectivity.

     
    Step 7 Recovery-shell # update tftp-ip-address image-filename  

    Installs the CIMC firmware image, which is located on a remote tftp server.

     
    Step 8 Recovery-shell # reboot  

    Reboots CIMC.

     

    This example recovers the CIMC firmware image:

    Router# hw-module subslot 2/0 stop
    Router# hw-module subslot 2/0 start 
    
    ***
    
    ucse-cimc > boot current recovery 
    recovery-shell# dedicated-interface 192.168.0.138 255.255.255.0 192.168.0.1
    Network configuration:
            IP config: addr: 192.168.0.138 Mask: 255.255.255.0
            Gateway: 192.168.0.1
    recovery-shell# ping 10.20.34.56
    PING 10.20.34.56 (10.20.34.56): 56 data bytes
    64 bytes from 10.20.34.56: seq=0 ttl=60 time=10.000 ms
    64 bytes from 10.20.34.56: seq=1 ttl=60 time=0.000 ms
    --- 10.20.34.56 ping statistics ---
    10 packets transmitted, 10 packets received, 0% packet loss
    round-trip min/avg/max = 0.000/1.000/10.000 ms
    recovery-shell# update 10.20.34.56 update_pkg-cimc.combined.bin
    downloading firmware image "update_pkg-cimc.combined.bin" from " 10.20.34.56 "
    download firmware image done, size in bytes: 22384144
    installing firmware image, please wait ...
    activating installed image
    done
    Stage: NONE
    Status: SUCCESS
    Error: Success
    recovery-shell# reboot
    

    Recovering from a Faulty SD Card

    If you have problems booting the E-Series Server, it could be because the SD card is faulty. Use this procedure to recover the CIMC firmware image on a new SD card.
    Before You Begin
    • Connect the E-Series Server to your PC. Depending on the type of E-Series Server, do one of the following as appropriate:
      • Double-wide—Connect one end of the serial cable to the E-Series Server's serial port and the other end to your PC.
      • Single-wide—First, connect a KVM connector to the E-Series Server's KVM port; and then connect one end of a serial cable to the DB9 port of the KVM connector and the other end to your PC.
    • Attach an Ethernet cable into the Management (dedicated) port of the E-Series Server.
    • To view the serial output, start the Hyper Terminal or Minicom as appropriate. Do one of the following:
      • Microsoft Windows—Start Hyper Terminal.
      • Linux—Start Minicom.
    • Make sure that the communications settings are configured as: 9600 baud, 8 bits, No parity, and 1 stop bit.

    Procedure
        Command or Action Purpose
      Step 1 Router# hw-module sm slot oir-stop  

      Shuts down the power to the specified E-Series Server.

      Note   

      The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

       
      Step 2 Remove the faulty SD card and insert a new one.  

      Replaces the faulty SD card.

       
      Step 3 Router# hw-module sm slot oir-start  

      Restarts the specified E-Series Server.

      Note   

      The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

       
      Step 4 ***
       

      From the Hyper Terminal or Minicom, enter the *** command to enter the bootloader prompt.

       
      Step 5 ucse-cimc > boot current recovery  

      Boots the E-Series Server from the current image.

       
      Step 6 Recovery-shell # dedicated-interface management-interface-ip-address netmask gateway-ip-address  

      Specifies the IP address, subnet mask, and the gateway IP address of the E-Series Server's Management (dedicated) interface.

       
      Step 7 Recovery-shell # ping tftp-ip-address  

      Pings the remote TFTP server in which the CIMC firmware is located to verify network connectivity.

       
      Step 8 Recovery-shell # update tftp-ip-address image-filename  

      Installs the CIMC firmware image, which is located on a remote tftp server.

       
      Step 9 Recovery-shell # reboot  

      Reboots CIMC.

       

      This example recovers the CIMC firmware from the current image:

      Router# hw-module subslot 2/0 stop
      Router# hw-module subslot 2/0 start  
      
      ***
      
      ucse-cimc > boot current recovery 
      recovery-shell# dedicated-interface 192.168.0.138 255.255.255.0 192.168.0.1
      Network configuration:
              IP config: addr: 192.168.0.138 Mask: 255.255.255.0
              Gateway: 192.168.0.1
      recovery-shell# ping 10.20.34.56
      PING 10.20.34.56 (10.20.34.56): 56 data bytes
      64 bytes from 10.20.34.56: seq=0 ttl=60 time=10.000 ms
      64 bytes from 10.20.34.56: seq=1 ttl=60 time=0.000 ms
      --- 10.20.34.56 ping statistics ---
      10 packets transmitted, 10 packets received, 0% packet loss
      round-trip min/avg/max = 0.000/1.000/10.000 ms
      recovery-shell# update 10.20.34.56 update_pkg-cimc.combined.bin
      downloading firmware image "update_pkg-cimc.combined.bin" from " 10.20.34.56 "
      download firmware image done, size in bytes: 22384144
      installing firmware image, please wait ...
      activating installed image
      done
      Stage: NONE
      Status: SUCCESS
      Error: Success
      recovery-shell# reboot
      

      Recovering from a Corrupted File System

      Use this procedure if you see the following error message in the CIMC boot log files.

      UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY
      Before You Begin
      • Connect the E-Series Server to your PC. Depending on the type of E-Series Server, do one of the following as appropriate:
        • Double-wide—Connect one end of the serial cable to the E-Series Server's serial port and the other end to your PC.
        • Single-wide—First, connect a KVM connector to the E-Series Server's KVM port; and then connect one end of a serial cable to the DB9 port of the KVM connector and the other end to your PC.
      • Attach an Ethernet cable into the Management (dedicated) port of the E-Series Server.
      • To view the serial output, start the Hyper Terminal or Minicom as appropriate. Do one of the following:
        • Microsoft Windows—Start Hyper Terminal.
        • Linux—Start Minicom.
      • Make sure that the communications settings are configured as: 9600 baud, 8 bits, No parity, and 1 stop bit.

      Procedure
          Command or Action Purpose
        Step 1 Router# hw-module sm slot oir-stop  

        Shuts down the power to the specified E-Series Server.

        Note   

        The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

         
        Step 2 Router# hw-module sm slot oir-start  

        Restarts the specified E-Series Server.

        Note   

        The Cisco 2900 series ISR G2 does not support OIR of the E-Series Servers. To avoid damaging the router, turn off the electrical power on the router and disconnect network cables before inserting or removing the E-Series Server from the Cisco 2900 ISR G2.

         
        Step 3 ***
         

        From the Hyper Terminal or Minicom, enter the *** command to enter the bootloader prompt.

         
        Step 4 ucse-cimc > boot current recovery  

        Boots the E-Series Server from the current image.

         
        Step 5 To check the file system of the specified partition and recover the corrupted file system, enter these commands. 
        1. Recovery-shell # fs-check [p3 | p4]
          Note   

          You can only use p3 and p4 partitions with this command. Use this command on the partition that is corrupted. The corrupted partition is the one that displays the run fsk error message during CIMC bootup.

        2. Do the following:
          • If the command output displays clean, it indicates that the corrupted files are recovered. Enter the reboot command to reboot CIMC.
            Note   

            Skip the steps that follow.

          • If the command output does not display clean, proceed to Step 6.
         
        Step 6 (Optional) If the fs-check [p3 | p4] command does not recover the corrupted file system, and the output does not display clean, enter these commands to format the partitions. 
        1. Recovery-shell # sd-card format [p3 | p4] Formats the specified corrupted partition on the SD card.
          Note   

          The corrupted partition is the one that displays the run fsk error message during CIMC bootup.

        2. Recovery-shell # reboot Reboots CIMC.
          Note   

          Skip the steps that follow.

          Note   

          When the p3 partition is formatted, the CIMC configuration is lost.

         
        Step 7 (Optional) If the sd-card format [p3 | p4] command does not recover the corrupted file system, enter these commands to partition and format the SD card. 
        1. Recovery-shell # sd-card partition Creates partitions on the SD card.
        2. Recovery-shell # sd-card format p3 Formats the p3 partition on the SD card.
        3. Recovery-shell # sd-card format p4 Formats the p4 partition on the SD card.
        4. Recovery-shell # reboot Reboots CIMC.
        5. (Optional) Recovery-shell # sd-partition show Displays the current partition on the SD card.
        Note   

        When you partition the SD card, the contents of the SD card, such as, the configuration and ISO file, are lost.

         
        Step 8 Recovery-shell # dedicated-interface management-interface-ip-address netmask gateway-ip-address  

        Specifies the IP address, subnet mask, and the gateway IP address of the E-Series Server's Management (dedicated) interface.

         
        Step 9 Recovery-shell # ping tftp-ip-address  

        Pings the remote TFTP server in which the CIMC firmware is located to verify network connectivity.

         
        Step 10 Recovery-shell # update tftp-ip-address image-filename  

        Installs the CIMC firmware image, which is located on a remote tftp server.

         
        Step 11 Recovery-shell # reboot  

        Reboots CIMC.

         

        This example recovers the CIMC firmware from the current image using the fs-check p3 command:

        Router# hw-module sm 2 oir-stop
        Router# hw-module sm 2 oir-start 
        
        ***
        
        ucse-cimc > boot current recovery 
        recovery-shell# fs-check p3
        e2fsck 1.41.14 (22-Dec-2010)
        /dev/mmcblk0p3: recovering journal
        /dev/mmcblk0p3: clean, 429/7840 files, 3331/31296 blocks
        recovery-shell# fs-check p4
        e2fsck 1.41.14 (22-Dec-2010)
        /dev/mmcblk0p4: clean, 51/506912 files, 1880262/2025296 blocks
        recovery-shell# reboot