Recovering from Upgrade Failures

Upgrade Failures

The following table describes some possible upgrade errors that you may encounter and what you must do to recover from them.

Table 1 Upgrade Failures

Symptom

Possible Cause

Recommended Action

Failed or unsuccessful upgrade on a bare-metal server.

Attempted upgrade of the controller is being made without meeting the system requirements for the release.

Access the latest Cisco APIC-EM release notes and review the system requirements. Be sure to review the appropriate specific system requirements for a bare-metal upgrade.

Try to upgrade the controller again by taking the following steps:

  1. If necessary, reinstall your previous version of controller software on your server.

  2. Restore the controller database and files from the backup file that you created and saved to a secure location on your network. See Back Up the Controller Database and Files.

  3. Attempt the upgrade a second time.

If failure persists, contact Cisco support. See the Cisco Application Policy Infrastructure Controller Enterprise Module Troubleshooting Guide, for Cisco TAC contact information.

Failed or unsuccessful upgrade on a virtual machine.

Attempted upgrade of the controller is being made without meeting the system requirements for the release.

Access the latest Cisco APIC-EM release notes and review the system requirements. Be sure to review the appropriate specific system requirements for a virtual machine upgrade, including the VMware resource pool requirements.

Try to upgrade the controller again by taking the following steps:

  1. If necessary, reinstall your previous version of controller software on your virtual machine.

  2. Restore the controller database and files from the backup file that you created and saved to a secure location on your network. See Back Up the Controller Database and Files.

  3. Attempt the upgrade a second time.

If failure persists, contact Cisco support. See the Cisco Application Policy Infrastructure Controller Enterprise Module Troubleshooting Guide, for Cisco TAC contact information.

Failed or unsuccessful upgrade on a virtual machine.

Error messages on controller indicate that there is an issue with the NTP server.

When upgrading the Cisco APIC-EM in a virtual machine within a VMware vSphere environment, you must ensure that the time settings on the ESXi host are also synchronized to the NTP server. Failure to ensure synchronization will cause the upgrade to fail.

Try to upgrade the controller again by taking the following steps:

  1. If necessary, reinstall your previous version of controller software on your virtual machine.

  2. Restore the controller database and files from the backup file that you created and saved to a secure location on your network. See Back Up the Controller Database and Files.

  3. If the NTP server settings are not synchronized, use SSH to log into the controller, run the reset_grapevine command and update the NTP server settings.

  4. Attempt the upgrade a second time.

If failure persists, contact Cisco support.

See the Cisco Application Policy Infrastructure Controller Enterprise Module Troubleshooting Guide, for information about using the reset_grapevine command , as well as Cisco TAC contact information.

Failed or unsuccessful upgrade on either a bare-metal server or virtual machine.

Error messages on the controller's GUI indicate that some services are failing to start up on the Cisco APIC-EM after the upgrade.

Try to upgrade the controller again by taking the following steps:

  1. If necessary, reinstall your previous version of controller software on your appliance, server, or virtual machine.

  2. Restore the controller database and files from the backup file that you created and saved to a secure location on your network. See Back Up the Controller Database and Files.

  3. Attempt the upgrade a second time.

If failure persists, take the following actions:

  • If possible, log into the controller's GUI.

  • Review the status of the services in the System Health tab and note any failed services.

  • Proceed to create an rca file, see Creating a Support File for an Upgrade Failure.

  • Send the rca file with any information about failed services displayed in the controller's GUI to support for additional assistance.

See the Cisco Application Policy Infrastructure Controller Enterprise Module Troubleshooting Guide, for information about the above steps, as well as Cisco TAC contact information.

Creating a Support File for an Upgrade Failure

You can troubleshoot a Cisco APIC-EM upgrade failure by creating a root cause analysis (rca) support file. The rca file consists of logs, configuration files, and command output. After you create this rca file, you can then email it to Cisco support for assistance.

    Step 1   Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
    Note   

    The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

    Step 2   When prompted, enter your Linux username ('grapevine') and password for SSH access.
    Step 3   Navigate to the bin directory on the host. The bin directory contains the grapevine scripts.
    Step 4   To create the support file, enter the rca command in this directory. 
    $ rca
mkdir: created directory '/tmp grapevine-rca-2016-08-05_16-22-20-PM_PDT-0700'

----------------------------------------------------------------------
RCA package created On Tues August 5 16:22:20 PDT 2016
----------------------------------------------------------------------

    The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output.

    Note   

    For a multi-host deployment (three hosts), you need to perform this procedure and run the rca command on each of the three hosts. Additionally, if you need to perform a root cause analysis for a specific service, then to identify which host the service is running on, access the controller's GUI, click admin, click System Administration from the drop down menu, and click Hosts at the top of the window to view the individual hosts and their services.

    What to Do Next

    Send the tar file or files created by this procedure to Cisco support for assistance in resolving your upgrade issue.