Chapter 1. Overview

About the Cisco Application Policy Infrastructure Controller Enterprise Module

Cisco APIC-EM Components and Architecture

Appliances

Hosts

Grapevine

Root and Clients

Services

Databases

Networks

Network Connections and NICs

Chapter 2. Troubleshooting Matrix

Troubleshooting Matrix

Chapter 3. Troubleshooting an Unsuccessful Installation or Update

Confirming that Core Services are Running

Confirming the Multi-Host Cluster Configuration Values

Resolving Access to the Cisco APIC-EM GUI

Updating Cisco APIC-EM Using the Apply Update Script

Updating Cisco APIC-EM Using the Apply Update Script (Releases 1.0.2.8, 1.0.3.4)

Chapter 4. Troubleshooting the Configuration

Updating the Configuration Using the Wizard

Resetting the Cisco APIC-EM

Restoring the Controller to the Factory Default

Creating a Support File for the Cisco APIC-EM

Chapter 5. Troubleshooting Cisco APIC-EM Multi-Host

Changing the Settings in a Multi-Host Cluster

Removing a Single Host from a Multi-Host Cluster

Removing a Faulted Host from a Multi-Host Cluster

Chapter 6. Troubleshooting Services Using System Health

Reviewing the Service Version and Status Using the System Health Tab

Removing a Service Instance Using the System Health GUI

Creating a Service Instance Using the System Health GUI

Chapter 7. Troubleshooting Services Using the Developer Console

Grapevine Developer Console

Logging into the Grapevine Developer Console

Reviewing the Service Version, Status, and Logs

Removing a Service Instance Manually

Creating a Service Instance Manually

Reviewing the User Logs

Monitoring Services and Clients Using the CLI

Chapter 8. Troubleshooting Using the Logs

Viewing Audit Logs

Changing the Logging Level

Downloading the Service Logs

Chapter 9. Troubleshooting Passwords

Performing Password Recovery with an Existing Administrator

Performing Password Recovery with No Existing Administrator

Performing Password Recovery for the Linux Grapevine User Account

Chapter 10. Troubleshooting Commands

Root Commands

Client Commands

Chapter 11. Troubleshooting Log Files

Root Log Files

Client Log Files

Chapter 12. Contacting the Cisco Technical Assistance Center

Contacting the Cisco Technical Assistance Center

Preface

Audience

Document Conventions

Related Documentation

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. Overview

- About the Cisco Application Policy Infrastructure Controller Enterprise Module

- Cisco APIC-EM Components and Architecture

About the Cisco Application Policy Infrastructure Controller Enterprise Module

The Cisco Application Policy Infrastructure Controller - Enterprise Module (APIC-EM) is Cisco's SDN Controller for Enterprise Networks (Access, Campus, WAN and Wireless).

The platform hosts multiple applications (SDN apps) that use open northbound REST APIs that drive core network automation solutions. The platform also supports a number of south-bound protocols that enable it to communicate with the breadth of network devices that customers already have in place, and extend SDN benefits to both greenfield and brownfield environments.

The Cisco APIC-EM platform supports both wired and wireless enterprise networks across the Campus, Branch and WAN infrastructures. It offers the following benefits:

- Creates an intelligent, open, programmable network with open APIs

- Saves time, resources, and costs through advanced automation

- Transforms business intent policies into a dynamic network configuration

- Provides a single point for network wide automation and control

The following table describes the features and benefits of the Cisco APIC-EM.

Cisco APIC Enterprise Module Features and Benefits

Feature

Description

Network Information Database (NIDB)

The Cisco APIC-EM periodically scans the network to create a “single source of truth” for IT. This inventory includes all network devices, along with an abstraction for the entire enterprise network.

Network topology visualization

The Cisco APIC-EM automatically discovers and maps network devices to a physical topology with detailed device-level data. You can use this interactive feature to troubleshoot your network.

EasyQoS

The EasyQoS feature enables you to configure quality of service on the devices in your network that have been discovered by the Cisco APIC-EM.

Using EasyQoS, you can group devices and then define the business relevance of applications that are used in your network. The Cisco APIC-EM takes your QoS selections, translates them into the proper command line interface (CLI) commands, and deploys them onto the selected devices.

Cisco Network Plug and Play application

The Cisco Network Plug and Play solution is a converged solution that extends across Cisco's enterprise portfolio. It provides a highly secure, scalable, seamless, and unified zero-touch deployment experience for customers across Cisco routers, switches and wireless access points.

Cisco Intelligent WAN (IWAN) application

The separately licensed IWAN application for APIC-EM simplifies the provisioning of IWAN network profiles with simple business policies. The IWAN application defines business-level preferences by application or groups of applications in terms of the preferred path for hybrid WAN links. Doing so improves the application experience over any connection and saves telecom costs by leveraging cheaper WAN links.

Public Key Infrastructure (PKI) server

The Cisco APIC-EM provides an integrated PKI service that acts as Certificate Authority (CA) to automate X.509 SSL certificate lifecycle management. Applications, such as IWAN and PnP, use the capabilities of the imbedded PKI service for automatic SSL certificate management.

Path Trace application

The path trace application helps to solve network problems by automating the inspection and interrogation of the flow taken by a business application in the network.

High Availability (HA)

HA is provided in N+ 1 redundancy mode with full data persistence for HA and Scale. All the nodes work in Active-Active mode for optimal performance and load sharing.

Back Up and Restore

The Cisco APIC-EM supports complete back up and restore of the entire database from the controller GUI.

Audit Logs (IWAN)

The Cisco APIC-EM provides a direct link to the IWAN Audit Logs, which allows you to view Cisco APIC-EM- and IWAN-related log entries.

Cisco APIC-EM Components and Architecture

The Cisco APIC-EM consists of the components and architecture discussed in this section. To better troubleshoot any issues with the Cisco APIC-EM, you should review the topics in this section.

- Appliances

- Hosts

- Grapevine

- Root and Clients

- Services

- Databases

- Networks

- Network Connections and NICs

Appliances

The Cisco APIC-EM can be deployed on either a physical or virtual appliance.

- For physical appliance support, the Cisco APIC-EM can be installed on the following Cisco UCS servers:

- Cisco APIC-EM has been tested and qualified to run on these servers.

- Cisco UCS C220 M4 Server

- Cisco UCS C220 M3S Server

- Cisco UCS C22 M3S Server

- Any Cisco UCS server that meets the minimum system requirements as listed in the Release Notes for the Cisco Application Policy Infrastructure Controller Enterprise Module.

- A physical appliance can also be a dedicated Cisco UCS server. The following two physical appliances are currently available:

- Cisco APIC-EM Controller Appliance 10C-64G-2T (APIC-EM-APL-R-K9)

- Cisco APIC-EM Controller Appliance 20C-128G-4T (APIC-EM-APL-G-K9)

Note

Contact Cisco support for additional information about the above appliances and for ordering information.

- For virtual appliance support, the Cisco APIC-EM can also be installed and deployed in a virtual machine that meets the minimum system requirements on VMware vSphere. For information about these requirements, see the Release Notes for the Cisco Application Policy Infrastructure Controller Enterprise Module

Hosts

You can set up either a single host or multi-host deployment for your network. A host is defined as an appliance, physical server, or virtual machine running instances of the Grapevine clients. The Grapevine root itself runs directly on the host's operating system. You can set up either a single host or multi-host deployment. A multi-host deployment with three hosts is best practice for both high availability and scale. Each Grapevine root in a multi-host configuration maintains an Active/Active status with the other Grapevine roots and is therefore able to coordinate with the other Grapevine roots the overall management of the cluster.

Note

For additional information about a multi-host deployment, see the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide.

Grapevine

The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network, using Grapevine as an Elastic Services platform to support the controller's infrastructure and services. The Grapevine root and clients are key components of this infrastructure.

Note

You can use a tool called the Grapevine developer console to troubleshoot roots and clients running services. The Grapevine developer console was bundled with the deployment files and installed when you first deployed the Cisco APIC-EM. You can access the Grapevine developer console at this controller IP address and port number: https://<controller IP address>:14141

Root and Clients

The Grapevine root handles all policy management in regards to service updates, as well as the service lifecycle for both itself and the Grapevine client. The Grapevine client is where the supported services run.

For a list of the supported services for this release, see the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide.

Note

You can remotely log into the root using SSH (Secure Shell) to troubleshoot any issues. A default idle timeout of 1 hour has been set for an SSH console login. You will be automatically logged out after 1 hour of inactivity on the SSH console.

Services

The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when demand increases, and frees instances of itself when demand decreases.

For a list of the supported services for this release, see the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide.

Databases

The Cisco APIC-EM supports two databases: application and Grapevine. The application database is used for the application and external networking data. The Grapevine database is used for the Grapevine and internal network data. Both databases are replicated in a multi-host environment for scale and high availability.

Networks

The Cisco APIC-EM architecture requires both external and internal networks to operate:

- The external network(s) consists of the network hosts, devices, and NTP servers, as well as providing access to the northbound REST APIs. The external network(s) also provides access to the controller GUI.

- The internal network consists of the Grapevine roots and clients that are connected to and communicate with each other (service to service). For forwarding to or receiving traffic from the larger external network (that consists of the connected devices and hosts, as well as NTP servers), all inbound and outbound traffic for this internal network passes through a subset of clients connected to the external network. The internal network is isolated and nonroutable from the external network(s), as well as any other internal network.

Network Connections and NICs

The network adapters (NICs) on the host (physical or virtual) are connected to the following external networks:

- Internet (network access required for Make A Wish requests, Telemetry, and trustpool updates)

- Network with NTP server(s)

- Network with devices that are to be managed by the Cisco APIC-EM

Note

The Cisco APIC-EM should never be directly connected to the Internet. It should not be deployed outside of a NAT configured or protected datacenter environment.

Chapter 2. Troubleshooting Matrix

- Troubleshooting Matrix

Troubleshooting Matrix

The following table lists solutions for the most common issues that relate to Cisco APIC-EM. Reference is made to the relevant chapters and topics throughout this guide.

Cisco APIC-EM Troubleshooting Matrix

Symptom

Possible Cause

Recommended Action

Installation Issues

Failed or unsuccessful installation on a bare-metal server.

Attempted installation of the controller is being made without meeting the system requirements for the release.

Access the latest Cisco APIC-EM release notes and review the system requirements. Be sure to review the appropriate specific system requirements for a bare-metal installation.

Failed or unsuccessful installation on a virtual machine.

Attempted installation of the controller is being made without meeting the system requirements for the release.

Access the latest Cisco APIC-EM release notes and review the system requirements. Be sure to review the appropriate specific system requirements for a virtual machine installation, including the VMware resource pool requirements.

Note:
For additional assistance with deploying the controller in a virtual machine, refer to the appendix in the latest version of the Cisco APIC-EM deployment guide that discusses virtual machine preparation.

Failed or unsuccessful installation on either a bare-metal server or virtual machine.

Core services failing to start up on the Cisco APIC-EM.

Perform the following actions:

- Log into the developer console.

- Review the core services in the developer console.

- Create an rca file and send to support for additional assistance.

References:

- Confirming that Core Services are Running

- Creating a Support File for the Cisco APIC-EM

Unable to log into the controller GUI.

Key services required to bring up the controller GUI are failing.

Log into the developer console and review the key services. If necessary, restart these services.

References:

- Resolving Access to the Cisco APIC-EM GUI

- Removing a Service Instance Manually

- Creating a Service Instance Manually

Configuration Issues

Controller in a single host configuration appears to be in an unstable state. For example, applications are not running, are inaccessible, and/or not appearing in the GUI.

Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard.

Log into the host, check the configuration values, and reenter any configuration values that are incorrect.

References:

- Updating the Configuration Using the Wizard

Controller in a multi-host configuration appears to be in an unstable state. For example, applications are not running, are inaccessible, and/or not appearing in the GUI.

Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard.

Log into the host, check the configuration values, and reenter any configuration values that are incorrect.

References:

- Changing the Settings in a Multi-Host Cluster

Controller was working fine for a period of time after its initial deployment, but then lapses into an unstable state. For example, applications are not running, are inaccessible, and/or not appearing in the GUI.

Possible user or other error in configuration values that occurs after the initial deployment.

Revert back to the first configuration or the factory default configuration.

References:

- Resetting the Cisco APIC-EM

- Restoring the Controller to the Factory Default

Controller was working fine for a multi-host configuration, but after a period of time one of the hosts becomes erratic and unstable.

Possible failed service or services in the multi-host cluster.

Remove and then reattach unstable host from the multi-host cluster.

References:

- Removing a Single Host from a Multi-Host Cluster

Controller was working fine for a multi-host configuration, but after a period of time one of the hosts fails.

Possible failed service or services in the multi-host cluster.

Remove and then reattach failed and inoperable host from the multi-host cluster.

References:

- Removing a Faulted Host from a Multi-Host Cluster

Hardware Issues

Single host (on an physical appliance or server) reboots, but the controller is not returning to former status.

Services are failing as a result of the reboot and not restarting

Log into the developer console, determine the failed service, restart the service.

Note:
You can review the status of controller services using the GUI, CLI, or developer console.

References:

- Logging into the Grapevine Developer Console

- Reviewing the Service Version, Status, and Logs

- Removing a Service Instance Manually

- Creating a Service Instance Manually

Service Issues

Service failure on a host that leads to one or more applications on the controller failing. For example, getting a "503 Service Unavailable" message when trying to access the Controller UI.

Log into the developer console, determine the failed service, restart the service.

Note:
You can review the status of controller services using the GUI, CLI, or developer console.

References:

- Logging into the Grapevine Developer Console

- Reviewing the Service Version, Status, and Logs

- Removing a Service Instance Manually

- Creating a Service Instance Manually

Password Issues

Lost password for controller access.

A user (administrator, installer or observer) has lost their password, and there exists at least one controller administrator (ROLE_ADMIN) user account.

Perform the appropriate password recovery for this scenario.

References:

- Performing Password Recovery with an Existing Administrator

Lost password for controller access.

A user (administrator, installer or observer) has lost their password and there exists only one controller administrator (ROLE_ADMIN) user account and that account cannot be successfully logged into.

Perform the appropriate password recovery for this scenario.

References:

- Performing Password Recovery with No Existing Administrator

Lost password for system access.

The Linux grapevine user password has been lost.

Perform the appropriate password recovery for this scenario.

References:

- Performing Password Recovery for the Linux Grapevine User Account

Chapter 3. Troubleshooting an Unsuccessful Installation or Update

The following procedures may be used to troubleshoot an unsuccessful installation or update:

- Confirming that Core Services are Running

- Confirming the Multi-Host Cluster Configuration Values

- Resolving Access to the Cisco APIC-EM GUI

- Updating Cisco APIC-EM Using the Apply Update Script

- Updating Cisco APIC-EM Using the Apply Update Script (Releases 1.0.2.8, 1.0.3.4)

Confirming that Core Services are Running

Before You Begin

You should have attempted to deploy the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the following command to display the status of the core services:

$ sudo service grapevine status

Step 4

Enter your password a second time when prompted.

$[sudo] password for grapevine: ********

Command output similar to the following should appear. The core services should have a RUNNING status.

grapevine is running
grapevine_beacon                 RUNNING    pid 30243, uptime 0:11:11
grapevine_capacity_manager       RUNNING    pid 30549, uptime 0:11:02
grapevine_capacity_manager_lxc_plugin RUNNING    pid 30247, uptime 0:11:11
grapevine_cassandra              RUNNING    pid 30244, uptime 0:11:11
grapevine_root                   RUNNING    pid 30537, uptime 0:11:03status

Step 5

If any of the core services are not in the RUNNING state, enter the root cause analysis (rca) command.

$ rca

The rca command runs a root cause analysis script that creates a tar file that contains the following data:

- Log files

- Configuration files

- Command output

Step 6

Send the tar file created by the rca command procedure to Cisco support for assistance in resolving your issue.

Confirming the Multi-Host Cluster Configuration Values

Before You Begin

You should have attempted to deploy the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the following command to display the multi-host configuration.

$ grape root display

Command output similar to the following should appear.

ROOT                                   PROPERTY             VALUE
----------------------------------------------------------------------

4cbe3972-9872-4771-800d-08c89463f1eb   hostname             root-1
4cbe3972-9872-4771-800d-08c89463f1eb   interfaces           [{'interface': 'eth0', 'ip': '209.165.200.10', 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1', 'ip': '209.165.200.10', 'mac': '00:50:56:95:5c:18', 'net  mask': '255.255.255.0'}, {'interface': 'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb   is_alive             True
4cbe3972-9872-4771-800d-08c89463f1eb   last_heartbeat       Wed Sep 09, 2015 11:02:52 PM (just now)

4cbe3972-9872-4771-800d-08c89463f1eb   public_key           ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root

4cbe3972-9872-4771-800d-08c89463f1eb   root_id              4cbe3972-9872-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb   root_index           0
4cbe3972-9872-4771-800d-08c89463f1eb   root_version         0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb   vm_password          ******
(grapevine)

#

ROOT                                   PROPERTY             VALUE
----------------------------------------------------------------------

4cbe3972-9872-4771-800d-08c89463f1eb   hostname             root-2
4cbe3972-9872-4771-800d-08c89463f1eb   interfaces           [{'interface': 'eth0', 'ip': '209.165.200.101, 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1', 'ip': '209.165.200.11', 'mac': '00:50:56:95:5c:18', 'net  mask': '255.255.255.0'}, {'interface': 'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb   is_alive             True
4cbe3972-9872-4771-800d-08c89463f1eb   last_heartbeat       Wed Sep 09, 2015 11:02:52 PM (just now)

4cbe3972-9872-4771-800d-08c89463f1eb   public_key           ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root

4cbe3972-9872-4771-800d-08c89463f1eb   root_id              4cbe3972-9873-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb   root_index           0
4cbe3972-9872-4771-800d-08c89463f1eb   root_version         0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb   vm_password          ******
(grapevine)

The following data is displayed by this command:

- hostname—The configured hostname.

- interfaces—The configured interface values, including Ethernet port, IP address, and netmask.

- is_alive—Status of the host. True indicates a running host, False indicates a host that has shut down.

- last_heartbeat—Date and time of last heartbeat message sent from the host.

- public_key—Public key used by host.

- root_id—Individual root identification number.

- root_index—Individual root index number.

- root_version—Software version of root.

- vm_password—VMware vSphere password that is masked.

Step 4

If any of the fields in the command output appear incorrect, enter the root cause analysis (rca) command.

$ rca

The rca command runs a root cause analysis script that creates a tar file that contains the following data:

- Log files

- Configuration files

- Command output

Step 5

Send the tar file created by the rca command procedure to Cisco support for assistance in resolving your issue.

Resolving Access to the Cisco APIC-EM GUI

You access the Cisco APIC-EM GUI by entering the IP address that you configured for the network adapter using the configuration wizard. This IP address connects to the external network. Enter the IP address in your browser in the following format:

https://IP address

If you are unable to access the Cisco APIC-EM GUI, you must access the Grapevine developer console to check for faulty or failed services.

The Grapevine developer console allows you to monitor the health of your Cisco APIC-EM deployment. The Grapevine developer console is part of the Service Elasticity Platform (Grapevine). You access the Grapevine developer console by entering the IP address that you configured for the network adapter using the configuration wizard, but with a specific port number (14141).

For a multi-host cluster, you do not have to log into each host. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts. Multiple instances of services running on different hosts will appear in the Grapevine developer console in a multi-host cluster.

Note

A default idle timeout of 1 hour has been set for the Grapevine developer console. You will be automatically logged out after 1 hour of inactivity on the Grapevine developer console.

Grapevine Developer Console

To access the Grapevine developer console to check for faulty or failed services, follow the procedure described below.

Before You Begin

You should have attempted to deploy the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Access the Grapevine developer console by opening a browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

After you enter the username and password, the Grapevine developer console appears. Each installed service with its version number appears in the console in an alphabetical list. Below each service is a square icon that represents the health of the service. Services that have been installed and are operational are green. Faulty or failed services are red.

Step 3

Scroll down the list and confirm that the following services are installed and running for your deployment:

- reverse-proxy

- router

- ui

Note whether the service is operational or faulty.

Step 4

Review the console Tasks tab below the list of services for any error messages about any faulty services. Note the reason given for the faulty or failed service.

Step 5

Contact Cisco support with the following information:

- Whether any of the services listed in Step 3 are inoperable or faulty.

- Whether any errors are in the console Tasks tab located at the bottom of the console.

Updating Cisco APIC-EM Using the Apply Update Script

When you are unable to update Cisco APIC-EM using the recommended standard methods (due to the fact that the controller's GUI is inaccessible, the appropriate grape command is not working, or any of the other methods are displaying error messages during the upload process), then use the procedure described below. This procedure involves using the apply_update script.

Note

If you are encountering errors after the upload process is completed (during the subsequent verification process or after), then running the apply_update script in this procedure will not solve the problem. This script is only provided as a workaround for issues encountered during the upload process.

Important:
The script should only be used when the recommended, standard methods to upload and update the controller are not working. This script should not be used as an alternative method.

Before You Begin

You have previously deployed Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Note

With most of the Cisco APIC-EM releases, the apply_update script is packaged with the Cisco APIC-EM itself and accessible within the host after installation. In the following releases though, you need to first download the script from the Download Software link:

- 1.0.2.8

- 1.0.3.4

For information about downloading and updating the controller on these releases with the apply_update script, see Updating Cisco APIC-EM Using the Apply Update Script (Releases 1.0.2.8, 1.0.3.4)

Step 1

Review the information in the Cisco notification about the Cisco APIC-EM upgrade.
The Cisco notification specifies the location of the release upgrade pack and verification values for either a Message Digest 5 (MD5) or Secure Hash Algorithm (SHA) 512 bits (SHA512) checksum.

Note:
The Cisco APIC-EM release upgrade pack is a bit file that varies in size based upon the requirements of the specific upgrade. The release upgrade pack can be as large as several Gigabits.

Step 2

Download the Cisco APIC-EM upgrade package from the Cisco website at the Download Software link.
The release upgrade pack is available for download as a tar file that is also compressed, so the release upgrade pack has a .tar.gz extension. The release upgrade pack itself may consist of any or all of the following update files:

- Service files

- Grapevine files

- Linux files

Note:
Each release upgrade pack contains an encrypted Cisco signature for security purposes, as well as release version metadata that validates the package.

Step 3

Run a checksum against the file using your own checksum verification tool or utility (either MD5 or SHA512).

Step 4

Review the displayed checksum verification value from your checksum verification tool or utility.
If the output from your checksum verification tool or utility matches the appropriate checksum value in the Cisco notification or from the Cisco secure website, then proceed to the next step. If the output does not match the checksum value, then download the release upgrade pack and perform another checksum. If checksum verification issues persist, contact Cisco support.

Step 5

Copy or move the file from your laptop or secure network location to the appliance, server, or virtual machine with the controller.

Step 6

Using a Secure Shell (SSH) client, log into the host (appliance, server or virtual machine) with the IP address that you specified using the configuration wizard.

Step 7

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 8

Navigate to the folder where the file is located and run the following command:

$ sudo /opt/cisco/grapevine/bin/apply_update [path-to-upgrade-file]

Note:
The script is located on /opt/cisco/grapevine/bin/apply_update, but you can run the script from anywhere on the cluster.

What to Do Next

Review the command output. If the upload is successful, then the update process will immediately follow.

If the script fails for any reason, then contact Cisco support for additional steps to take.

Updating Cisco APIC-EM Using the Apply Update Script (Releases 1.0.2.8, 1.0.3.4)

When you are unable to update Cisco APIC-EM using the recommended standard methods (due to the fact that the controller's GUI is inaccessible, the appropriate grape command is not working, or any of the other methods are displaying error messages during the upload process), then use the procedure described below. This procedure involves using the apply_update script.

Note

If you are encountering errors after the upload process is completed (during the subsequent verification process or after), then running the apply_update script in this procedure will not solve the problem. This script is only provided as a workaround for issues encountered during the upload process.

Important:
The script should only be used when the recommended, standard methods to upload and update the controller are not working. This script should not be used as an alternative method.

Before You Begin

You have previously deployed Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

With most of the Cisco APIC-EM releases, the apply_update script is packaged with the Cisco APIC-EM itself and accessible within the host after installation. In the following releases though, you need to also download the script from the Download Software link:

- 1.0.2.8

- 1.0.3.4

Step 1

Determine that your controller's Cisco APIC-EM release version is either 1.0.2.8 or 1.0.3.4.
Access the controller's GUI and review the release version on the Home page.

Important:
This procedure should only be performed on controllers running those release versions.

Step 2

Access the download page for Cisco APIC releases located at the Download Software link.

Step 3

Download the script called apply_update.

Step 4

Using a Secure Shell (SSH) client, log into the host (appliance, server or virtual machine) with the IP address that you specified using the configuration wizard.

Step 5

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 6

Using SCP or another secure method, copy the apply_update script to the Grapevine root for your cluster.

Step 7

Next, review the information in the Cisco notification about the Cisco APIC-EM upgrade.
The Cisco notification specifies the location of the release upgrade pack and verification values for either a Message Digest 5 (MD5) or Secure Hash Algorithm (SHA) 512 bits (SHA512) checksum.

Note:
The Cisco APIC-EM release upgrade pack is a bit file that varies in size based upon the requirements of the specific upgrade. The release upgrade pack can be as large as several Gigabits.

Step 8

Download the Cisco APIC-EM upgrade package from the Cisco website at the Download Software link.
The release upgrade pack is available for download as a tar file that is also compressed, so the release upgrade pack has a .tar.gz extension. The release upgrade pack itself may consist of any or all of the following update files:

- Service files

- Grapevine files

- Linux files

Note:
Each release upgrade pack contains an encrypted Cisco signature for security purposes, as well as release version metadata that validates the package.

Step 9

Run a checksum against the file using your own checksum verification tool or utility (either MD5 or SHA512).

Step 10

Review the displayed checksum verification value from your checksum verification tool or utility.
If the output from your checksum verification tool or utility matches the appropriate checksum value in the Cisco notification or from the Cisco secure website, then proceed to the next step. If the output does not match the checksum value, then download the release upgrade pack and perform another checksum. If checksum verification issues persist, contact Cisco support.

Step 11

Copy or move the file from your laptop or secure network location to the appliance, server, or virtual machine with the controller.

Step 12

Run the script on the Grapevine root with root permissions on the upgrade file. For example, run the following command:

$ sudo ./apply_update [path-to-upgrade-file]

What to Do Next

Review the command output. If the upload is successful, then the update process will immediately follow.

If the script fails for any reason, then contact Cisco support for additional steps to take.

Chapter 4. Troubleshooting the Configuration

The following procedures may be used to troubleshoot the configuration:

- Updating the Configuration Using the Wizard

- Resetting the Cisco APIC-EM

- Restoring the Controller to the Factory Default

- Creating a Support File for the Cisco APIC-EM

Updating the Configuration Using the Wizard

You can troubleshoot the Cisco APIC-EM deployment by running the configuration wizard a second time and updating any earlier configuration entries. The configuration wizard saves and displays your previous configuration settings, so you do not have to reenter them.

Note

When performing this procedure, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period. For information about changing settings for a multi-host configuration, see Changing the Settings in a Multi-Host Cluster.

Before You Begin

You have deployed Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Restart the configuration wizard using the following command.

$ config_wizard

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 4

Review the current configuration values in the configuration wizard and click next>>, until you access the specific step where you wish to update your previous configuration entry.
For example, if you need to enter a new NTP server IP address, click next >> until you get to the NTP SERVER SETTINGS screen.

Step 5

Update the value that was previously entered in the configuration wizard and is currently displayed.
For example, you can update the NTP server settings by entering a new IP address.

Step 6

Click next>> until the last step of the configuration wizard process.

Step 7

Click proceed>> to have the configuration wizard save and apply your configuration changes to your Cisco APIC-EM deployment.

Resetting the Cisco APIC-EM

You can troubleshoot a Cisco APIC-EM deployment by resetting the controller back to configuration values that were originally set using the configuration wizard the first time. A reset of the controller is helpful, when the controller has gotten itself into an unstable state and other troubleshooting activities have not resolved the situation.

Note

In a multi-host environment, you need to perform this procedure on only a single host. After performing this procedure on a single host, the other two hosts will be automatically reset.

Before You Begin

You have deployed the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Navigate to the bin directory on the Grapevine root. The bin directory contains the grapevine scripts.

Step 4

Enter the reset_grapevine command at the prompt to run the reset grapevine script.

$ reset_grapevine

The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: /etc/grapevine/controller-config.json.

The reset_grapevine command uses the data in the controller-config.json file to return to the earlier configuration settings, so do not delete this file. If you delete this file, you must run the configuration wizard again and reenter your configuration data.

You are then prompted to reenter your Grapevine password.

Step 5

Enter your Grapevine password a second time.

[sudo] password for grapevine:********

You are then prompted to delete all virtual disks The virtual disks are where the Cisco APIC-EM database resides. For example, data about devices that the controller discovered are saved on these virtual disks. If you enter yes (y), all of this data is deleted. If you enter no (n), then the new cluster will come up populated with your existing data once the reset procedure completes.

Step 6

Enter n to prevent the deletion all of the virtual disks.

THIS IS A DESTRUCTIVE OPERATION
Do you want to delete all VIRTUAL DISKS in your APIC-EM cluster? (y/n):n

You are then prompted to delete all Cisco APIC-EM authentication timeout policies, user password policies, and user accounts other than the primary administrator account.

Step 7

Enter n to prevent the deletion of all authentication timeout policies, user password policies, and user accounts other than the primary administrator account.

THIS IS A DESTRUCTIVE OPERATION
Do you want to delete authentication timeout policies, user password policies,
and Cisco APIC-EM user accounts other than the primary administrator account? (y/n): n

You are then prompted to delete any imported certificates.

Step 8

Enter n to prevent the deletion of any imported certificates.

THIS IS A DESTRUCTIVE OPERATION
Do you want to delete the imported certificates? (y/n): n

You are then prompted to delete any backups.

Step 9

Enter n to prevent the deletion of any backups.

THIS IS A DESTRUCTIVE OPERATION
Do you want to delete the backups? (y/n): n

The controller then resets itself with the configuration values that were originally set using the configuration wizard the first time. When the controller is finished resetting, you are presented with a command prompt from the controller.

Step 10

Using the Secure Shell (SSH) client, log out of the host.

Restoring the Controller to the Factory Default

In certain situations, you may want to restore the Cisco APIC-EM to its original factory default settings. For example, if your controller appliance is being replaced or simply has an undesirable configuration that needs to be completely removed. Under these circumstances, you can restore the controller to its factory defaults and then proceed to reconfigure it as a new controller.

This procedure describes how to a restore the factory defaults to the controller.

Caution

This procedure shuts down both the Cisco APIC-EM and the host (physical or virtual) on which it resides. At the end of this procedure, you will need to access the host and restart it.

Before You Begin

You have already deployed the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

You have access to the Cisco APIC-EM using either a physical console or a Telnet connection.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the reset_grapevine factory command at the prompt.

$ reset_grapevine factory

Step 4

Enter your Linux grapevine password a second time to start the reset process.

$ sudo password for grapevine *********

After entering this command a warning appears that the reset_grapevine factory command will shut down the controller.

You are then prompted to confirm your desire to run the reset_grapevine factory command.

Step 5

Enter Yes to confirm that you want to run the reset_grapevine factory command.
The controller then performs the following tasks:

- Stops all running clients and services

- Deletes all cluster data

- Deletes all user data

- Deletes the configuration files including secrets and private keys

- Shuts down the controller

- Shuts down the host (physical or virtual)

What to Do Next

Perform the following tasks:

- Start up the host (physical or virtual).

- After start up, the configuration wizard appears and prompts you to re-deploy the Cisco APIC-EM.

- Proceed to re-deploy the Cisco APIC-EM using the configuration wizard.

Creating a Support File for the Cisco APIC-EM

You can troubleshoot the Cisco APIC-EM deployment by creating a support file. This support file consists of logs, configuration files, and command output. After you create this support file, you can then email it to Cisco support for assistance.

Before You Begin

You have deployed the Cisco APIC-EM following the procedure described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Navigate to the bin directory on the host. The bin directory contains the grapevine scripts.

Step 4

To create the support file, enter the rca command in this directory.

$ rca
mkdir: created directory '/tmp grapevine-rca-2014-05-27_04-22-20-PM_PDT-0700'

----------------------------------------------------------------------
RCA package created On Tues May 27 16:22:20 PDT 2014
----------------------------------------------------------------------

[INFO] Generating log for 'vmstat 1 10'
[sudo] password for grapevine:

The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output.

What to Do Next

Send the tar file created by this procedure to Cisco support for assistance in resolving your issue.

Chapter 5. Troubleshooting Cisco APIC-EM Multi-Host

The following procedures may be used to troubleshoot a Cisco APIC-EM multi-host configuration:

- Changing the Settings in a Multi-Host Cluster

- Removing a Single Host from a Multi-Host Cluster

- Removing a Faulted Host from a Multi-Host Cluster

Changing the Settings in a Multi-Host Cluster

To troubleshoot an issue with a multi-host cluster, you may need to change its configuration settings. This procedure describes how to change the Cisco APIC-EM external network settings, NTP server address, and/or password for the Linux grapevine user in a multi-host cluster. The external network settings that can be changed include:

- Host IP address

- Virtual IP address

- DNS server

- Default gateway

- Static routes

Note

In order to change the external network settings, NTP server address, and/or the Linux grapevine user password in a multi-host deployment, you need to first break up the multi-host cluster. As a result, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period. For information about changing settings for a single host configuration, see Updating the Configuration Using the Wizard

Before You Begin

You must have successfully configured the Cisco APIC-EM as a multi-host cluster using the configuration wizard, as described in the Cisco APIC-EM deployment guide.

Step 1

Using a Secure Shell (SSH) client, log into one of the hosts in your cluster.
Log in using the IP address that you specified using the configuration wizard.

Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the appliance to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the following command to access the configuration wizard.

$ config_wizard

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 4

Review the Welcome to the APIC-EM Configuration Wizard! screen and choose the option to remove the host from the cluster:

- Remove this host from its APIC-EM cluster

Step 5

A message appears with the following options:

- [cancel]—Exit the configuration wizard.

- [proceed]—Begin the process to remove this host from its cluster.

Choose proceed>> to begin. After choosing proceed>>, the configuration wizard begins to remove this host from its cluster.
At the end of this process, this host is removed from the cluster.

Step 6

Repeat the above steps (steps 1-5) on a second host in the cluster.
Note:
You must repeat the above steps on each host in your cluster, until you only have a single host remaining. You must make your configuration changes on this final remaining host.

Step 7

Using a Secure Shell (SSH) client, log into that final host in your cluster and run the configuration wizard.

$ config_wizard

After logging into the host, begin the configuration process.

Step 8

Make any necessary changes to the configuration values for the external network settings, NTP server address, and/or password for the Linux grapevine user using the wizard.
After making your configuration change(s), continue through the configuration process to the final message.

Step 9

At the end of the configuration process, a final message appears stating that the wizard is now ready to proceed with applying the configuration.
The following options are available:

- [back]—Review and verify your configuration settings.

- [cancel]—Discard your configuration settings and exit the configuration wizard.

- [save & exit]—Save your configuration settings and exit the configuration wizard.

- [proceed]—Save your configuration settings and begin applying them.

Enter proceed>> to complete the installation. After entering proceed>>, the configuration wizard applies the configuration values that you entered above.

Note:

At the end of the configuration process, a CONFIGURATION SUCCEEDED! message appears.

Step 10

Log into the other hosts in your multi-host cluster and use the configuration wizard to recreate the cluster.
Refer to Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide for information about this specific procedure.

Removing a Single Host from a Multi-Host Cluster

To troubleshoot an issue with a multi-host cluster, you may need to remove a single host from a multi-host cluster. This procedure describes how to remove one of the hosts running Cisco APIC-EM from a multi-host cluster. You use the Cisco APIC-EM configuration wizard to perform this procedure.

Note

The configuration wizard option to remove a host only appears if the host on which you are running the configuration wizard is part of a multi-host cluster. If the host is not part of a multi-host cluster, then the option to remove a host does not display. When performing this procedure, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period.

Before You Begin

You should have deployed Cisco APIC-EM on a multi-host cluster as described in the Cisco APIC-EM deployment guide.

You must perform this procedure on the single host that is to be removed from the multi-host cluster.

Step 1

Using a Secure Shell (SSH) client, log into the host (appliance, server, or virtual machine) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the appliance to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the following command to access the configuration wizard.

$ config_wizard

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 4

Review the Welcome to the APIC-EM Configuration Wizard! screen and choose the option to remove the host from the cluster:

- Remove this host from its APIC-EM cluster

Step 5

A message appears with the following options:

- [cancel]—Exit the configuration wizard.

- [proceed]—Begin the process to remove this host from its cluster.

Choose proceed>> to begin. After choosing proceed>>, the configuration wizard begins to remove this host from its cluster.
At the end of this process, you must then either run the configuration wizard again to configure the host as a new Cisco APIC-EM or join the Cisco APIC-EM to a cluster.

What to Do Next

If you wish to use this host again as either a stand-alone controller or operating within a cluster, then you must run the configuration wizard again and re-install the Cisco APIC-EM. Do not attempt to use this host again as either a standalone host or within a cluster without re-installing the Cisco APIC-EM.

Removing a Faulted Host from a Multi-Host Cluster

Perform the steps in the following procedure to remove a faulted or inoperative host (running Cisco APIC-EM) from a multi-host cluster. You use the Cisco APIC-EM configuration wizard to perform this procedure. A host becomes faulted when it can no longer participate in the cluster due to hardware or software issues.

After following this procedure on a three host cluster (moving from three hosts to two hosts), you will lose high-availability protection against loss of a host. After following this procedure for a two host cluster, then the cluster will become inoperable until that second host is brought back up and added to the cluster.

Note

The fact that the host becomes "faulted" results in replacement instances of the services on the faulted host being grown on the remaining hosts in the cluster. During the time period when the replacement instances are being grown and depending on the types of services being grown, certain Cisco APIC-EM functionality may not be available.

Before You Begin

You have deployed Cisco APIC-EM on a multi-host cluster following the procedure described in the Cisco APIC-EM deployment guide.

You must perform this procedure on an active host in the multi-host cluster. You cannot perform this procedure on the faulted host that is to be removed from the multi-host cluster. A faulted host is displayed as red in the Hosts view in the Grapevine developer console.

Note

You should always first attempt to bring the faulted host back online. After determining that the faulted host can no longer participate in the cluster, then try to remove the faulted host using the Remove this host from its APIC-EM cluster configuration wizard option. You should only follow this procedure and the Remove a faulted host from this APIC-EM cluster configuration wizard option, if that other option is tried first and is unsuccessful in removing the host.

Step 1

Using a Secure Shell (SSH) client, log into the host (appliance, server, or virtual machine) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the appliance to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Enter the following command to access the configuration wizard.

$ config_wizard

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 4

Review the Welcome to the APIC-EM Configuration Wizard! screen and choose the option to forcibly remove the faulted host from the cluster:

- Remove a faulted host from this APIC-EM cluster

Step 5

A message appears with the following options:

- <Remove IP Address from cluster>—Forcibly removes the faulted host (identified by its IP address) from the multi-host cluster.

- <exit>—Exit the configuration wizard without removing the faulted host.

Choose <Remove IP Address from cluster> to begin. After choosing <Remove IP Address from cluster>, the configuration wizard begins to remove this faulted host from its cluster.
At the end of this process, you must then either run the configuration wizard again to configure the host as a new Cisco APIC-EM or join the Cisco APIC-EM to a cluster.

What to Do Next

If you wish to use this host again as either a stand-alone controller or operating within a cluster, then you must run the configuration wizard again and re-install the Cisco APIC-EM. Do not attempt to use this host again as either a standalone host or within a cluster without re-installing the Cisco APIC-EM.

Chapter 6. Troubleshooting Services Using System Health

The following procedures may be used to troubleshoot services using System Health:

- Reviewing the Service Version and Status Using the System Health Tab

- Removing a Service Instance Using the System Health GUI

- Creating a Service Instance Using the System Health GUI

Reviewing the Service Version and Status Using the System Health Tab

You are able to perform the following tasks using the System Health tab in the Home page:

- Review the status of each service

- Review the number of instances of each service running

- Review the version of each service

- Review the IP address of the host where the service is running

Note

In addition to using the System Health tab in the Home page to troubleshoot the controller services, you can also log into the host and access a developer console to perform the same tasks. For additional information, see .

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have administrator (ROLE_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

Log into the controller to view the controller's GUI.

Step 2

Click the System Health tab in the Home page to view information about the controller's health.
The following information is displayed in the System Health tab.

System (Host) Health Data

Data displayed include:

- Host IP address

- CPU—Host CPU usage is displayed in MHZ. Both the currently used and available host CPU is displayed.

- Memory—Host memory usage is displayed in GB. Both the currently used and available host memory is displayed.

- Storage—Host storage usage is displayed in GB. Both the currently used and available host storage is displayed.

Color indicates status for the above host data:

- Green—Indicates proper usage and support.

- Yellow—Indicates usage is approaching improper levels and triggers this warning (color change).

- Red—Indicates a failure based upon the usage exceeding the maximum supported value.

Additionally, a graphical representation of the above data over the last 24 hours is displayed in this tab. Moving your cursor or mousing over the graph displays a data summation for specific date and time.

Note:
By placing your cursor over (mouseover) a color warning in the window, further information about the warning or failure message appears.

Application Health Data

Displays applications available from the Navigation pane, and the services that support each application. For example, the Topology application accessible in the GUI is supported by topology-service.

Color bars indicate the status for the applications and the supporting service(s):

- Green —Indicates that an application instance is starting. An application instance is the aggregation of the service instances. You can configure a minimum or maximum number of service instances, as well as grow and harvest these service instances (spin up or spin down the services).

- Yellow—Indicates application instance and its supporting service instance(s) are experiencing issues and triggers this warning (color change).

- Red—Indicates a failure of the application instance and its supporting service instance(s). You can harvest a service instance and then regrow it using the GUI. If the service instance does not regrow using the GUI, then you can manually regrow it. When you harvest a service instance, the controller will determine which instance is regrown (load balancing among them).

- Blue—Indicates an in-progress state for the application or service instance (growing or harvesting).

Step 3

Review the status and version of each service and application listed in the System Health tab.

What to Do Next

If there are any problems with any of the services or applications, then review the following procedures to troubleshoot a service.

Removing a Service Instance Using the System Health GUI

You are able to remove or harvest a service instance manually by using the System Health tab in the controller's GUI. You may wish to harvest a service instance and then regrow (recreate) it to correct for a faulty or unstable service.

Note

In addition to using the System Health tab in the Home page to troubleshoot the controller services, you can also log into the host and access a developer console to perform the same tasks. For additional information, see .

Caution

Only advanced users should perform the tasks described in this procedure or attempt to troubleshoot the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have administrator (ROLE_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

Log into the controller to view the controller's GUI.

Step 2

Click the System Health tab in the Home page to view information about the controller's health.
For information about what is displayed in the System Health tab, see Reviewing the Service Version and Status Using the System Health Tab.

Step 3

Review the list of operational services in the Systems Health tab.
Each service is represented by a square. A green-tinged colored square represents an active instance of the service, and a red-tinged colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

Note:
Placing your cursor over a square displays the version of the service, number of instances running, and host IP address where the service instance is running.

Step 4

Locate the service where you want to manually remove (harvest) an instance of a service and click the subtraction sign (-) at the upper right.
You are then prompted to confirm your action to remove a service instance.

Step 5

Choose Yes in the dialog box to confirm that you want to remove an instance of the service.
The instance of the service is then spun down.

When the process is finished, the square representing the service instance is removed.

What to Do Next

Manage your services by either manually removing (harvesting) additional instances or growing (restoring) instances for the services.

Creating a Service Instance Using the System Health GUI

You are able to create or restore a service instance manually by using the System Health tab in the controller's GUI. You may wish to create or restore a service after previously harvesting or removing it for faulty or unstable behavior.

Note

In addition to using the System Health tab in the Home page to troubleshoot the controller services, you can also log into the host and access a developer console to perform the same tasks. For additional information, see .

Caution

Only advanced users should perform the tasks described in this procedure or attempt to troubleshoot the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have administrator (ROLE_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

Log into the controller to view the controller's GUI.

Step 2

Click the System Health tab in the Home page to view information about the controller's health.
For information about what is displayed in the System Health tab, see .

Step 3

Review the list of operational services in the Systems Health tab.
Each service is represented by a square. A green-tinged colored square represents an active instance of the service, and a red-tinged colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

Note:
Placing your cursor over a square displays the version of the service, number of instances running, and host IP address where the service instance is running.

Step 4

Locate the service where you want to manually create or restore an instance of a service and click the addition sign (+) at the upper right.
You are then prompted to confirm your action to create or restore an instance.

Step 5

Choose Yes in the dialog box to confirm that you want to create or restore an instance of the service.
The instance of the service is then spun up.

When the process is finished, the square representing the service instance is created.

What to Do Next

Manage your services by manually growing (restoring) additional instances or removing (harvesting) instances from the services.

Chapter 7. Troubleshooting Services Using the Developer Console

The following procedures may be used to troubleshoot services using the developer console:

- Grapevine Developer Console

- Logging into the Grapevine Developer Console

- Reviewing the Service Version, Status, and Logs

- Removing a Service Instance Manually

- Creating a Service Instance Manually

- Reviewing the User Logs

- Monitoring Services and Clients Using the CLI

Grapevine Developer Console

The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when increasing loads occur on a client within the network. You use the Grapevine developer console to troubleshoot these services. The Grapevine developer console tools were bundled with the deployment files and installed when you first deployed the Cisco APIC-EM.

Grapevine Developer Console

Note

For a multi-host cluster, you do not have to log into each host to view the Grapevine developer console. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts.

The Grapevine developer console provides the following windows and functionality:

- Overview—Provides a list of services with information about their version and status. You can add or remove services in this window.

- Clients—Provides detailed client information in this window.

- Hosts—Provides detailed host information in this window.

- Waiting Queue—Provides information about the waiting queue.

- Services—Provides detailed service information. You can add or remove services in this window.

- Logs—Provides detailed task, instance, and client logs

Note

You cannot access the Grapevine developer console as a Linux root user. You can only access the Grapevine developer console using the administrator username and password that you configured during the deployment process.

Logging into the Grapevine Developer Console

You are able to perform the following tasks using the Grapevine developer console:

- Review the status of each service

- Review the version of each service

- Review the logs of each service

Caution

Only advanced users should access the Grapevine developer console to perform any troubleshooting tasks for the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM.

Step 1

Access the Grapevine developer console by opening a browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

The console for the Elastic Service Platform (Grapevine) appears.

What to Do Next

Proceed to review any of the service versions, status, and logs for any troubleshooting purpose.

Reviewing the Service Version, Status, and Logs

You are able to perform the following tasks using the Grapevine developer console:

- Review the status of each service

- Review the version of each service

- Review the logs of each service

Caution

Only advanced users should access the console to perform the tasks described in this procedure or attempt to troubleshoot the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

Step 1

Access the Grapevine developer console by opening a browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

The console for the Elastic Service Platform (Grapevine) appears.

Step 3

Review the status of each service listed in the Overview window in the console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.

Step 4

Review the version of each service in the Overview window in the console.
The version is located in the header of each listed service.

Step 5

Review the service logs by clicking a specific active instance of a service (green square icon) and then viewing the Instance or Client logs located at the bottom of the window.

What to Do Next

When finished with the Grapevine developer console, click the Logout button to log out of the console.

Removing a Service Instance Manually

You are able to remove or harvest a service instance manually by using the Grapevine developer console. The Grapevine developer console tools are bundled within the ISO image and installed when you first deploy the Cisco APIC-EM.

Caution

Only advanced users should access the Grapevine developer console to perform the tasks described in this procedure or attempt to troubleshoot the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

Step 1

Access the Grapevine developer console by opening a Google Chrome browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

The Grapevine developer console for the Elastic Service Platform (Grapevine) appears.

Step 3

Review the list of operational services in the Overview window in the console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

Placing your cursor over a square displays the IP address of the Grapevine client where the service is running.

In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.

Note:
At the right of the console window are spare Grapevine clients that are not running any service instances.

Step 4

Locate the service where you want to manually harvest an instance of a service and click the subtraction sign (-) at the lower right.
You are then prompted to confirm your action to harvest an instance.

Step 5

Choose Yes in the dialog box to confirm that you want to harvest an instance of the service.
Grapevine then proceeds to spin down the instance of the service.

When the process is finished, the square representing the service instance is removed.

What to Do Next

Manage your services by manually growing additional instances or removing (harvesting) instances from the services. When finished with the Grapevine developer console, click the Logout button to log out of the console.

Creating a Service Instance Manually

You can create a service instance manually using the Grapevine developer console.

Caution

Only advanced users should access the Grapevine developer console to perform the tasks described in this procedure or attempt to troubleshoot the services.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

Step 1

Access the Grapevine developer console by opening a Google Chrome browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

The Grapevine developer console for the Elastic Service Platform (Grapevine) appears.

Step 3

Review the list of operational services in the Overview window in the Grapevine developer console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

Placing your cursor over a square displays the IP address of the Grapevine client where the service is running.

In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.

Note:
At the right of the console window are spare Grapevine clients that are not running any service instances.

Step 4

Locate the service where you want to manually grow an instance of a service and click the addition sign (+) at the lower right.
You are then prompted to confirm your action to grow an instance.

Step 5

Choose Yes in the dialog box to confirm that you want to grow an instance of the service.
Grapevine then takes a Grapevine client from the Spare Capacity Pool and spins up an instance of the service.

When the process is finished, the square that represents the new service instance turns green.

What to Do Next

Manage your services by manually growing additional instances or removing (harvesting) instances from the services. When finished with the Grapevine developer console, click the Logout button to log out of the console.

Reviewing the User Logs

You are able to review the user logs using the Elastic Service Platform (Grapevine) developer console.

Note

We recommend that only advanced users use the developer console to perform the steps described in this procedure.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

Step 1

Access the Grapevine developer console by opening a Google Chrome browser window and entering the IP address that you configured for the network adapter using the configuration wizard.
Note:
This IP address connects the appliance to the external network.

For example, enter the following IP address with required port number:

https://external network IP address:14141

Step 2

Enter your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard.

The Grapevine developer console for the Elastic Service Platform (Grapevine) appears.

Step 3

Review the status of each service listed in the Overview window in the console.
Each service is represented as a square. A green square represents an active instance of the service, and a red square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

Step 4

Review the version of each service in the Overview window in the console.
The version is located in the header of each listed service.

Step 5

Review the service logs by clicking a specific active instance of a service (green square icon) and then viewing the Instance logs located at the bottom of the window.

Step 6

Search through the logs for the keyword: USER-ACCOUNTING.
The following data is displayed for these log entry types:

- Date and timestamp

- API

- Action—GET, POST, PUT, or DELETE method

- Successful or unsuccessful

Note:
You can also use the grep command to search on the keyword in the logs.

What to Do Next

After reviewing the user data in the logs, proceed to troubleshoot any user activity.

When finished with the developer console, click Logout.

Monitoring Services and Clients Using the CLI

In addition to the developer console, a command-line interface on the host is also provided for troubleshooting purposes. This CLI allows you to monitor the health of the Cisco APIC-EM from the command line.

Before You Begin

You have deployed the Cisco APIC-EM using the procedures described in this guide.

Step 1

Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

When prompted, enter your Linux username ('grapevine') and password for SSH access.

Step 3

Display all of the Cisco APIC-EM services currently installed by entering the grape service display command. Key data about each service is then displayed separated by hash marks.

$ grape service display

...

#
rabbitmq                  config                    {}
rabbitmq                  core_service              True
rabbitmq                  enabled                   False
rabbitmq                  endpoint_config           {u'default':
{u'backend_protocol': u'amqp', u'backend_path': u'',
u'frontend_protocol': u'', u'frontend_path': u'', u'frontend_port':
0, u'backend_port': 5672}}
rabbitmq                  kill_as_group             True
rabbitmq                  max_instances             1
rabbitmq                  min_instances             0
rabbitmq                  priority                  1
rabbitmq                  queue_config              {u'queues': [],
u'bindings': [], u'exchanges': []}
rabbitmq                  requirements              {u'template_id':
u'default', u'persistent_disk': False}
rabbitmq                  run_as_group              grapevine
rabbitmq                  run_as_user               grapevine
rabbitmq                  service_type              rabbitmq
rabbitmq                  spare_count               0
rabbitmq                  start_secs                10
rabbitmq                  static_load               10
rabbitmq                  status_interval           60
rabbitmq                  stop_as_group             True
rabbitmq                  stop_signal               TERM
rabbitmq                  version                   1.0.0
#

...

Step 4

Display running instances of the Cisco APIC-EM services by entering the grape instance display command. Key data about running instances of service is then displayed separated by hash marks.

$ grape instance display

...

#
4c4c83db-2da6-4a04-9af6-96c8dac692d1   client_id
4c4c83db-2da6-4a04-9af6-96c8dac692d1
4c4c83db-2da6-4a04-9af6-96c8dac692d1   endpoint_config
{u'default': {u'backend_port': 5672, u'backend_protocol': u'amqp'}}
4c4c83db-2da6-4a04-9af6-96c8dac692d1   interfaces
[{'interface': 'eth0', 'ip': '192.168.0.1', 'mac': '00:50:56:9f:6c:c4'},
{'interface': 'eth1', 'ip': '172.16.0.15', 'mac': '00:50:56:9f:71:46'}]
4c4c83db-2da6-4a04-9af6-96c8dac692d1   is_error           None
4c4c83db-2da6-4a04-9af6-96c8dac692d1   service_type       rabbitmq
4c4c83db-2da6-4a04-9af6-96c8dac692d1   state              running
4c4c83db-2da6-4a04-9af6-96c8dac692d1   task_id            None
4c4c83db-2da6-4a04-9af6-96c8dac692d1   timestamp
Fri Oct 03, 2014 02:05:43 PM (4 days ago)
4c4c83db-2da6-4a04-9af6-96c8dac692d1   version            1.0.0
#

...

Note:
All services should have a state property value of running.

Step 5

Display all Grapevine clients currently running in Cisco APIC-EM by entering the grape client display command. Key data about each client is then displayed separated by hash marks.

$ grape client display

CLIENT                                 PROPERTY             VALUE
----------------------------------------------------------------------
ae63a6c1-a946-4df7-a68d-33227eed8134   client_id        ae63a6c1-a946-4df7-a68d-33227eed8134
ae63a6c1-a946-4df7-a68d-33227eed8134   client_version   0.1.0.212.dev633-gf7e21de
ae63a6c1-a946-4df7-a68d-33227eed8134   interfaces       [{'interface': 'eth0', 'ip': '192.168.0.32',
'mac': '00:50:56:9f:3a:90'}]
ae63a6c1-a946-4df7-a68d-33227eed8134   is_alive         True
ae63a6c1-a946-4df7-a68d-33227eed8134   last_heartbeat   Wed Oct 08, 2014 10:22:50 AM (15 secs ago)
ae63a6c1-a946-4df7-a68d-33227eed8134   template_id      default
ae63a6c1-a946-4df7-a68d-33227eed8134   vm_id            ce0a634a-5475-4450-9dce-f3217d855ac4
#

...

All clients should have a is alive property of True.

Chapter 8. Troubleshooting Using the Logs

The following logs may be used to troubleshoot Cisco APIC-EM:

- Audit Logs—Logs used primarily to monitor Cisco APIC-EM policy creation and application.

- Service Logs—Logs used to monitor Cisco APIC-EM services.

- Viewing Audit Logs

- Changing the Logging Level

- Downloading the Service Logs

Viewing Audit Logs

Cisco APIC-EM audit logs are used primarily to keep track of policies for the EasyQoS and iWAN applications.

You can view audit logs using the Audit Logs window in the Cisco APIC-EM GUI. The Cisco APIC-EM also supports the ability to export the audit logs to a local system.

Audit Logs

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have either administrator (ROLE_ADMIN) or policy administrator (ROLE_POLICY_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, "Managing Users and Roles," in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.

Step 2

Click the Audit Logs link from the drop-down menu.
The Audit Logs window appears. In the Audit Logs window, you view current policies that exist in your network. These are policies applied by either the IWAN or EasyQoS applications.

The following information is displayed for each policy in the window:

- Description—Application or policy audit log description

- Site—Name of site for the specific audit log

- Device—Device or devices for the audit log

- Requestor—User requesting audit log

- Created On—Date application or policy audit log was created.

Step 3

Click on the addition icon (+) next to an audit log to view the children audit logs in the Audit Logs window.
Each audit log is a parent to several child audit logs. By clicking on this icon, you can view a series of additional audit logs.

Step 4

Perform a search of the audit logs by clicking on the Search field in the Audit Logs window, entering a specific parameter, and then clicking the Submit button.
You can search for a specific audit log by the following parameters:

- Description

- Requestor

- Device

- Site

- Start Date

- End Date

Step 5

Click on the dual arrow icon to refresh the data displayed in the window.
The data displayed in the window is refreshed with the latest audit log data.

Step 6

Click on the down arrow icon to download a local copy of the audit log in .csv file format.
A .csv file containing audit log data is downloaded locally to your system. You can use the .csv file for additional review of the audit log or archive it as a record of activity on the controller.

What to Do Next

Proceed to review any additional log files using the controller's GUI, or download individual audit logs as .csv files for further review or for archiving purposes.

Changing the Logging Level

To assist in troubleshooting any service issues, you can change the logging level for Cisco APIC-EM services by using the Changing the Logging Level window in the Cisco APIC-EM GUI.

Service Logging Level Window

A logging level determines the amount of data that is captured to the controller's log files. Each logging level is cumulative, that is, each level contains all the data generated by the specified level and any higher levels. For example, setting the logging level to Info also captures Warn and Error logs.

The default logging level for services in the controller is informational (Info). You can change the logging level from informational (Info) to a different logging level (Debug or Trace) to capture more information.

Caution

Any logs collected at the Debug level or higher should be handled with restricted access.

Note

The log files are created and stored in a centralized location on your controller. From this location, the controller can query and display them in the GUI The total compressed size of the log files is 2GB. If log files created are in excess of 2GB, then the pre-existing log files are overwritten with the newer log files.

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have administrator (ROLE_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.

Step 2

Click the Settings link from the drop-down menu.

Step 3

In the Settings navigation pane, click Changing the Logging Level to view the Changing Logging Level window.
The Logging Level table appears with the following fields:

- Services

- Logging Level

- Timeout

Step 4

In the Changing Logging Level window, choose a service from the Services field to adjust its logging level.
Note:
The Services field displays any services that are currently configured and running on the controller.

Step 5

In the Changing Logging Level window, choose the new logging level for the service from the Logging Level field.
The following logging levels are supported on the controller:

- Trace—Trace messages

- Debug—Debugging messages

- Info—Normal but significant condition messages

- Warn—Warning condition messages

- Error—Error condition messages

Step 6

In the Changing Logging Level window, choose the time period for the logging level from the Timeout field for the logging level adjustment.
You configure logging level time periods in increments of 15 minutes up to an unlimited time period.

Step 7

Review your selection and click the Apply button.
To cancel your selection click the Cancel button.

The logging level for the specified service is set.

Downloading the Service Logs

For troubleshooting purposes, you can search and download service logs using the Search Logs window in the Cisco APIC-EM GUI. You can first narrow your search down to certain specific parameters with the GUI, and then download only these logs locally for further review or to send to support for assistance.

Downloading Logs

The following types of log files are reviewed during a search and download:

- Linux logs

- Grapevine logs

- Grapevine service logs

- Database logs

Before You Begin

You must have successfully deployed the Cisco APIC-EM and it must be operational.

You must have either administrator (ROLE_ADMIN) or policy administrator (ROLE_POLICY_ADMIN) permissions to perform this procedure.

For information about the user permissions required to perform tasks using the Cisco APIC-EM, see the chapter, "Managing Users and Roles," in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Step 1

In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.

Step 2

Click the Logs link from the drop-down menu.
The Search Logs window appears. In the Search Logs window, you can download the controller service logs by performing the following tasks:

- Download service logs by entering a string value.

- Download service logs by configuring the GUI drop-down menus and fields.

- Download service logs by both entering a string value and by configuring the GUI drop-down menus and fields as filters to that string value.

Step 3

(Optional) Enter a string value in the Search Logs field at the top of the Search Logs window and click the Download button.
The log download results are displayed at the bottom of the Search Logs window.

Step 4

(Optional) In the Search Log window, choose a service from the Services drop-down menu for the download and click the plus sign (+).
You can add several different services to your download, by choosing from the drop-down menu and then clicking the plus sign(+).

Note:
The Services drop-down menu displays any services that are currently configured and running on the controller.

Step 5

(Optional) In the Search Log window, choose a Java class from the Class drop-down menu for the download and click the plus sign (+).
You can add several different Java classes to your download, by choosing from the drop-down menu and then clicking the plus sign(+).

Step 6

(Optional) In the Search Logs window, choose a logging level from the Log Level drop-down menu.
The following logging levels are supported:

- Trace—Trace messages

- Debug—Debugging messages

- Info—Normal but significant condition messages

- Warn—Warning condition messages

- Error—Error condition messages

Step 7

(Optional) Adjust the logging level by choosing an appropriate condition in the second Log Level drop-down menu.
The following logging level adjustments are supported:

- And Below—Search for the specified logging level and any other logging level that has a lower level. For example, a Trace has a lower logging level than a Warn.

- Only—Search only for the specified logging level. Ignore any other logging levels in the results.

- And Above—Search for the specified logging level and any other logging level with a higher level. For example, a Warn has a higher logging level than a Debug.

Step 8

(Optional) In the Search Logs window, enter a start time for the logs in the Start Time field for the download or use the calendar icon.
If entering a date and time directly, use the following formats:

- Hour: Minutes, AM or PM

- MM/DD/YYYY

Step 9

(Optional) In the Search Logs window, enter an end time for the logs in the End Time field for the download or use the calendar icon.
If entering a date and time directly, use the following formats:

- Hour: Minutes, AM or PM

- MM/DD/YYYY

Step 10

Review your log search settings and then click the Download button. The log download results are displayed at the bottom right of the Search Log window as a page icon displaying the number of logs using the following format: Search Results (5).log.

Step 11

Click on the icon for the log download results.
A new window opens that displays the log download data. This data is organized using the following parameters:

- Timestamp—Time message was sent to the log file

- Service type—Service

- Class—Java class that executed the request.

- Log level—Log level

- Message—Actual detail of message that was sent to the log file. For example, "File not found" or "Resource xxx not found".

- Version Number—Version of the service.

What to Do Next

Proceed with any additional service log searches and downloads.

Chapter 9. Troubleshooting Passwords

The following procedures may be used to troubleshoot password problems:

- Performing Password Recovery with an Existing Administrator

- Performing Password Recovery with No Existing Administrator

- Performing Password Recovery for the Linux Grapevine User Account

Performing Password Recovery with an Existing Administrator

To perform password recovery for a user (administrator, installer or observer) where there exists at least one controller administrator (ROLE_ADMIN) user account, take the following steps:

1. Contact the existing administrator to set up a temporary password for the user that requires password recovery.

Note

The administrator can set up a temporary password by deleting the user's account and then recreating it with the lost password. The user can then log back into the controller to regain access and change the password once again to whatever he or she desires.

2. The user then needs to log into the controller with the temporary password and change the password.

Note

Passwords are changed in the controller GUI using the Change Password window. For information about changing passwords, see Chapter 4, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.

Performing Password Recovery with No Existing Administrator

The following procedure describes how to perform password recovery where there exists only one controller administrator (ROLE_ADMIN) user account and this account cannot be successfully logged into.

Note

We recommend that you create at least two administrator accounts for your deployment. With two administrator accounts, if one account is locked for whatever reason then the other account can be used to unlock that locked account.

Step 1

If there are no other existing administrator (ROLE_ADMIN) user accounts, use an SSH client from your terminal to log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
Note:
The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

Step 2

Enter the Linux username ('grapevine') and password when prompted.

Step 3

On the console, enter the following command on the Grapevine root.

$ config_wizard

This command starts up the Cisco APIC-EM configuration process.

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 4

Choose the <Create a new APIC-EM cluster> option.

Step 5

Proceed through the configuration process until reaching the step to configure the APIC-EM ADMIN USER SETTINGS.

Step 6

Specify a new administrator user password.

Step 7

Reenter the new administrator user password for confirmation.

Step 8

Proceed through the configuration wizard and its process until completion.
Caution:
To save the data in the Cisco APIC-EM database as part of the reset, ensure that no is chosen when prompted in HARVEST ALL VIRTUAL DISKS.

This final step will bring down the cluster and then bring it back up again (similar to running the reset_grapvine command).

Performing Password Recovery for the Linux Grapevine User Account

You can use the following procedure to recover from the loss of the Linux grapevine user password. This procedure reconfigures the Linux grapevine user password that is required for accessing the host's Linux operating system.

Before You Begin

You should be logged into the host (physical or virtual) using a Linux console to access the Linux kernel.

Step 1

Reboot the host (physical or virtual) while logged into the Linux console.

Step 2

Press "e" upon seeing the GNU GRUB menu to edit the boot commands.
Note:
In a VMware environment, you may need to press a different key to view the GNU GRUB menu. Refer to your VMware documentation for information about access to the GNU GRUB menu. Additionally, there may be different keys to press to enter the boot sequence depending upon the BIOS used for the host.

Step 3

Search for the line in the GNU GRUB menu output that begins with "linux" and change “ro” to “rw”, and append "init=/bin/bash" to that line.
For example, search for this line:

linux /vmlinuz-3.13.0-24-generic root=/dev/mapper/grapevine--vg-root ro cgroup_enable=memory swapaccount=1 quiet sqlash $vt_handoff

And change it to this line:

linux /vmlinuz-3.13.0-24-generic root=/dev/mapper/grapevine--vg-root rw cgroup_enable=memory swapaccount=1 quiet sqlash $vt_handoff init=/bin/bash

Step 4

Press Ctrl-x or the F10 key to proceed with the boot process.
Note:
We recommend that you use the F10 key to proceed with the boot process.

At this point, the host will boot up in root mode. You can now enter the Linux passwd command to reset the password for the Linux grapevine user.

Step 5

Enter the Linux passwd command to reset the password for the Linux grapevine user.

$  passwd grapevine

Caution:
This procedure permits you to change the Linux grapevine user password. Do not change the Linux root user password at any point in this procedure. Resetting the Linux root user password reduces the security of the host.

Step 6

When prompted, enter a new Linux grapevine password.

Step 7

When prompted, confirm the new Linux grapevine password by entering it a second time.

Step 8

Enter the following reboot command to reboot the system.

$ /sbin/reboot -f

The system reboots and will start up with new configuration and password.

At the end of the reboot process, you are presented with the GNU GRUB menu.

Step 9

Press Enter to boot up in the Ubuntu OS.

Step 10

After booting up in the Ubuntu OS, log back into the host by entering your Linux grapevine username and password.
Note:
Enter the Linux grapevine password created in step 6 above.

Step 11

Restart the configuration wizard using the following command.

$ config_wizard

Proceed through the configuration wizard process by clicking next>> and accepting the pre-configured values until you reach the LINUX USER SETTINGS step.

Note:
The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

Step 12

When prompted to enter values for the LINUX USER SETTINGS, enter the new Linux grapevine password that you created earlier in step 6.
Note:
You need to start up the configuration wizard and run through the configuration process to synchronize the Linux grapevine user password to the controller itself.

Step 13

Click next>> and continue through the configuration wizard process, until the last step of this process.
Note:
When prompted to enter values for the CONTROLLER CLEAN-UP step, be sure to enter no for both Harvest All Virtual Disks and Delete All Users.

Step 14

At the end of the configuration wizard process, click proceed>> to have the configuration wizard save and apply your configuration changes to the Cisco APIC-EM.

Chapter 10. Troubleshooting Commands

The following commands are available to troubleshoot the Cisco APIC-EM:

- Root Commands

- Client Commands

Root Commands

The following table describes commands that you can issue on the Grapevine root to troubleshoot the Cisco APIC-EM.

Note

Enter the grape help command on Grapevine root to view a list of available commands. For more details on a specific command, type grape help command_from_list. For example, by entering grape help update , a new list of commands including history and upload appears. By further entering grape help update upload, specific command information and usage are then displayed.

The following table displays the read-only Grapevine root commands. Use these commands to display controller status, network configurations, service versions, and other key Cisco APIC-EM data to assist in troubleshooting procedures.

Root Commands—Read Only

Command

Description

grape client display

Displays the number of Grapevine clients that are currently running. This command also displays any clients that have faulted.

grape client status

Displays the status of the Grapevine clients.

grape ctlstatus

Displays the open ports on the Grapevine root and clients. An open port is any TCP or UDP port, for which a listener exists on one or more IP addresses.

grape host status host_id

Displays all of the services on the host where it is issued.

grape instance display

Displays the Cisco APIC-EM services that are currently running.

grape instance status

Displays the status of the Cisco APIC-EM services.

grape network display

Displays the current external network configuration used by both Grapevine and the Cisco APIC-EM including:

- IP addresses

- Netmask

- DNS servers

grape release display current

Displays the versions of the Cisco APIC-EM services that are currently running.

grape release display latest

Displays the latest available versions of the Cisco APIC-EM services.

grape root display

Displays the Grapevine root properties, including:

- hostname

- interfaces

- root ID

- version

grape service display

Displays the Cisco APIC-EM services that are currently installed.

grape update_service display

Displays the current automatic services update configuration.

Additional fields in the command output indicate the last time the Cisco cloud was polled for updates (last_connect_time), as well as the Cisco cloud status(last_connect_status). For example, whether the Cisco cloud is reachable, unreachable, etc.

grape version

Displays the version of Grapevine that the Grapevine root is running.

sudo service grapevine status

Determines the status of the Grapevine root core services and whether all of the root core services are running.

The following table displays the read and write Grapevine root commands. Use these commands to perform delete, harvest, update, and capture actions on the controller.

Caution

You should exercise caution when issuing any of the read/write commands in this table.

Root Commands-Read and Write

Command

Description

grape backup delete

Deletes the existing controller backup file.

Note:
If you are unable to back up your current controller database, then you may want to delete it and restore an earlier smaller controller backup file.

grape host evacuate host_id

Harvests all of the services on the host where it is issued. If you issue this command on a host in a multi-host cluster, then the services are harvested and transferred to the remaining two hosts in the cluster. If you issue this command on a host in a single host configuration, all the services are harvested.

grape release update force

Updates a service for Cisco APIC-EM.

Use this command to force a service update.

grape tls

Use this command to set configure the TLS version to use for security for the controller.

rca

The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output. After running this command and creating the tar file, you can send the file to Cisco support for assistance in resolving an issue.

For examples of using this command:

- Confirming that Core Services are Running

- Confirming the Multi-Host Cluster Configuration Values

- Creating a Support File for the Cisco APIC-EM

reset_grapevine

The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: $ /etc/grapevine/controller-config.json.

For examples of using this command:

- Resetting the Cisco APIC-EM

reset_grapevine factory

The reset_grapevine factory command returns the configuration settings back to their factory defaults.

Caution:
This command shuts down both Cisco APIC-EM and the host (physical or virtual) where the controller resides. After running this command, you will need to access the host and reboot it.

For examples of using this command:

- Restoring the Controller to the Factory Default

Client Commands

The following table describes commands that you can issue on the Grapevine client to troubleshoot the Cisco APIC-EM.

Client Commands- Read Only

Command

Description

grape version

Displays the version of Grapevine that the Grapevine client is running.

grape ctlstatus

Determines the status of the Grapevine client core services and whether all of the client core services are running.

Chapter 11. Troubleshooting Log Files

The following log files are available to assist in troubleshooting any issues with the Cisco APIC-EM:

- Root Log Files

- Client Log Files

Root Log Files

The following table lists log files located on the Grapevine root that can be used to troubleshoot the Cisco APIC-EM.

Root Log Files

Directory

Filename

Description

/var/log/

boot.log

This log file provides you with the details of the boot process and any errors that may occur during this process. Use this log file to troubleshoot any of the following problems:

- Why did the Grapevine client fail to communicate with the Grapevine root?

- Why did the Grapevine client fail to upgrade?

/var/log/

config_wizard.log

Use this log file to determine if there were any errors during the initial Grapevine configuration and deployment.

/var/log/

grapevine_manager.log

Use this log file to determine if there were any errors during a manual file update using the Software Update fields in the controller's UI.

/var/log/

grapevine_manager_activity.log

Use this log file to determine if there were any errors during an update using the Software Update fields in the controller's UI.

/var/log/grapevine

supervisord.log

Use this log file to determine whether any of the Grapevine root core services unexpectedly expired.

/var/log/

cassandra.log

Use this log file to determine whether the Grapevine database is healthy.

/var/log/grapevine

grapevine_root.log

Use this log file to troubleshoot any of the following problems:

- Why did Grapevine fail to grow or harvest a service instance?

- Why did an automatic service update fail?

- Why did Grapevine fail to communicate with a Grapevine client?

/var/log/grapevine

grapevine_capacity_manager.log

Use this log file to determine why Grapevine failed to grow or harvest a Grapevine client.

Client Log Files

The following table lists log files located on the Grapevine client that can be used to troubleshoot the Cisco APIC-EM.

Client Log Files

Directory

Filename

Description

/var/log

boot.log

Use this log file to determine the following:

- Why did this Grapevine client fail to communicate with the root?

- Why did this Grapevine client fail to upgrade?

/var/log/grapevine

supervisord.log

Use this log file to determine if any of the Grapevine client core services unexpectedly died?

/var/log/grapevine

grapevine_client.log

This log file provides you with the details of the Grapevine client daemon bootstrap process and any errors that may occur during this process.

Use this log file to determine the following:

- Why did this Grapevine client fail to grow or harvest a service instance?

- Why did this Grapevine client fail to communicate to the Grapevine root?

/var/log/grapevine/services

service-name/version/service-name.log

Use this log file to determine why did a service fail to perform some operation?

Chapter 12. Contacting the Cisco Technical Assistance Center

- Contacting the Cisco Technical Assistance Center

Contacting the Cisco Technical Assistance Center

If you cannot locate the source and potential resolution for a problem in the prior sections of this guide, then contact a Cisco customer service representative for information on how to best proceed with resolving the issue. For Cisco Technical Assistance Center (TAC), see the Cisco Information Packet publication that is shipped with your appliance or visit the following website:

Click Here

Before you contact Cisco TAC, make sure that you have the following information ready:

- If your deployment is using a Cisco APIC-EM appliance(s), then the appliance chassis type and serial number.

- If your deployment is using a Cisco APIC-EM appliance(s), then the date you received the new Cisco APIC-EM appliance.

- The maintenance agreement or warranty information (see the Cisco Information Packet).

- The name, type of software, and version or release number (if applicable).

- A brief description of the problem or condition you experienced, the steps you have taken to isolate or re-create the problem, and a description of any steps you took to resolve the problem.

Note

Be sure to provide the customer service representative with any upgrade or maintenance information that was performed on the controller after your initial installation.

Preface

- Audience

- Document Conventions

- Related Documentation

- Obtaining Documentation and Submitting a Service Request

Audience

This publication is for experienced network administrators who will deploy the Cisco Application Policy Infrastructure Controller Enterprise Module (Cisco APIC-EM) in their network. Use this guide to troubleshoot any issues with the Cisco APIC-EM.

For additional information about deployment, security, and access for the Cisco APIC-EM, see the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide . For information about using the controller's GUI for the first time, see the Cisco APIC-EM Quick Start Guide.

Note

The Cisco Application Policy Infrastructure Controller Enterprise Module (Cisco APIC-EM) is also referred to within this upgrade guide as a controller.

Document Conventions

This document uses the following conventions:

Convention

Description

^ or Ctrl

Both the ^ symbol and Ctrl represent the Control (Ctrl) key on a keyboard. For example, the key combination ^D or Ctrl-D means that you hold down the Control key while you press the D key. (Keys are indicated in capital letters but are not case sensitive.)

bold font

Commands and keywords and user-entered text appear in bold font.

Italic font

Document titles, new or emphasized terms, and arguments for which you supply values are in italic font.

Courier font

Terminal sessions and information the system displays appear in courier font.

Bold Courier font

Bold Courier font indicates text that the user must enter.

[x]

Elements in square brackets are optional.

...

An ellipsis (three consecutive nonbolded periods without spaces) after a syntax element indicates that the element can be repeated.

|

A vertical line, called a pipe, indicates a choice within a set of keywords or arguments.

[x | y]

Optional alternative keywords are grouped in brackets and separated by vertical bars.

{x | y}

Required alternative keywords are grouped in braces and separated by vertical bars.

[x {y | z}]

Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

< >

Nonprinting characters such as passwords are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

Reader Alert Conventions

This document may use the following conventions for reader alerts:

Note

Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Tip

Means the following information will help you solve a problem.

Caution

Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Timesaver

Means the described action saves time. You can save time by performing the action described in the paragraph.

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device. Statement 1071

SAVE THESE INSTRUCTIONS

Related Documentation

- Cisco APIC-EM Documentation:

- Release Notes for the Cisco Application Policy Infrastructure Controller Enterprise Module

- Supported Platforms for the Cisco Application Policy Infrastructure Controller Enterprise Module

- Cisco APIC-EM Quick Start Guide (directly accessible from the controller's GUI)

- Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide

- Cisco Application Policy Infrastructure Controller Enterprise Module Upgrade Guide

- Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide

- Cisco Application Policy Infrastructure Controller Enterprise Module Hardware Installation Guide

- Cisco Application Policy Infrastructure Controller Enterprise Module Troubleshooting Guide

- Open Source Used In Cisco APIC-EM

- Cisco IWAN Documentation for the Cisco APIC-EM:

- Release Notes for Cisco IWAN

- Release Notes for Cisco Intelligent Wide Area Network (Cisco IWAN)

- Software Configuration Guide for Cisco IWAN on APIC-EM

- Open Source Used in Cisco IWAN and Cisco Network Plug and Play

- Cisco Network Plug and Play Documentation for the Cisco APIC-EM:

- Release Notes for Cisco Network Plug and Play

- Solution Guide for Cisco Network Plug and Play

- Configuration Guide for Cisco Network Plug and Play on Cisco APIC-EM

- Cisco Open Plug-n-Play Agent Configuration Guide

- Mobile Application User Guide for Cisco Network Plug and Play

Note

For information about developing your own application that interacts with the controller by means of the northbound REST API, see the developer.cisco.com/site/apic-em Web site.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation.

To receive new and revised Cisco technical content directly to your desktop, you can subscribe to the What's New in Cisco Product Documentation RSS feed. RSS feeds are a free service.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)