New and Changed Information

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

Cisco ACI Virtual Edge Version

Feature

Description

Cisco ACI Virtual Edge 1.2(1a)

Health monitoring

Beginning with this release, Cisco ACI Virtual Edge faults are supported.

Cisco ACI Virtual Edge Health Monitoring

The Cisco ACI reports errors that occur on nodes in the fabric to the Cisco APIC as an aid to troubleshooting. Beginning with Cisco ACI Virtual Edge Release 1.2(1), Cisco ACI Virtual Edge faults are reported in addition to faults for leaf and spine switches in the ACI fabric.

Viewing the health status for the Cisco ACI Virtual Edge can alert you to problems and let you know where they occur. For example, viewing health status can tell you when a port does not attach or when a policy fails to download.

The Cisco ACI Virtual Edge monitors states of objects—an EPG, port, global policy, or Virtual Tunnel Endpoint (VTEP)—listed in a database. When an object undergoes a state change, that change is recorded. The database is polled every 10 seconds, and when the Cisco ACI Virtual Edge detects an abnormal state, it reports the fault to the Cisco APIC. When the object returns to a normal state, the Cisco ACI Virtual Edge clears the fault.

No action is required to configure collection of data into a health score.

The following sections list the faults monitored for Cisco ACI Virtual Edge and instructions for viewing faults.

For detailed information about health scores and health monitoring, see the following documents on Cisco.com:

  • Cisco Application Centric Infrastructure Fundamentals Guide

  • Cisco APIC Troubleshooting Guide

Faults Monitored for Cisco ACI Virtual Edge

The Cisco ACI Virtual Edge monitors two types of faults: host faults and port faults. For example: A host fault is raised if there are not enough VTEPs on the host; a port fault is raised if OpFlex fails to download a base EPG for a port after one minute.

The following table lists the name, type, and description for each fault monitored for the Cisco ACI Virtual Edge:

Fault Name

Fault Type

Fault Description

Switch OpFlex Channel is down

Host

In VPC mode, one OpFlex channel is down.

Switch VMM Domain Config isn’t downloaded

Host

The Cisco ACI Virtual Edge hasn’t received the switching mode, encap type, or NS GIPO from the fabric.

AVE Process has crashed

Host

The Cisco ACI Virtual Edge process has crashed.

VTEP IP isn’t assigned

Port

A VTEP hasn’t received an IP address from the fabric.

AVE process isn't running Host Cisco ACI Virtual Edge auto process restart has failed.

Port Attach isn’t acked

Port

A VM port hasn’t received an attach acknowledgment from the fabric.

Port Detach isn’t acked

Port

A VM port hasn’t received a detach acknowledgment from the fabric.

Port EPG isn’t downloaded

Port

A VM port hasn’t received an EPG from the fabric.

See the section Viewing Faults for Cisco ACI Virtual Edge in this guide.

Viewing Faults for Cisco ACI Virtual Edge

You can view the status of Cisco ACI Virtual Edge faults in Cisco APIC in two ways. You can view all the faults of a specific domain or type through the System tab. Alternatively, you can navigate to and select an object and then view the different faults for just that object.

For general information about monitoring network health—including viewing faults—in Cisco APIC, see the Cisco APIC Troubleshooting Guide. For detailed information about faults, see the Cisco APIC Faults, Events, and System Messages Management Guide.

Viewing Cisco ACI Virtual Edge Faults by Type

If you think you know what kind of fault has occurred, you can navigate to a controller or EPG in Cisco APIC. You can then select it to view fault information. Port faults can appear under a controller or an EPG, depending on the individual fault. Host faults appear under a controller.

Procedure

Step 1

Log in to the Cisco APIC.
Step 2

Take one of the following actions:

Option Description
To look for... Then...
A host fault under a controller

  1. Choose Virtual Networking > Inventory > VMM Domains > VMware > VMM domain.

  2. In the work pane for the domain, scroll down to the vCenter area, and double-click the controller with the host where you think the problem occurred.

    The VMM Controller dialog box displays the properties for the controller, including the fault icons near the top.

A host or port fault under an EPG

  1. The work pane lists the faults for the EPG with the fault icons at the top of the list.

    When you choose the EPG, the work pane displays properties for the EPG, including fault icons near the top of the pane.

  2. In the work pane for the EPG, click the Faults tab.

    The work pane lists the faults for the EPG with the fault icons at the top of the list.

Note 
If you click the Health tab instead of the Faults tab, the central pane displays the EPG health score and fault icons.
Step 3

Hover your cursor over the dark orange major fault icon to see the number of faults for the controller or EPG.

Step 4

Click the Faults tab.

The Faults window appears, listing information about the major faults for the controller or EPG: Severity, Acked, Cause, Creation Time, Affected Object, Description, Code, Last Transition, and Lifecycle.
Step 5

For an explanation of the fault and recommended action, double-click the row with the fault.

The Fault Properties window for the fault opens. You can copy and paste information from the Fault Properties window, but you cannot do so from the Faults pane.
Step 6

In the Fault Properties window, click History to see a list of all the faults that have occurred for the controller or the EPG and have not yet been cleared.

What to do next

Once you have pinpointed the fault, follow procedures in this guide to resolve the problem. If the problem persists, contact Cisco Customer Support.

Viewing All Cisco ACI Virtual Edge Faults by Domain or Type

In the Cisco APIC, you can click different areas in the GUI to display all the faults of a specific domain or type for the Cisco ACI Virtual Edge. You can then click to display detailed information about specific faults.

Procedure

Step 1

Log in to the Cisco APIC.

The System tab Dashboard opens by default.

Step 2

To view all the Cisco ACI Virtual Edge faults, double-click the dark orange major fault icon in the Fault Counts By Domain area or the Fault Counts By Type area.

All Cisco ACI Virtual Edge faults are classified as major in Cisco APIC. However, not all major faults are related to Cisco ACI Virtual Edge.

The list of major faults by domain or type opens in the Faults pane. Cisco APIC displays the faults by Severity, Domain, Type, Code, Count, Cause, and Sample Fault Description. All Cisco ACI Virtual Edge faults are in the External domain.
Step 3

Read the Sample Fault Description to help determine whether the fault is related to Cisco ACI Virtual Edge.

For example, the fault may refer to a problem with OpFlex. You can hover your cursor over the Sample Fault Description for more information.

Step 4

To see detailed information about a particular fault, double-click the row with the fault.

The Faults pane displays additional information about the fault: Severity, Acked, Cause, Creation Time, Affected Object, Description, Code, Last Transition, and Lifecycle. Descriptions of Cisco ACI Virtual Edge faults begin with the string "Virtual Switch fault."
Step 5

For an explanation of the fault and recommended action, double-click the row with the fault.

The Fault Properties window for the fault opens. You can copy and paste information from the Fault Properties window, but you cannot do so from the Faults pane.

What to do next

Once you have pinpointed the fault, follow procedures in this guide to resolve the problem. If the problem persists, contact Cisco Customer Support.

Troubleshooting Cisco ACI Virtual Edge Faults

Use the procedures in this section to troubleshoot faults that occur on the Cisco ACI Virtual Edge. Faults are listed according to their names in the Cisco APIC GUI. Each section describes the fault and provides steps that you can take to address the fault.

In all cases, we recommend that you collect Cisco ACI Virtual Edge (vLeaf) and TOR (leaf) log files before and after debugging the fault.

Switch OpFlex Channel is down

In virtual port channel (VPC) mode, one OpFlex channel is down.

Procedure

Step 1

Confirm that at least one OpFlex channel is not active.

Example:

cisco-ave:~$ vemcmd show opflex
Status: 12 (Active)
Channel0: 12 (Active), Channel1: 5 (Disconnected)
Step 2

Check that all dpdk ports are up.

Example:

cisco-ave:/# vemcmd show port
  LTL     Port  Admin Link  State     Cause  PC-LTL  SGID  ORG  svcpath  Type               Owner  Vem Port
   19   Eth2/1     UP   UP    FWD         -       0          0        0                int-uplink  dpdk
   20   Eth2/2     UP   UP    FWD         -    1041     0    0        0                ext-uplink  dpdk
   51              UP   UP    FWD         -       0     0    0        0                kni-opflex  dpdk
   52              UP   UP    FWD         -       0     0    0        0              kni-ave-ctrl  dpdk
 1041      Po1     UP   UP    FWD         -       0          0        0
Step 3

Check that both leaves are operational.

Step 4

Check the port channel and VPC configuration in Cisco APIC.

Step 5

Restart the DPA.

Example:

cisco-ave:~$ vem restart
Step 6

Check that at least one OpFlex channel is active.

Example:

cisco-ave:~$ # vemcmd show opflex
Status: 12 (Active)
Channel0: 12 (Active), Channel1: 12 (Active)

Switch VMM Domain Config isn't downloaded

The Cisco ACI Virtual Edge hasn't received the switching mode, encapsulation type, or NS GIPO from the fabric.

Procedure

Step 1

Confirm by checking that one or more VMM Domain configurations are unknown or have a value of 0.0.0.0.

Example:

cisco-ave:~$ vemcmd show opflex
Switching Mode: unknown
Encap Type: unknown
NS GIPO: 0.0.0.0
Step 2

Check the VMM domain configuration in Cisco APIC.
Step 3

Restart the DPA.

Example:

# vem restart

Host Process has crashed

The DPA process has crashed.

Procedure

Step 1

Confirm by checking for a DPA core file.

Example:

# ls -l /var/crash/
	total 0
	-rwx------    1 root     root       6000000 Jun 19 16:35 core.vem
Step 2

Collect the Cisco ACI Virtual Edge and leaf switch log files and send them to Cisco Customer Support.

VTEP IP isn't assigned

A VTEP hasn't received an IP address from the fabric.

Procedure

Step 1

Confirm that the VTEP's vmknic has a valid IP address.

Example:

cisco-ave:~$ ifconfig kni0
kni0: flags=4163 UP,BROADCAST,RUNNING,MULTICAST mtu 1500
inet 10.0.40.66 netmask 255.255.0.0 broadcast 10.0.255.255
inet6 fe80::250:56ff:fe9a:7589 prefixlen 64 scopeid 0x20ink
ether 00:50:56:9a:75:89 txqueuelen 1000
Step 2

Check that both leaf switches are operational.

Step 3

Check the port channel and VPC configuration in Cisco APIC.

Step 4

Reboot the Cisco ACI Virtual Edge VM.

Port Attach isn't acked

A VM port hasn't received an attach acknowledgment from the fabric. The port may not forward traffic or may forward traffic with a stale EPG.

Procedure

Step 1

Confirm that the port state is WAIT_ATT_ACK.

Example:

# vemcmd dpa dump attach
	=>dpa command is: dump attach
	LTL    Name               EPG-Alias               State         Tries  EPCP
	---------------------------------------------------------------------------
	51     ubuntu-vm-1        dvportgroup-7980        WAIT_ATT_ACK  0      N
Step 2

Check that at least one OpFlex channel is active.

Example:

# vemcmd show opflex
	Status: 12 (Active)
	Channel0: 12 (Active), Channel1: 12 (Active)
Step 3

Detach and reattach the port.

Step 4

Restart the DPA.

Example:

# vem restart

Port Detach isn't acked

A VM port hasn't received a detach acknowledgment from the fabric. The port can't be reattached until it finishes this detach.

Procedure

Step 1

Confirm that the port state is DETACHED.

Example:

# vemcmd dpa dump attach
	=>dpa command is: dump attach
	LTL    Name               EPG-Alias               State         Tries  EPCP
	---------------------------------------------------------------------------
	51     ubuntu-vm-1        dvportgroup-7980        DETACHED      0      N
Step 2

Check that at least one OpFlex channel is active.

Example:

# vemcmd show opflex
	Status: 12 (Active)
	Channel0: 12 (Active), Channel1: 12 (Active)
Step 3

Restart the DPA.

Example:

# vem restart

Port EPG isn't downloaded

A VM port hasn't received an EPG from the fabric. The port may not forward traffic or may forward traffic with a stale EPG.

Procedure

Step 1

Confirm that the port state is WAIT_FOR_EPP.

Example:

# vemcmd dpa dump attach
	=>dpa command is: dump attach
	LTL    Name               EPG-Alias               State         Tries  EPCP
	---------------------------------------------------------------------------
	51     ubuntu-vm-1        dvportgroup-7980        WAIT_FOR_EPP  0      N
Step 2

Check that at least one OpFlex channel is active.

Example:

# vemcmd show opflex
	Status: 12 (Active)
	Channel0: 12 (Active), Channel1: 12 (Active)
Step 3

Detach and reattach the port.

Step 4

Restart the DPA.

Example:

# vem restart