The following Cisco APIC-EM technical support options are provided:

• Cisco APIC-EM hardware appliance:

  Hardware support is provided through the Cisco SMARTnet® Service.

• Cisco APIC-EM controller, basic apps, and services:

  If you have SMARTnet on any Cisco networking device, Cisco® TAC support is offered at no additional cost.

• Cisco APIC-EM solutions apps and services:

  TAC support is offered at no additional cost, if you have a SWSS (maintenance contract) on Cisco® Enterprise Management 3.x device licenses.

• When performing a Cisco APIC-EM discovery using an IP range, the ping sweep protocol (ICMP ping) is required. If ICMP ping is not enabled in your network, then use either the CDP discovery option for a discovery or directly add devices to the Cisco APIC-EM Device Inventory using the controller's GUI.

• Path Trace: Cisco Performance Routing or PfR is not supported with DMVPN tunnels for this release.

• The web GUI may take a few seconds to begin after the controller is started.

• When working with the Cisco APIC-EM in a network with several thousand supported devices, the Topology window may load slowly. Additionally, filtering within the other controller windows may also proceed slowly.

• Up to 4096 IP addresses are supported per discovery scan.

  Note	The IP address limit applies for one or more configured IP ranges in the controller’s GUI.

• The Cisco APIC-EM does not support duplicate IP addresses across VRFs in this release.

• Inventory and Topology VRF filters are only supported for Cisco IOS devices. Cisco non-IOS devices such as the Nexus devices are not supported with VRF filters.

• In a deployment with multiple inventory service instances running, re-sharding (rebalancing the devices to a remaining inventory instance when one of the inventory instances fails) occurs if any single inventory service instance fails. During this time, any device controlled by the failed service inventory instance displays in an inventory-collection pending state for a longer time than usual. Eventually, an inventory sync should occur without any issue.

• After deleting a user from the controller's database, we recommend that you do not reuse that username when creating a new user for at least 6 hours. This waiting period is required to ensure that the deleted user's access rights and privileges are not inherited when reusing the username.

• Cisco APIC-EM uses a master-slave database management system for the multi-host cluster. If the master host fails for any reason, then you will experience a 10 to 11 minute time interval when the controller UI is unavailable. This is due to the other two hosts recovering from that failure and re-establishing communications. If one of the slave hosts fails, there is no impact to the controller UI.

• If the interfaces of a switch connected to a 3 host APIC-EM cluster are shut down for 30 minutes (which disconnects the Cisco APIC-EM from the network), and then brought up, it takes a very long time for controller's GUI to become accessible.

• When power-cycling Cisco APIC-EM cluster hosts, the recommended procedure is to first power on the host that gets powered off last, and then power on the rest of the hosts within 30 seconds. This avoids any possible clustering issues. If this timing recommendation is exceeded, the host power-up sequence is not maintained, or any other problem occurs, then the best way to recover is to run the reset_grapevine command on one of the hosts in the cluster. When prompted, answer "n" to all questions.

• For a multi-host configuration with Cisco APIC-EM located behind a firewall that NATs the IP address of the controller, note the following information and requirement:

  • The Virtual IP address of the Cisco APIC-EM controller is used as a destination address for HTTP(S) traffic such as Cisco PnP and PKI download requests.

  • Any outbound connections initiated from the Cisco APIC-EM controller, such as during a Discovery, Inventory Collection, etc., will use the host IP address of one of the three Cisco APIC-EM hosts.

  • Therefore, you need to PAT (Port Address Translation) the host IP addresses of the Cisco APIC-EM hosts to a global public facing IP address for outbound connections from Cisco APIC-EM controller.

  The following illustration depicts an example of a Cisco APIC-EM multi-host or multi-node (N1, N2, N3) configuration with the Cisco APIC-EM deployed behind a firewall/NAT. Also depicted are examples of the NAT Public IP-to-VIP and PAT-to-Public IP settings.

  

• If a host fails within a multi-host configuration, the time for the cluster to recover is usually 20 minutes. The time to recover may change depending upon the Cisco APIC-EM disk I/O configuration. The recommended disk I/O speed is 200 MBps for a Cisco APIC-EM configuration.

• If you need to replace an existing 10 GB NIC card in a Cisco APIC-EM appliance that is part of a multi-host cluster, you need to first remove that host from the multi-host cluster, then power-off the appliance and replace the 10 GB NIC card, then re-install the Cisco APIC-EM ISO on the appliance. After these steps are finished, then rejoin the host to the multi-host cluster. For information about the procedures to remove and add a host to a multi-host cluster, see the Cisco Application Infrastructure Controller Enterprise Module Troubleshooting Guide.

• In a multi-host cluster with three hosts, if a single host (host A) is removed from the cluster for any reason, and the second host (host B) fails, then the last host (host C) will also immediately fail. To work around this limitation, perform the following procedure:

  1. Log into the last active host (host C) and run the config_wizard command.

  2. In the configuration wizard display, choose <Remove a faulted host from this APIC-EM cluster>.

  3. In the configuration wizard display, choose <Revert to single-host cluster>.

     The Grapevine services underpinning the original multi-host cluster are then removed and restarted.

  4. Access the displayed IP address with a browser to view the Grapevine developer console and view the progress as each service restarts.

  5. After host C is up and running, then proceed to reconfigure the multi-host cluster.

     Note	For information about configuring a multi-host cluster, see the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.

• To enable external authentication with a AAA server in a multi-host environment, you must configure all individual Cisco APIC-EM host IP addresses and the Virtual IP address for the multi-host cluster on the AAA server. For additional information about external authentication with the Cisco APIC-EM, see the Cisco Application Policy Infrastructure Controller Enterprise Module Administrator Guide.

• In certain circumstances, you may have only two operational hosts within a multi-host cluster (three hosts). For example, when in the process of setting up a multi-host cluster, you may have only two hosts set up before configuring the third or if a single host fails in your existing multi-host cluster. In these cases, the following functionality is unsupported for a multi-host cluster (three hosts) consisting of only two operational hosts:

  • Upgrading the software version

  • Installing the applications

  • Restoring a backup file

  • Restarting the cluster

  • Removing an active host, when there is an already a faulty host that exists and there is no reachability (IP connectivity) to the multi-host cluster

  • The reset grapevine command is not supported on a two-host cluster (either a 2 host multi-host cluster or 3 host multi-host cluster with one faulty host)

  Important

  The above functionality is only supported on a multi-host cluster that consists of three hosts.

• Simultaneous removal of two hosts from a multi-host cluster (three hosts) at once or a simultaneous addition of two hosts to a multi-host cluster (three hosts) at once is not supported.

• Cisco APIC-EM does not support shutting down 2 hosts in a 3 host cluster running a High Availability (HA) configuration. Only a single host at a time can be shut down and restarted when performing maintenance or troubleshooting in a 3 host cluster with HA.

• The restore of a controller backup file should only be done on a Cisco APIC-EM controller running the same software version with same set of applications as that used to create the backup

• When enabling or disabling an application on the controller, user downtime may result. For this reason, we recommend that you only perform these procedures during a maintenance time period.

• Once enabled, an application cannot be subsequently revoked and returned to its previous version or disabled status.

• You should not attempt to enable or disable an application on a host within a multi-host cluster if any one of the hosts within the multi-host cluster is down. If you do attempt to enable or disable an application in this situation, then the attempt will fail and a message will be displayed that the operation cannot be performed until all the hosts are up and running.

• As part of the application upgrade process, the controller only deletes the existing application that it successfully upgraded from in the controller's grape application display command output. The controller does not delete the record of any failed application upgrade attempts (stale application entries) in the controller's grape application display command output.

  For example, assume the last successful application installation version is 5.10 and attempts have been made to upgrade to other application versions (5.13, 5.14, and 5.15) which all failed due to various reasons. Assume that a successful upgrade was made to version 5.16. The following information will appear in the grape application display command output:

  
  $ grape application display
  
  apic-core                 enable_time               "Tue Aug 02, 2016 10:39:37 PM"
  apic-core                 enabled                   true
  apic-core                 enabled_by_default        true
  apic-core                 in_transition             false
  apic-core                 last_error_code           null
  apic-core                 last_result               "Successfully upgraded application=apic-core from version=5.10.1 to version=5.16"
  
  
  $ grape application status
  
  APPLICATION               VERSION         ENABLED         ENABLE TIME
  ------------------------------------------------------------------------------------------
  apic-core                 5.13            No              None
  apic-core                 5.14            No              None
  apic-core                 5.15            No              None
  apic-core                 5.16            Yes             Tue Aug 02, 2016 10:39:37 PM
  
  

  You can remove the stale application entries by using the grape application remove command, but this is not required for controller application separation functionality.

  The stale application entries will only appear in the controller's CLI command output. The stale application entries do not appear in the controller's GUI.

• With this release, the default option for intra-host communications is IPSec and not GRE. If you choose not to use the default option and to configure GRE using the configuration wizard, then privacy is not enabled for all of the communications that occur between the hosts. For this reason, we strongly recommend that any multi-host cluster that is not configured with IPSec tunneling be set up and located within a secure network environment.

• The Cisco APIC-EM should never be directly connected to the Internet. It should not be deployed outside of a NAT configured or protected datacenter environment. Additionally, when using the IWAN or PNP solution applications in a manner that is open to the Internet, you must configure a white-listing proxy or firewall to only allow incoming connections from your branch IP pools.

• The Cisco APIC-EM platform management service (Grapevine) no longer supports port 14141.

• Ensure that any external access to the Cisco APIC-EM using SSH (through port 22) is strictly controlled. We recommend that stringent measures be used, such as a segmented subnet as well as strict source address-based access policies in the port's access path.

• Ensure that the strict physical security of the Cisco APIC-EM appliance or server is enforced. For Cisco APIC-EM deployed within a virtual machine, ensure that strong and audited access restrictions are in place for the hypervisor management console.

• The Cisco APIC-EM backups are not encrypted when they are downloaded from the controller. If you download the backups from the controller, ensure that they are stored in a secure storage server and/or encrypted for storage.

• The Update button in the controller's Trustpool GUI window will become active when an updated version of ios.p7b file is available and Internet access is present. The Update button will remain inactive if there is no Internet access.

• As with any network management application, it is a general best practice to ensure that the traffic sent from Cisco APIC-EM to the managed devices is controlled in such a way as to minimize any security risks. More secure protocols (such as SSHv2 and SNMPv3) should be used rather than less secure ones (TELNET, SNMPv2), and network management traffic should be controlled (for example via access control lists or other types of network segmentation) to ensure that the management traffic is restricted to devices and segments of the network where it is needed.

• If you are currently using the Device PKI certificate management functionality for the network devices (for example, when using the IWAN App) and want to take advantage of the new feature to convert the private (internal) device PKI CA in the Cisco APIC-EM from Root CA mode to a Subordinate CA (or Intermediate CA to an external CA), then you must re-provision any of the Cisco APIC-EM provisioned network devices so they obtain the new PKCS12 bundle issued from this newly subordinated CA. If you later decide to convert the Cisco APIC-EM device PKI CA back to Root CA from Subordinate CA (also referred to as SubCA), then you need to reset the controller in order to accomplish this. Additionally, any devices provisioned with certificates by the Cisco APIC-EM in SubCA mode, need to be reprovisioned. For information about the new Cisco APIC-EM PKI certificate management functionality, see the Cisco Application Policy Infrastructure Controller Enterprise Module Administrator Guide.

• The Cisco APIC-EM controller does not provide a GUI or an API for replacing a subordinate CA certificate. Once SubCA mode is enabled using the controller, then the only way to replace the certificate of the Device PKI CA is to do a complete reset that brings the controller back to default root CA mode. You must then redo the conversion to subordinate CA mode using a new subordinate CA certificate. Before converting the controller back to the SubCA mode, you must also remove all device ID certificates and keys issued to network devices under the previous configuration of the Device PKI CA. The devices must be taken off line before converting the controller to SubCA mode with the new subordinate CA certificate, and then all devices will need to be reprovisioned by the PKI broker service using the new configuration of the Device PKI CA.

• Currently, there is no subordinate CA certificate rollover capability available for the Cisco APIC-EM device PKI. This capability is targeted for a near future release. For this reason, we strongly recommend that the subordinate CA certificate lifetime be set to at least two years. This will prevent any disruption to the device PKI due to a CA certificate expiration.

• When using the northbound REST API and creating a POST /trust-point request, this request must provide a trustProfileName attribute that has sdn-network-infra-iwan as its value (default). In the current release, no other values are valid for this required attribute.

• Due to a Cisco IOS XE crypto PKI import limitation, Cisco devices running certain IOS versions cannot import a PKCS bundle (made up of a device certificate, device key and the subordinate CA certificate) exceeding 4KB size.

  Note	Cisco devices running the following Cisco IOS versions or a higher version will not encounter this issue: 16.5(0.132) 16.3(2.11) 15.5(03)M4.1

  This problem occurs when the Cisco APIC-EM device PKI CA is changed to SubCA mode with a subordinate CA certificate that has several and/or lengthy X509 attributes defined, thereby increasing the size of the device PKCS bundle beyond 4KB. To circumvent this issue, get the subordinate CA certificate issued with very minimal attributes. For example, do not include CDP distribution and OCSP settings.

  The following command output is provided as an example of content from a subordinate CA certificate that can impact the file size, as well as the fields within the certificate where content should be minimized:

  
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              2e:00:00:00:0e:28:d7:1f:24:a1:1e:ef:70:00:00:00:00:00:0e
      Signature Algorithm: sha256WithRSAEncryption
          Issuer: DC=com, DC=apic-em, CN=apic-em-CA
          Validity
              Not Before: Oct 18 19:56:54 2016 GMT
              Not After : Oct 19 19:56:54 2016 GMT
          Subject: CN=sdn-network-infra-subca
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (2048 bit)
                  Modulus:
                      00:cd:a7:65:a4:c4:64:e6:e0:6b:f2:39:c0:a2:3b:
                      <snip>
                      85:a3:44:d1:a2:b3:b1:f5:ff:28:e4:12:41:d3:5f:
                      bf:e9
                  Exponent: 65537 (0x10001)
          X509v3 extensions:
              X509v3 Subject Key Identifier:
                  D2:DD:FA:E4:A5:6A:3C:81:29:51:B2:17:ED:82:CE:AA:AD:91:C5:1D
              X509v3 Authority Key Identifier:
                  keyid:62:6F:C7:83:42:82:5F:54:51:2B:76:B2:B7:F5:06:2C:76:59:7F:F8
  
              X509v3 Basic Constraints: critical
                  CA:TRUE
              X509v3 Key Usage: critical
                  Digital Signature, Certificate Sign, CRL Sign
              1.3.6.1.4.1.311.21.7:
                  0-.%+.....7.....#...I......^...Q...._...S..d...
      Signature Algorithm: sha256WithRSAEncryption
           18:ce:5b:90:6b:1d:5b:b4:df:fa:d3:8e:80:51:6f:46:0d:19:
  

• For this specific release your upgrade may fail if the Cisco Remote Troubleshooter application is installed and enabled. As a workaround, perform the following tasks:

  1. Disable the Cisco Remote Troubleshooter application in the App Management window of the controller's GUI.

  2. Install the Cisco APIC-EM software upgrade.

  3. Return to the App Management window and enable the Cisco Remote Troubleshooter application.

• Updating from earlier Cisco APIC-EM releases to this latest release may take up to an hour to complete.

• When updating Cisco APIC-EM in a virtual machine within a VMware vSphere environment, you must ensure that the time settings on the ESXi host are also synchronized to the NTP server. Failure to ensure synchronization will cause the upgrade to fail.

• Prior to beginning the software update process for the Cisco APIC-EM, we recommend that you configure the idle timeout value in the Auth Timeout GUI window for at least an hour. If a user is logged out due to an idle timeout during the software update process, then this process will fail and need to be re-initiated again.

  In case a failure occurs on a multi-host cluster during any software updates (Linux files) and you have not increased the idle timeout using the GUI, then perform the following steps:

  1. Log into each host and enter the following command: $ sudo cat /proc/net/xt_recent/ROGUE | awk '{print $1}’

     Note	This command will list all IP addresses that have been automatically blocked by the internal firewall because requests from these IP addresses have exceeded a predetermined threshold.

  2. If the command in Step 1 returns an IP address, then perform a reboot on the host where the above command has been entered (same host as the user is logged in).

     Note	The hosts should be rebooted in a synchronous order and never two hosts rebooted at the same time.

  3. After the host or hosts reboot, upload the software update package file to the controller again using the GUI.

Important

For the IWAN solution application, you must review the Software Configuration Guide for Cisco IWAN on APIC-EM before attempting a back up and restore. There is important and detailed information about how these processes work for the IWAN application that includes what is backed up, what is not backed up, recommendations, limitations, and caveats.

• Before attempting a back up and restore with a host in a multi-host cluster, note the following:

  • You cannot take a back up from a single host (not in a multi-host cluster) and then restore it to a host in a multi-host cluster.

  • You cannot take a back up from a host in a multi-host cluster and restore it to a single host (not in a multi-host cluster).

• When a user restores the controller from a backup file using the Cisco APIC-EM GUI, the password of that user and all other users will be reset to what is in the backup file.

• You can only restore a backup from a controller that is the same version from which the backup was taken. In addition to the controller version being the same as the backup, the enabled applications and version on the controller also need to be the same as the one on which the backup was taken.

• If you have configured a multi-host cluster with two or three hosts and not all of the hosts are running when you initiate a restore operation, then the restore operation will fail. All of the hosts that comprise the cluster must be in the cluster and operational at the time of the restore.

• Prior to beginning the backup and restore process for the Cisco APIC-EM, we recommend that you log out and then log back into the controller. This will ensure that the default forced session timeout for the Cisco APIC-EM does not occur during this process.

• Prior to beginning the backup and restore process for the Cisco APIC-EM, we recommend that you configure the idle timeout value in the Auth Timeout GUI window for at least an hour. If a user is logged out due to an idle timeout during the restore file upload process, then the restore process will fail and need to be re-initiated again.

• The default timezone (Universal Time Coordinated or UTC) should not be changed for the Cisco APIC-EM controller.

• Before deploying Cisco APIC-EM for NIC bonding setup, ensure that the port-channel in the switch side is in Bundled state.

• For a multi-host deployment, when joining a host to a cluster there is no merging of the data on the two hosts. The data that currently exists on the host that is joining the cluster is erased and replaced with the data that exists on the cluster that is being joined.

• For a multi-host deployment, when joining additional hosts to form a cluster be sure to join only a single host at a time. You should not join multiple hosts at the same time, as doing so will result in unexpected behavior.

• For a multi-host deployment, you should expect some service downtime when adding or removing hosts to a cluster, since the services are then redistributed across the hosts. Be aware that during the service redistribution, there will be downtime.

• The controller GUI starts up and becomes accessible prior to all the Cisco APIC-EM services starting up and becoming active. For this reason, you need to wait a few minutes before logging into the controller GUI under the following circumstances:

  • Fresh ISO image installation

  • Resetting the controller using the reset_grapevine command

  • Power failure and the controller restarts

• If you are installing the Cisco APIC-EM ISO image on a physical server using local media, you can use either a DVD drive, a bootable USB device, or a mounted VirtualMedia via CIMC (Cisco Integrated Management Controller for a Cisco UCS server). If you use a mounted VirtualMedia via CIMC, the installation process may take up to an hour. If you use a DVD drive or a bootable USB device, the installation process may take approximately 15 minutes.

• If you burn the APIC-EM ISO to a bootable USB flash drive and then boot the server from the USB flash drive, a “Detect and mount CD-ROM” error might display during installation. This typically occurs when you perform the installation on a clean, nonpartitioned hard drive. The workaround for the above issue is to perform the following steps:

  1. Press Alt+F2 to access the shell prompt.

  2. Enter the mount command to determine the device that is attached to the /media mount point. This should be your USB flash drive.

  3. Enter the umount /media command to unmount the USB flash drive.

  4. Enter the mount /dev/ device_path /cdrom command (where device_path is the device path of the USB flash drive) to mount the USB flash drive to the CD-ROM. For example:

     mount /dev/sda1 /cdrom

  5. Press Alt+F1 to return to the installation error screen.

  6. Click “Yes” to retry mounting the CD-ROM.

• When the configuration wizard is run to deploy the Cisco APIC-EM and the <save & exit> option is selected at the end of the configuration process instead of the proceed>> option, then you should always run the reset_grapevine command to bring the Cisco APIC-EM to an operational state. Failure to run the reset_grapevine command at the end of the deployment process after choosing the <save & exit> option in the configuration wizard will cause certain services to fail. The services that will fail are services that are brought up in the new VMs that are created and that depend upon the PKI certificates and stores. Services that do not depend upon the PKI certificates and stores will function properly.

• We strongly recommend that when creating usernames for the Cisco APIC-EM, that you always use lower case characters. Do not create two usernames that are the same, but have a different case. For example, do not create the following usernames: USER123 and user123.

• This version of the Cisco APIC-EM has been tested for external authentication with Cisco ISE based AAA servers, but it may support integration with other types of AAA servers.

• An installer (ROLE_INSTALLER) uses the Cisco Plug and Play Mobile App to access the Cisco APIC-EM controller remotely for the purposes of and triggering device deployment and viewing device status. An installer cannot access the Cisco APIC-EM GUI directly. If an installer needs to change his or her password, the admin must delete the user then create a new user with the same username and a new password.

• Users who are working with the controller's IWAN and PnP applications to monitor and manage devices and hosts must have their Groups values set to All. The IWAN and PnP applications do not support Custom groups. You set both roles and groups when configuring internal users in the Settings | Internal Users GUI window.

The Cisco APIC-EM developer website is located on the Cisco DevNet website