Contents

• PKI Planes in Cisco APIC-EM
• Overview of Secure Connections in Cisco APIC-EM
• PKI-Based Connections
• Non-PKI Secure Connections
• PKI Planes in Cisco APIC-EM
• HTTPS Connections to the Controller
• Expiration of the Controller's Server Certificate
• Revocation of the Controller's Server Certificate
• Device-to-Device DMVPN Connections
• Summary: PKI Planes in Cisco APIC-EM 1.2.x

PKI Planes in Cisco APIC-EM

---

Effective management of the Cisco APIC-EM PKI requires an understanding of the mechanisms that secure various types of network connection. This topic concerns itself primarily with PKI-based controller and device connections, describing other kinds of connections only for purposes of comparison and contrast with PKI-secured connections. Detailed descriptions of non-PKI connections are outside the scope of this discussion.

The Cisco APIC-EM provides PKI-based connections in two distinct PKI planes that do not share certificates, keys or CAs:

• Controller PKI Plane: HTTPS connections in which the controller is the server in the client-server model, and the controller's server certificate secures the connection.

• Device PKI Plane: DMVPN connections between devices in the control plane of the network, bilaterally authenticated and secured by the device ID certificates of both devices that participate in the connection.

Table 1 PKI Planes in Cisco APIC-EM

	Authentication	Encryption	Use Case
Controller PKI Plane: external caller initiates connection to controller
HTTPS	Caller presents username/pw or service ticket; Controller presents server certificate	Yes	REST client, including Cisco Network Plug N Play (PnP) mobile app or Cisco Prime Infrastucture
HTTPS	One-way: controller presents its server certificate	Yes	Cisco Network Plug N Play (PnP) provisioning workflows
Device PKI Plane: device-to-device connections
DMVPN	Bilateral authentication via IKEv2 using certs/keys issued by embedded private CA	Yes	DMVPN connections between devices

Overview of Secure Connections in Cisco APIC-EM

Two independent PKI planes (Controller PKI Plane and Device PKI Plane) secure two main categories of secure connection. The controller also supports other types of secure connection that do not use PKI.

PKI-Based Connections

All HTTPS connections use the Controller PKI Plane. Device-to-device connections use the Device PKI Plane, which is completely separate from the Controller PKI Plane.

Controller PKI Plane: Externally Initiated HTTPS Connections TO the Controller

When the controller responds to a request for an HTTPS session, it is the server in a client-server model that uses PKI to secure the connection. In response to the request for an HTTPS session, the controller presents its server certificate. Therefore, externally initiated HTTPS connections to the controller take place in the Controller PKI Plane.

HTTPS requests can come from devices in the control plane of the network or they can come from NB REST API callers. The controller never initiates HTTPS connections to devices.

Device PKI Plane: DMVPN Connections Between IWAN-Managed Devices

A separate PKI plane secures the Distributed Multipath VPN (DMVPN) connections that IWAN-managed devices form amongst themselves. This Device PKI Plane is managed by a private CA embedded in the APIC-EM controller. The Device PKI Plane and its private CA do not share any certificates, keys or CAs with the Controller PKI Plane, nor can any external CA manage the certificates and keys that secure the Device PKI Plane.

Non-PKI Secure Connections

The controller also supports the following secure connections that do not use PKI.

Controller-Initiated Non-PKI Secure Connections to Devices

Controller-initiated secure connections to devices can use SSH or Authenticated SNMPv3. These connections are authenticated, but they do not use a CA; therefore, these connections do NOT take place in the Controller PKI Plane.

• SSH from the controller: When the controller initiates an SSH connection to a device, the controller presents a shared secret (username/password pair) and the device presents its public key to create a secure connection.
• Authenticated SNMPv3: Authentication-enabled SNMPv3 uses a shared secret to establish trust between the controller and the device. When the controller uses authentication-enabled SNMPv3 to initiate a connection to a device, it presents a username and password that a trusted administrator supplied out-of-band by creating discovery credentials on the controller. If the device accepts the username and password, then the controller trusts the device. (Note that SNMPv3 can be configured not to authenticate the connection; if so, the connection is not secured and it is outside the scope of this discussion of secure connections. Optionally, authenticated SNMPv3 connections can also be encrypted.)

Externally Initiated Non-PKI Secure Connections to the Controller

SSH to the controller: When an external caller (such as an administrative remote terminal session) initiates an SSH connection to the controller, the controller presents its host public RSA or ECDSA key. Requests for SSH sessions come from administrators opening remote console sessions with the grapevine root. Network devices never initiate SSH connections to the controller.

Controller-to-Controller Secure Tunneling

Optionally, APIC-EM controllers in a multi-host cluster can be configured to use a secure, encrypted channel for communicating amongst themselves. This communications infrastructure is not accessible by means of any API or user interface. The security mechanism is not PKI-based; it uses IPSec tunnels secured by private keys that the grapevine root manages. Use of secure tunneling is optional and it must be enabled by the administrator when configuring the multi-host cluster; if secure tunneling is not enabled before starting the cluster, the hosts in a multi-host cluster communicate by means of GRE tunneling. For more information, see "Configuring IPSec Tunneling for Multi-Host Communications" in the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide.

PKI Planes in Cisco APIC-EM

The APIC-EM maintains multiple separate PKI planes that do not share certificates, keys or CAs. Each PKI plane secures a particular set of connections:

• Controller PKI Plane: Client-initiated HTTPS connections to the controller

  When an external caller initiates an HTTPS connection to the controller, the controller presents its server certificate. Such connections include the following:

  • Logins to the APIC-EM GUI via HTTPS

  • Logins to the Grapevine console GUI (port 14141) via HTTPS

  • Invocations of the NB REST API via HTTPS

    When a NB REST API caller initiates an HTTPS connection to the controller to invoke a NB REST API or to download a file (such as a device image, a configuration, and so on) the controller (server) presents its server certificate to the caller (client) that requested the connection.

    Only two NB REST APIs use HTTP instead of HTTPS: the API that downloads the trustpool bundle (GET /ca/trustpool), and the API that downloads the controller's certificate (GET /ca/pem). All other NB REST APIs utilize HTTPS.

  Note that controller-initiated connections to devices do NOT take place within the Controller PKI Plane. Even if the connections use SSH or SNMPv3, no CA manages the keys involved, so the connection is not considered to be PKI-based. The controller may initiate connections to devices for purposes that include discovery, managing tags, pushing policy to devices, or interacting with devices on on behalf of a REST caller. For compatibility with older devices, discovery can optionally use the TELNET protocol, which is insecure and therefore outside the scope of this PKI discussion.

• Device PKI Plane: Device-to-device DMVPN connections

  IWAN-managed control-plane devices form Dynamic Multipoint VPN (DMVPN) connections among themselves. A private Certificate Authority (CA) embedded in the Cisco APIC-EM provisions the certificates and keys that secure these DMVPN connections. An external CA CANNOT manage these certificates and keys. The PKI broker embedded in the Cisco APIC-EM manages these certificates and keys as directed by an admin in the IWAN GUI or as directed by a REST caller that uses the /certificate, /certificate-p12, /certificate-authority and /trust-point NB REST APIs. For more information, see Device-to-Device DMVPN Connections.

Important:

In the current release (1.2.x), the private CA embedded in the Cisco APIC-EM cannot be a sub-CA or Intermediate CA to any external CA. Until the Cisco APIC-EM adds such support, these two PKI planes shall remain completely independent of one another. In the current release (1.2.x), the IWAN devices’ mutual interaction certificates are managed only by the private CA that is embedded in the Cisco APIC-EM. External CAs cannot manage the IWAN-specific device ID certificates that devices present to each other for DMVPN tunnel creation and related operations.

HTTPS Connections to the Controller

When an external caller initiates an HTTPS connection to the Cisco APIC-EM controller, that connection takes place in the PKI plane that the controller's server certificate secures (the Controller PKI Plane.) In this client-server model, the controller is the server that presents its certificate to the client (REST client or Web browser) to establish a trusted connection. (If the controller is behind a firewall, its gateway-proxy certificate participates in the trust chain.)

The certificate that the controller (or proxy) presents can be self-signed or CA-issued. If the certificate is CA-issued, the caller may refer to the trustpool bundle to establish trust with the CA itself. A PnP-managed Cisco device downloaded the trustpool bundle from the controller as part of the Cisco Network Plug N Play (PnP) workflow that provisioned the device. A non-PnP Cisco device can also be configured to use the trustpool bundle. Non-Cisco devices or callers cannot use the trustpool bundle "as-is" but they can establish trust with trustpool CAs by means of their own certificate chain.

Upon establishing trust with the controller and any required CAs, the caller can use HTTPS for controller APIs, such as those which provide one-time downloads of configuration files, certificates, keys, and so on. For example, the PnP mobile application may initiate HTTPS with the controller for the purpose of provisioning network devices, but the deployment of configuration files to the devices by the controller takes place over a controller-initated SSH connection that is NOT within the Controller PKI Plane.

Expiration of the Controller's Server Certificate

The controller examines the expiration date of its installed server certificate and warns administrators of impending expiration. This warning appears in the controller GUI only. A ROLE_ADMIN user must take explicit action to replace the server certificate before it expires. This administrator can use the Controller Settings panel in the GUI or the /certificate NB REST API to replace the server certificate. Similarly, this admin can use the GUI or the /proxy-certificate NB REST API to replace the proxy certificate.

Revocation of the Controller's Server Certificate

Securely configured network devices typically would not connect to a controller that presents an expired or revoked server certificate, although it is possible for them to do so at their own risk. If configured to do so, a network device can contact the appropriate CA to learn of the revocation of a CA-issued server certificate by means of the Certificate Revocation List (CRL) or Online Certificate Status Protocol (OSCP).

Note	You can configure a non-PnP device to use an external CA. PnP-managed devices use the APIC-EM private CA only. In the current release of APIC-EM (1.2.x), a device cannot use both the private CA and an external CA.

It is important to understand that a non-PnP device may use CA-issued certificates without contacting the CA to determine whether the server certificate presented has been revoked. A device performs a revocation cross-check with the external CA only when configured to do so; that is, only when its CRL distribution point (CDP) points to the external CA. Without CRL check enabled, the device simply checks its own internal truststore of valid and/or private CA root certs along with the expiration date of the server cert that the controller presents to it. The device cannot determine whether the external CA that issued the controller cert has revoked that certificate unless the device is configured to check a CRL or to issue an OSCP request to the external CA.

The most likely circumstance under which a controller's server certificate might be revoked would be an explicit request by the controller admin to the external CA to revoke the server certificate. The administrator might issue this request if the controller was stolen or if failed hardware was returned to Cisco without having been processed properly for return. In this situation, it is reasonable to assume that the admin issuing the revocation request would know of the revocation and would take steps to generate and install a new, valid, CA-managed replacement server certificate on the controller or on a replacement controller.

Devices configured to interact with the trusted CA that manages the server certificate should continue to work correctly upon installation of another valid, CA-managed server certificate on the controller. Devices that do not interact with the trusted CA might need to be updated manually as necessary to trust the new server certificate; until this update takes place, these devices may refuse to connect to the controller, and REST API requests that involve these devices may fail.

PnP-managed devices never interact with any CA other than the APIC-EM private CA; therefore, the status of the APIC-EM server certificate is of no consequence to PnP-managed devices. Such devices might respond to a change in status of the private CA server certificate, however, as described in Device-to-Device DMVPN Connections.

If a Northbound REST caller follows best practice and requires a valid server certificate, it would refuse to connect to a controller that presents an invalid server certificate.

The APIC-EM controller itself does NOT interact with any external CA directly; therefore, it has no way to learn of the revocation of its server certificate by an external CA. Note, also, that the controller does not update its server certificate automatically under any circumstances. Replacement of an expired or revoked server certificate requires explicit action on the part of a ROLE_ADMIN user. Thus, it is important for the server admin to monitor GUI notifications and audit logs for PKI-related events.

Invalidation of an intermediary CA certificate in the trustpool bundle is a special case. The controller GUI does notify ROLE_ADMIN users when the trustpool bundle changes, and it provides a button that the admin can click to download and install a new trustpool bundle on the controller. However, the means by which network elements get the new trustpool bundle vary according to how the bundle was installed on those devices. Devices not managed by PnP cannot get the trustpool bundle from the controller, but they may be configured to download a new trustpool bundle from the cloud automatically. PnP-managed devices that got the trustpool bundle from the controller cannot get the new trustpool bundle from the controller, nor can they download it from the cloud, nor can the controller push a new trustpool bundle to these devices. However, as long as the Network Plug and Play-installed trustpool bundle has a valid root CA certificate, those devices will continue to trust the controller after installation of the new trustpool bundle on the controller. The same is true of devices not managed by PnP. Therefore, although best practice recommends manual update of devices with the new trustpool bundle in timely fashion, a change to an intermediary CA is not likely to cause an immediate problem.

Important:

The private CA that manages Distributed Multipoint VPN connections does not manage the controller's server certificate. Therefore, it cannot provide revocation status of the controller's server certificate. For more information, see Device-to-Device DMVPN Connections.

Device-to-Device DMVPN Connections

IWAN-managed devices can form Dynamic Multipoint VPN (DMVPN) connections among themselves. The controller-embedded private CA and the pki-broker service work together to provision the device certificates and keys that secure these DMVPN connections. The pki-broker service also exposes a NB REST API that can be used to manage these device ID certificates and keys manually. You can use this API for purposes such as programmatically migrating an existing ("brownfield") device deployment from the use of shared-secret authentication of DMVPN connections to the use of PKI-based authentication of DMVPN connections.

It is important to understand that the controller's embedded private CA is not recognized by any external CA as an intermediate CA in the current release (1.2.x). Therefore, this internal CA is not a member of the trustpool (ios.p7b) bundle that the APIC-EM provides to devices in the Network Plug n Play provisioning workflow. External CAs in the trustpool bundle have no knowledge of the certificates that the controller's internal CA doles out to IWAN devices privately. Certificates issued by the embedded private CA can be revoked only by using the /trust-point NB REST API that the Cisco APIC-EM controller exposes.

IWAN-participating devices learn of the NB API-based revocation of internal CA-issued certificates by means of the CRL distribution point that is the internal CA itself. When two devices attempt to create a DMVPN tunnel, they present device ID certificates to each other. To determine whether the certificate presented to it has been revoked, each device polls its CRL. Whenever a certificate is revoked, the private CA prompts all IWAN-participating devices to get a new CRL.

A private CA-issued certificate (which is used to secure DMVPN connections) is valid for one year from the date of issue. IWAN-participating devices can attempt automated renewal of a private CA-issued certificate before the certificate actually expires. Expiration or renewal of a private CA-issued certificate generates PKI events that appear in the audit logs.

The request for a certificate or a CRL from the controller's embedded private CA could be viewed as the point at which the two PKI planes intersect, though in different contexts. This request from a device to the controller requires a trust relationship that the controller’s server certificate guarantees; this certificate is NOT issued by the APIC-EM’s embedded private CA. However, the payload of the response concerns the device ID certificate that the embedded private CA issues to the device as part of the initial PnP provisioning workflow. This device ID certificate protects the DMVPN connections that IWAN-participating devices form in the Device PKI Plane. A similar context applies when the device responds to the prompt to retrieve a new CRL from the private CA.

To summarize, device lifecycle-management operations occur in the Controller PKI Plane that the controller's server certificate protects. Such operations include initial provisioning of PnP-managed devices, retrieval of updated CRLs and renewals of device ID certificates. Although the requests themselves are protected by the server certificate, the PKI payloads of the responses concern device ID certificates, keys and CRLs that protect transactions in the Device PKI Plane, such as DMVPN tunneling among IWAN devices themselves.

Summary: PKI Planes in Cisco APIC-EM 1.2.x

The following factors govern secure connections to the APIC-EM controller and secure device-to-device connections:

• PKI for network control plane is completely separate from PKI for connections to APIC-EM controller

  Two completely independent PKI planes separate the controller's interaction with external callers from device-to-device interactions in the network control plane. These PKI planes do not share any CAs, certs or keys.

  • Device PKI Plane: DMVPN connections between network devices

    Devices managed by Cisco Network Plug N Play (PnP-managed devices) make device-to-device DMVPN connections autonomously among themselves. A private Certificate Authority (CA) embedded in the controller manages the certificates and keys that secure these connections. Certificates and keys managed by this embedded private CA cannot be managed by any external CA. In the current release (1.2.x), the private CA cannot be a sub-CA or intermediate CA to any other CA. Devices cannot be "hybrid provisioned" to interact with both the embedded private CA and another CA.

  • Controller PKI Plane: HTTPS connections TO the controller secured by controller's certificate

    The controller presents its server certificate in response to HTTPS connection requests. This certificate can be self-signed (default) or CA-managed (recommended), but the controller itself does not interact with any external CA.

  Note	Connections FROM the controller to devices do NOT take place within the controller PKI plane, even if they use SSH or SNMPv3.

• PnP-managed devices vs. non-PnP devices

  Devices managed by Cisco Network Plug N Play (PnP-managed devices) interact with the controller's embedded private CA. They do not interact with any other CA. Therefore they cannot learn of revocation of the controller's server certificate or public key from an external CA. The embedded private CA does not manage the controller's server certificate or anything else in the controller PKI plane.

  Devices not managed by PnP can interact with an external CA to learn of revocation of the controller's server certificate or public key, but the controller itself does not interact with an external CA, nor does it provide any sort of automated management of its server certificate. For example, upon impending expiry of its server certificate that was issued by a valid external CA, the controller does not automatically initiate renewal with the external CA. The controller admin must take explicit action to upload a renewed certificate to the controller. In contrast, as the certificate on a PnP-managed device approaches expiry, the device can renew its certificate automatically with the controller's internal RootCA (embedded private CA) before the certificate actually expires.

  Devices cannot be "hybrid provisioned" to interact with both the embedded private CA and any other CA. Effectively, there are two classes of devices: those managed by PnP, and those not managed by PnP.

• Device-initiated connections vs. controller-initiated connections

  When a device initiates connection to the controller, it uses HTTP(S) or SNMPv2. Only HTTPS is secure; in response to an HTTPS request, the controller presents its certificate.

  • If the device is PnP-managed, it cannot interact with an external CA to learn of revocation of the controller's server certificate. Therefore, unless the certificate has expired (which the device can determine without external assistance), the device accepts the cert and the connection succeeds. This is because the "CRL distribution point" (CDP) for a PnP-enabled device is the controller's internal private CA itself, which has no knowledge of the controller's server certificate or any other CA.

  • If the device is not PnP-managed and the device is configured to contact an external CA for certificate revocation status, the device may reject the controller's server certificate based on information it gets from the CA. For example, if the CA says that the certificate is revoked, or if the certificate is not managed by the particular CA that the device queries about this certificate, the device rejects the connection. If the controller presents a self-signed certificate, the external CA will not recognize the certificate as valid and the device will refuse to connect. However, it is possible for devices to use CA-signed certificates without performing revocation cross-check; for important details, see Revocation of the Controller's Server Certificate.

  When the controller initiates connection to the device, it can use SSH, SNMP or TELNET. Only SSH and SNMPv3 (if configured appropriately) are secure, but controller-initiated connections to devices do NOT take place within the controller PKI plane.

  • For SSH connections, the controller presents a username/password pair to the device and the device returns its public host RSA or ECDSA key (not its device ID certificate) to the controller. The shared secret (username/password pair) validates the identity of the controller. For such connections, it is unimportant whether the controller uses a self-signed or CA-managed certificate, because the controller never presents its certificate to the device.

  • For SNMPv3 connections, the controller presents a username/password pair to the device. The device opens a connection according to its SNMPv3 configuration. The SNMPv3 protocol provides separate parameters that control the authentication and encryption behavior of the connection. The minimum secure configuration requires authentication; if authentication is enabled, the option to encrypt the connection is also available. For more information, see "Configuring SNMPv3" in the Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide.

Copyright © 2016, Cisco Systems, Inc. All rights reserved.