The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This chapter contains the following sections:
Within the Cisco Application Centric Infrastructure (ACI) fabric, time synchronization is a crucial capability upon which many of the monitoring, operational, and troubleshooting tasks depend. Clock synchronization is important for proper analysis of traffic flows as well as for correlating debug and fault time stamps across multiple fabric nodes.
An offset present on one or more devices can hamper the ability to properly diagnose and resolve many common operational issues. In addition, clock synchronization allows for the full utilization of the atomic counter capability that is built into the ACI upon which the application health scores depend. Nonexistent or improper configuration of time synchronization does not necessarily trigger a fault or a low health score. You should configure time synchronization before deploying a full fabric or applications so as to enable proper usage of these features. The most widely adapted method for synchronizing a device clock is to use Network Time Protocol (NTP).
Prior to configuring NTP, consider what management IP address scheme is in place within the ACI fabric. There are two options for configuring management of all ACI nodes and Application Policy Infrastructure Controllers (APICs), in-band management and/or out-of-band management. Depending upon which management option is chosen for the fabric, configuration of NTP will vary. Another consideration in deploying time synchronization is where the time source is located. The reliability of the source must be carefully considered when determining if you will use a private internal clock or an external public clock.
Out-of-band management NTP—When an ACI fabric is deployed with out-of-band management, each node of the fabric, inclusive of spines, leaves, and all members of the APIC cluster, is managed from outside the ACI fabric. This IP reachability will be leveraged so that each node can individually query the same NTP server as a consistent clock source. To configure NTP, a Date and Time policy must be created that references an out-of-band management endpoint group. Date and Time policies are confined to a single pod and must be deployed across all pods provisioned in the ACI fabric. Currently only one pod per ACI fabric is allowed.
In-Band Management NTP—When an ACI fabric is deployed with in-band management, consider the reachability of the NTP server from within the ACI in-band management network. In-band IP addressing used within the ACI fabric is not reachable from anywhere outside the fabric. To leverage an NTP server external to the fabric with in-band management, construct a policy to enable this communication. The steps used to configure in-band management policies are identical to those used to establish an out-of-band management policy. The distinction is around how to allow the fabric to connect to the NTP server.
NTP over IPv6 addresses is supported in hostnames and peer addresses. The gai.conf can also be set up to prefer the IPv6 address of a provider or a peer over an IPv4 address. The user can provide a hostname that can be resolved by providing an IP address (both IPv4 or IPv6, depending on the installation or preference).
When an ACI fabric is deployed with out-of-band management, each node of the fabric is managed from outside the ACI fabric. You can configure an out-of-band management NTP server so that each node can individually query the same NTP server as a consistent clock source.
This example shows how to configure a preferred out-of-band NTP server and how to verify the configuration and deployment.
apic1# configure t apic1(config)# template ntp-fabric pol1 apic1(config-template-ntp-fabric)# server 192.0.20.123 use-vrf oob-default apic1(config-template-ntp-fabric)# no authenticate apic1(config-template-ntp-fabric)# authentication-key 12345 apic1(config-template-ntp-fabric)# trusted-key 12345 apic1(config-template-ntp-fabric)# exit apic1(config)# template pod-group allPods apic1(config-pod-group)# inherit ntp-fabric pol1 apic1(config-pod-group)# exit apic1(config)# pod-profile all apic1(config-pod-profile)# pods all apic1(config-pod-profile-pods)# inherit pod-group allPods apic1(config-pod-profile-pods)# end apic1#
apic1# show ntpq nodeid remote refid st t when poll reach delay offset jitter ------ - ------------ ------ ---- -- ----- ----- ----- ------ ------ ------ 1 * 192.0.20.123 .GPS. u 27 64 377 76.427 0.087 0.067 2 * 192.0.20.123 .GPS. u 3 64 377 75.932 0.001 0.021 3 * 192.0.20.123 .GPS. u 3 64 377 75.932 0.001 0.021
A DHCP relay policy may be used when the DHCP client and server are in different subnets. If the client is on an ESX hypervisor with a deployed vShield Domain profile, then the use of a DHCP relay policy configuration is mandatory.
When a vShield controller deploys a Virtual Extensible Local Area Network (VXLAN), the hypervisor hosts create a kernel (vmkN, virtual tunnel end-point [VTEP]) interface. These interfaces need an IP address in the infrastructure tenant that uses DHCP. Therefore, you must configure a DHCP relay policy so that the APIC can act as the DHCP server and provide these IP addresses.
When an ACI fabric acts as a DHCP relay, it inserts the DHCP Option 82 (the DHCP Relay Agent Information Option) in DHCP requests that it proxies on behalf of clients. If a response (DHCP offer) comes back from a DHCP server without Option 82, it is silently dropped by the fabric. Therefore, when the ACI fabric acts as a DHCP relay, DHCP servers providing IP addresses to compute nodes attached to the ACI fabric must support Option 82.
The port and the encapsulation used by the application Endpoint Group must belong to a physical or VM Manager (VMM) domain. If no such association with a domain is established, the APIC continues to deploy the EPG but raises a fault.
Cisco APIC supports DHCP relay for both IPv4 and IPv6 tenant subnets. DHCP server addresses can be IPv4 or IPv6. DHCPv6 relay will occur only if IPv6 is enabled on the fabric interface and one or more DHCPv6 relay servers are configured.
Deploying DHCP Relay Policy for an Endpoint Group
Make sure that Layer 2 or Layer 3 management connectivity is configured.
Deploying DHCP Relay Policy for a Layer 3 Out
The port and the encapsulation used by the application Endpoint Group must belong to a physical or VM Manager (VMM) domain. If no such association with a domain is established, the APIC continues to deploy the EPG but raises a fault.
Cisco APIC supports DHCP relay for both IPv4 and IPv6 tenant subnets. DHCP server addresses can be IPv4 or IPv6. DHCPv6 relay will occur only if IPv6 is enabled on the fabric interface and one or more DHCPv6 relay servers are configured.
Ensure that Layer 2 or Layer 3 connectivity is configured to reach the DHCP server address.
This task is a prerequisite for users who want to create a vShield Domain Profile.
The port and the encapsulation used by the application Endpoint Group must belong to a physical or VM Manager (VMM) domain. If no such association with a domain is established, the APIC continues to deploy the EPG but raises a fault.
Cisco APIC supports DHCP relay for both IPv4 and IPv6 tenant subnets. DHCP server addresses can be IPv4 or IPv6. DHCPv6 relay will occur only if IPv6 is enabled on the fabric interface and one or more DHCPv6 relay servers are configured.
Make sure that Layer 2 or Layer 3 management connectivity is configured.
A DNS policy is required to connect to external servers, for example AAA, RADIUS, vCenter, and services by hostname. A DNS service policy is a shared policy, so any tenant and VRF that uses this service must be configured with the specific DNS profile label. To configure a DNS policy for the ACI fabric, you must complete the following tasks:
Ensure that the management EPG is configured for the DNS policy, otherwise this policy will not take into effect on the switches.
Create a DNS profile (default) that contains the information about DNS providers and DNS domains.
Associate the DNS profile (default or another DNS profile) name to a DNS label under the required tenant.
It is possible to configure a per-tenant, per-VRF DNS profile configuration. Additional DNS profiles can be created and applied to specific VRFs of specific tenants using the appropriate DNS label. For example, if you create a DNS profile with a name of acme, you can add a DNS label of acme to the appropriate
policy configuration in the tenants configuration.Configure the external destinations for the services as follows:
Source | In-Band Management | Out-of-Band Management | External Server Location | ||
---|---|---|---|---|---|
APIC |
IP address or Fully Qualified domain name (FQDN) |
IP address or FQDN |
Anywhere |
||
Leaf switches |
IP address |
IP address or FQDN
|
Anywhere |
||
Spine switches |
IP address |
IP address or FQDN
|
Directly connected to a leaf switch |
The following is a list of external servers:
Call Home SMTP server
Syslog server
SNMP Trap destination
Statistics Export destination
Configuration Export destination
Techsupport Export destination
Core Export destination
The recommended guidelines are as follows:
The external servers must be atatched to the leaf access ports.
Use in-band connectivity for the leaf switches to avoid extra cabling for the management port.
Use out-of-band management connectivity for the spine switches. Connect this out-of-band network for spine switches to one of the leaf ports with in-band management virtual routing and forwarding (VRF) so that the spine switches and the leaf switches can reach the same set of external servers.
Use IP addresses for the external servers.
DNS servers have primary DNS records which can be A records (IPV4) or AAAA records (IPV6). Both A and AAAA records associate domain name with a specific IP address (IPv4 or IPv6).
The ACI fabric can be configured to use reputable public DNS servers that run on IPv4. These servers are able to resolve and respond with A record (IPv4) or AAAA record (IPv6).
In a pure IPv6 environment, the system administrators must use IPv6 DNS servers. The IPv6 DNS servers are enabled by adding them to /etc/resolv.conf.
A more common environment is to have dual-stack IPv4 and IPv6 DNS servers. In the dual-stack case, both IPv4 and IPv6 name servers are listed in /etc/resolv.conf. However, in a dual-stack environment, simply appending the IPv6 DNS servers to the list may cause a large delay in DNS resolutions. This is because the IPv6 protocol takes precedence by default, and it is unable to connect to the IPv4 DNS servers (if they are listed first in /etc/resolv.conf). The solution is to list IPv6 DNS servers ahead of IPv4 DNS servers. Also add “options single-request-reopen” to enable the same socket to be used for both IPv4 and IPv6 lookups.
options single-request-reopen nameserver 2001:4860:4680::8888 nameserver 2001:4860:4680::8844 nameserver 8.8.8.8 nameserver 8.8.4.4
If the management network in the ACI fabric supports both IPv4 and IPv6, the Linux system application (glibc) will use the IPv6 network by default because getaddrinfo() will return IPv6 first.
Under certain conditions however, an IPv4 address may be preferred over an IPv6 address. The Linux IPv6 stack has a feature which allows an IPv4 address mapped as an IPv6 address using IPv6 mapped IPv4 address (::ffff/96). This allows an IPv6 capable application to use only a single socket to accept or connect both IPv4 and IPv6. This is controlled by the glibc IPv6 selection preference for getaddrinfo() in /etc/gai.conf.
In order to allow glibc to return multiple addresses when using /etc/hosts, “multi on” should be added to the /etc/hosts file. Otherwise, it may return only the first match.
If an application is not aware whether both IPv4 and IPv6 exist, it may not perform fallback attempts using different address families. Such applications may require a fallback implementation.
The DNS profile supports version preference choices between IPv4 and IPv6. Using the user interface, you can enable your preference. IPv4 is the default.
The following is an example of a policy based configuration using Postman REST API:
<?xml version="1.0" encoding="UTF-8”?> <!— api/node/mo/uni/fabric/dnsp-default.xml —> <dnsProfile dn="uni/fabric/dnsp-default" IPVerPreference="IPv6" childAction="" descr="" > </dnsProfile>
The gai.conf settings control destination address selection. The file has a label table, precedence table, and an IPv4 scopes table. The changes for prioritizing IPv4 or IPv6 over the other need to go into the precedence table entries. Given below are sample contents of the standard file as it is used in Linux systems for many flavors. A single line of precedence label in the file overrides any default settings.
The following is an example of a gai.conf to prioritize IPv4 over IPv6:
# Generated by APIC label ::1/128 0 label ::/0 1 label 2002::/16 2 label ::/96 3 label ::ffff:0:0/96 4 precedence ::1/128 50 precedence ::/0 40 precedence 2002::/16 30 precedence ::/96 20 # For APICs prefering IPv4 connections, change the value to 100. precedence ::ffff:0:0/96 10
Make sure that Layer 2 or Layer 3 management connectivity is configured.
Step 1 | On the menu bar, choose Navigation pane, expand , and click the default DNS profile. . In the |
Step 2 | In the Work pane, in the Management EPG field, from the drop-down list, choose the appropriate management EPG (default (Out-of-Band)). |
Step 3 | Expand
DNS
Providers, and perform the following actions:
|
Step 4 | Expand
DNS
Domains, and perform the following actions:
|
Step 5 | Click Submit. The DNS server is configured. |
Step 6 | On the menu bar, click . |
Step 7 | In the Navigation pane, expand . |
Step 8 | In the Work pane, under Properties, in the DNS labels field, enter the appropriate DNS label (default). Click Submit. The DNS profile label is now configured on the tenant and VRF. |
Make sure that Layer 2 or Layer 3 management connectivity is configured.
Step 1 | On the menu bar, choose Navigation pane, expand , and click the default DNS profile. . In the |
Step 2 | In the Work pane, in the Management EPG field, from the drop-down list, choose the appropriate management EPG (default (Out-of-Band)). |
Step 3 | Expand
DNS
Providers, and perform the following actions:
|
Step 4 | Expand
DNS
Domains, and perform the following actions:
|
Step 5 | Click Submit. The DNS server is configured. |
Step 6 | On the menu bar, click . |
Step 7 | In the Navigation pane, expand , and click oob. |
Step 8 | In the Work pane, under Properties, in the DNS labels field, enter the appropriate DNS label (default). Click Submit. The DNS profile label is now configured on the tenant and VRF. |
Step 1 | In the NX-OS
CLI, get into configuration mode, shown as follows:
Example: apic1# configure apic1(config)# |
Step 2 | Configure a DNS
server policy.
Example: apic1(config)# dns apic1(config-dns)# address 172.21.157.5 preferred apic1(config-dns)# address 172.21.157.6 apic1(config-dns)# domain company.local default apic1(config-dns)# use-vrf oob-default |
Step 3 | Configure a DNS
profile label on any VRF where you want to use the DNS profile.
Example: apic1(config)# tenant mgmt apic1(config-tenant)# vrf context oob apic1(config-tenant-vrf)# dns label default |
Make sure that Layer 2 or Layer 3 management connectivity is configured.
Step 1 | Configure the
DNS service policy.
Example: POST URL : https://apic-IP-address/api/node/mo/uni/fabric.xml <dnsProfile name="default"> <dnsProv addr="172.21.157.5" preferred="yes"/> <dnsProv addr="172.21.157.6"/> <dnsDomain name="cisco.com" isDefault="yes"/> <dnsRsProfileToEpg tDn="uni/tn-mgmt/mgmtp-default/oob-default"/> </dnsProfile> |
Step 2 | Configure the
DNS label under the out-of-band management tenant.
Example: POST URL: https://apic-IP-address/api/node/mo/uni/tn-mgmt/ctx-oob.xml <dnsLbl name="default" tag="yellow-green"/> |
Step 1 | Verify the
configuration for the default DNS profile.
Example: apic1# show running-config dns # Command: show running-config dns # Time: Sat Oct 3 00:23:52 2015 dns address 172.21.157.5 preferred address 172.21.157.6 domain company.local default use-vrf oob-default exit |
Step 2 | Verify the
configurations for the DNS labels.
Example: apic1# show running-config tenant mgmt vrf context oob # Command: show running-config tenant mgmt vrf context oob # Time: Sat Oct 3 00:24:36 2015 tenant mgmt vrf context oob dns label default exit exit |
Step 3 | Verify that the
applied configuration is operating on the fabric controllers.
Example: apic1# cat /etc/resolv.conf # Generated by IFC nameserver 172.21.157.5 nameserver 172.21.157.6 |
Wildcard certificates (such as *.cisco.com, which is used across multiple devices) and its associated private key generated elsewhere are not supported on the APIC as there is no support to input the private key or password in the APIC. Also, exporting private keys for any certificates, including wildcard certificates, is not supported.
You must download and install the public intermediate and root CA certificates before generating a Certificate Signing Request (CSR). Although a root CA Certificate is not technically required to generate a CSR, Cisco requires the root CA certificate before generating the CSR to prevent mismatches between the intended CA authority and the actual one used to sign the CSR. The APIC verifies that the certificate submitted is signed by the configured CA.
To use the same public and private keys for a renewed certificate generation, you must satisfy the following guidelines:
You must preserve the originating CSR as it contains the public key that pairs with the private key in the key ring.
The same CSR used for the originating certificate must be resubmitted for the renewed certificate if you want to re-use the public and private keys on the APIC.
Do not delete the original key ring when using the same public and private keys for the renewed certificate. Deleting the key ring will automatically delete the associated private key used with CSRs.
CAUTION: PERFORM THIS TASK ONLY DURING A MAINTENANCE WINDOW AS THERE IS A POTENTIAL FOR DOWNTIME. The downtime affects access to the APIC cluster and switches from external users or systems and not the APIC to switch connectivity. The NGINX process on the switches will also be impacted but that will be only for external connectivity and not for the fabric data plane. Access to the APIC, configuration, management, troubleshooting and such will be impacted. Expect a restart of all web servers in the fabric during this operation.
Determine from which authority you will obtain the trusted certification so that you can create the appropriate Certificate Authority.
Step 1 | On the menu bar, choose . | ||
Step 2 | In the Navigation pane, choose . | ||
Step 3 | In the Work pane, choose . | ||
Step 4 | In the Create Certificate Authority dialog box, in the Name field, enter a name for the certificate authority. | ||
Step 5 | In the Certificate Chain field, copy the intermediate and root certificates for the certificate authority that will sign the Certificate Signing Request (CSR) for the Application Policy Infrastructure Controller (APIC).
-----BEGIN CERTIFICATE----- <Intermediate Certificate> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <Root CA Certificate> -----END CERTIFICATE----- | ||
Step 6 | Click Submit. | ||
Step 7 | In the Navigation pane, choose . | ||
Step 8 | In the Work pane, choose . | ||
Step 9 | In the Create Key Ring dialog box, in the Name field, enter a name. | ||
Step 10 | In the Certificate field, do not add any content. | ||
Step 11 | In the Modulus field, click the radio button for the desired key strength. | ||
Step 12 | In the Certificate Authority field, from the drop-down list, choose the certificate authority that you created earlier. Click Submit.
| ||
Step 13 | In the Navigation pane, choose . | ||
Step 14 | In the Work pane, choose . | ||
Step 15 | In the Subject field, enter the fully qualified domain name (FQDN) of the APIC. | ||
Step 16 | Fill in the remaining fields as appropriate.
| ||
Step 17 | Click Submit. The object is created and displayed in the Navigation pane under the key ring you created earlier. In the Navigation pane, click the object and in the Work pane, in the Properties area, in the Request field the CSR is displayed. Copy the contents from the field to submit to the Certificate Authority. | ||
Step 18 | In the Navigation pane, choose . | ||
Step 19 | In the Work pane, in the Certificate field, paste the signed certificate that you received from the certificate authority. | ||
Step 20 | Click Submit.
| ||
Step 21 | On the menu bar, choose . | ||
Step 22 | In the Navigation pane, choose . | ||
Step 23 | In the Work pane, in the Admin Key Ring drop-down list, choose the desired key ring. | ||
Step 24 | Click Submit. All web servers restart. The certificate is activated, and the non-default key ring is associated with HTTPS access. |
You must remain aware of the expiration date of the certificate and take action before it expires. To preserve the same key pair for the renewed certificate, you must preserve the CSR as it contains the public key that pairs with the private key in the key ring. Before the certificate expires, the same CSR must be resubmitted. Do not delete or create a new key ring as deleting the key ring will delete the private key stored internally on the APIC.