Chapter 1. New and Changed Information

This chapter contains the following sections:

- New and changed information

New and changed information

The table provides an overview of the significant changes to this guide for this release. The table does not provide an exhaustive list of all changes made to the guide or of the new features up to this release.

New features and changed information for Cisco APIC release 6.2(1)

Feature

Description

Where documented

Secure initial admin password for virtual APIC

An extra layer of security has been added for clusters with virtual APICs. After you bring up a new virtual APIC, you will be prompted to change the password.

Bring up the APIC cluster using the GUI

Chapter 2. Initial Setup

This chapter contains the following sections:

- Additional documents

- APIC configuration methods

- Change the default BIOS password

- Cisco Application Centric Infrastructure

- Local serial connection for basic configuration

- Access the APIC GUI

- Access REST API

- Access the NX-OS style CLI

- Access the object model CLI

Additional documents

Provide information about related documents that are useful references along with the Cisco APIC Getting Started Guide.

For next steps, see....

All related documents are available on the APIC documents landing page. The document names are indicative and will help you to find relevant information about APIC concepts, funtionality and behavior.

APIC configuration methods

You can configure Cisco APIC using the GUI, REST API or the CLI method.

In several instances, the NX-OS style CLI can create the ACI model constructs implicitly for a user's ease of use, and they also provide validations to ensure consistency in configuration. This functionality reduces and prevents faults.

For further details about configurations and tasks, see the Cisco APIC Basic Configuration Guide and the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide.

Change the default BIOS password

Use this procedure to change the BIOS password.

Cisco APIC ships with a default BIOS password. The default password is "password". When the boot process starts, the boot screen displays the BIOS information on the console server.

Procedure

Step 1

During the BIOS boot process, when the screen displays Press <F2> Setup, press F2.

The Entering Setup message displays as it accesses the setup menu.

Step 2

At the Enter Password dialog box, enter the current password.

The default password can be password or Insieme123.

Step 3

In the Setup Utility, choose the Security tab, and choose Set Administrator Password.

Step 4

In the Enter Current Password dialog box, enter the current password.

Step 5

In the Create New Password dialog box, enter the new password.

Step 6

In the Confirm New Password dialog box, re-enter the new password.

Step 7

Choose the Save & Exit tab.

Step 8

In the Save & Exit Setup dialog box, choose Yes.

Step 9

Wait for the reboot process to complete.

The updated BIOS password is effective.

Cisco Application Centric Infrastructure

The Cisco Application Centric Infrastructure (ACI) is a distributed, scalable, multitenant infrastructure with external endpoint connectivity controlled and grouped through application-centric policies.

The centralized management controller for ACI is the Application Policy Infrastructure Controller (APIC). The Application Policy Infrastructure Controller (APIC) is a cluster-based management controller that

- serves as the unified point for automation, management, monitoring, and programmability for the ACI. The APIC supports the deployment, management, and monitoring of any application anywhere, with a unified operations model for the physical and virtual components of the infrastructure.

- programmatically automates network provisioning and control based on application requirements and policies. It is the central control engine for the broader cloud network. It simplifies management and allows flexibility in how application networks are defined and automated.

- provides northbound REST APIs to support integration and custom workflows.

The APIC is a distributed system that is implemented as a cluster of many controller instances.

Local serial connection for basic configuration

This section describes how to establish a local serial connection to the Cisco APIC server to begin the initial basic configuration. For additional connection information, including instructions on connecting to the server remotely for setup, refer to "Initial Server Setup" in the Cisco APIC M4/L4 server installation and service guide. If you are on Cisco APIC release 6.1(4), you can refer the Cisco APIC G5 server installation and service guide.

Initial Connection

The Cisco APIC server operates on a Cisco Integrated Management Controller (CIMC) platform. You can make an initial connection to the CIMC platform using one of these methods:

- Use a KVM cable (Cisco PID N20-BKVM) to connect a keyboard and monitor to the KVM connector on the front panel of the server.

If you want to use vKVM, see Installing an OS Using the KVM Console in Cisco UCS C-Series Integrated Management Controller GUI Configuration Guide, Release 4.3.

- Connect a USB keyboard and VGA monitor to the corresponding connectors on the rear panel of the server.

Note

You cannot use the front panel VGA and the rear panel VGA at the same time.

You can make a serial connection using one of the following methods. Two of these methods require a configuration change in the CIMC:

Note

You cannot use more than one of these methods simultaneously.

- Use the DB9 connector of the KVM cable

- Use the rear panel RJ-45 console port (after enabling in the CIMC)

- Connect by Serial-over-LAN (SoL) (after enabling in the CIMC)

The default connection settings from the factory are:

- The serial port baud rate is 115200

- The RJ-45 console port located on the rear panel is disabled in the CIMC

- SoL is disabled in the CIMC

The following are additional notes about serial access:

- If you are using a Cisco Integrated Management Controller (CIMC) for your setup, setup the CIMC first, and then access the Cisco APIC through the CIMC KVM or continue to access the Cisco APIC locally through the rear panel USB/VGA port. If you choose the CIMC KVM access, you will have remote access available later which is required during operations.

- If you are using the RJ-45 console port, connect to CIMC using SSH and enable the SoL port using the following commands:

scope sol
set enabled yes
set baud-rate 115200
commit
exit

After enabling SoL, enter the command connect host to access the APIC console.

Note

When using SoL, physically disconnect the rear panel RJ-45 console port.

Initial Cisco APIC Setup

When the Cisco APIC is launched for the first time, the APIC console presents a series of initial setup options. For many options, you can press Enter to choose the default setting that is displayed in brackets. At any point in the setup dialog, you can restart the dialog from the beginning by pressing Ctrl-C.

Important Notes

- If the UNIX user ID is not explicitly specified in the response from the remote authentication server, then some Cisco APIC software releases assign a default ID of 23999 to all users. If the response from the remote authentication server fails to specify a UNIX ID, all users will share the same ID of 23999 and this can result in the users being granted higher or lower privileges than the configured privileges through the RBAC policies on the Cisco APIC.

- Cisco recommends that you assign unique UNIX user IDs in the range of 16000 to 23999 for the AV Pairs that are assigned to the users when in Bash shell (using SSH, Telnet, or Serial/KVM consoles). If a situation arises where the Cisco AV Pair does not provide a UNIX user ID, the user is assigned a user ID of 23999 or similar number from the range that also enables the user's home directories, files, and processes accessible to the remote users with a UNIX ID of 23999.

To ensure that your remote authentication server does not explicitly assign a UNIX ID in its cisco-av-pair response, open an SSH session to the Cisco APIC and log in as an administrator (using a remote user account). Once logged in, run the following commands (replace userid with the username that you logged in with):

- admin@apic1: remoteuser-userid> cd /mit/uni/userext/remoteuser-userid

- admin@apic1: remoteuser-userid> cat summary

- Cisco recommends against modifying any parameters using CIMC. If there are any issues, ensure that the default setting for CIMC management node is Dedicated mode and not Shared. If Dedicated mode is not used, it can prevent the discovery of fabric nodes.

- Do not upgrade software or firmware using the CIMC user interface, XML, or SSH interfaces unless the modified property and software or firmware version are supported with your specific Cisco APIC version.

- Set the NIC mode to Dedicated, when setting up the CIMC, in the CIMC Configuration Utility. After the CIMC is configured, in the CIMC GUI, verify that you have the following parameters set.

Parameters

Settings

LLDP

Disabled on the VIC

TPM Support

Enabled on the BIOS

TPM Enabled Status

Enabled

TPM Ownership

Owned

- If you log in to your Cisco APIC using https, and then attempt to log in to the same Cisco APIC using http in the same browser window without first logging out of the Cisco APIC in the https window, you might see the following error message:

Need a valid webtoken cookie (named APIC-Cookie) or a signed request with signature in the cookie.
If this occurs, resolve the issue using either of the following methods:

- Log out of the Cisco APIC in the https window, or

- Delete the cookies in the browser window

You should be able to successfully log into the Cisco APIC using http after resolving the issue with either of the methods above.

- During the initial setup, the system will prompt you to select IPv4, or IPv6, or dual stack configuration. Choosing dual stack will enable accessing the Cisco APIC and Cisco ACI fabric out-of-band management interfaces with either IPv4 or IPv6 addresses. While the examples in the table below use IPv4 addresses, you can use whatever IP address configuration options you chose to enable during the initial setup.

- A minimum subnet mask of /19 is recommended.

- Connecting the Cisco APIC to the Cisco ACI fabric requires a 10G interface on the ACI-mode leaf switch. You cannot connect the Cisco APIC directly to the Cisco Nexus 9332PQ, Cisco Nexus 93180LC, or Cisco Nexus 9336C-FX2 ACI-mode leaf switches unless you use a 40G to 10G converter (part number CVR-QSFP-SFP10G), in which case the port on the leaf switches will auto-negotiate to 10G without requiring any manual configuration.

- The fabric ID is set during the Cisco APIC setup and it cannot be changed unless you perform a clean reload of the fabric. To change the fabric ID, export the Cisco APIC configuration, change the sam.config file, and perform a clean reload of the Cisco APIC and leaf switches. Remove the "fvFabricExtConnP" setting from the exported configuration before importing the configuration into the Cisco APIC after the Cisco APIC comes up. All Cisco APICs in a cluster must have the same fabric ID.

- All logging is enabled by default.

- For login and cluster operations, non-default HTTPS port (default is 443) is not supported for layer 3 physical and layer 3 virtual APICs (on ESXi and AWS). Virtual APICs on ESXi/ AWS are supported from release 6.0(2).

- Bring up the APIC cluster using the GUI

- IPv6 management addresses on Cisco APIC

Bring up the APIC cluster using the GUI

This procedure provides details of the initial cluster set up and bootstrapping process. Enter the relevant details for each of the screens

The APIC Cluster Bringup GUI supports virtual and physical APIC platforms. The virtual APICs (deployed using ESXi or AWS), and physical APICs can be connected to the ACI fabric directly to the leaf switches or remotely attached through a layer 3 network. The GUI supports both the scenarios. A major advantage of using the APIC Cluster Bringup GUI is that, you do not need to enter the parameters for every APIC in a cluster. One APIC can relay the information to the other APICs of the cluster.

Beginning with Cisco APIC release 6.2(1), an extra layer of security has been added for clusters with virtual APICs. After you bring up a new virtual APIC to a cluster, you will be prompted to change the password. See Step 12, below, for details.

Note

: When deploying the virtual APIC (VMware or Nutanix), you should enter a temporary password. This is crucial especially when you are adding the virtual APIC to an existing cluster. If the cluster password is used during the VM deployment, you will need to change the existing cluster password after the APIC has joined the cluster.

Alternatively, you can perform the initial setup and cluster bringup using the REST APIs. See the Getting Started section of the APIC REST API Configuration Procedures guide.

Before you begin

- For virtual APIC on ESXi, ensure to complete the deployment of the APIC VM using the OVF template on the VMware vCenter GUI. For a three-node cluster, configure three VMs with the management IP address, gateway, and admin passwords. The number of VMs is dependent on the size of the APIC cluster.

- For virtual APIC on AWS, ensure to complete the deployment of the APIC VM using the cloud formation template (CFT) on the AWS GUI. AWS allocates IP addresses dynamically from the out-of-band (OOB)/infra/inband subnets accordingly, to correspond with the network adapters of the virtual APIC's EC2 instance.

- For virtual APICs (deployed using AWS/ ESXi), ensure that the admin password(s) are the same for all the APICs in a cluster.

- For the physical APIC cluster, configure the OOB address for APIC 1. Ensure that the CIMC addresses of APICs 2 to N (where N is the cluster size) are reachable via the OOB address of APIC 1.

- Connectivity between out-of-band and the CIMC is mandatory.

Limitations:

- No support for IPv6 addresses on virtual APICs deployed using AWS.

- For login and cluster operations, non-default HTTPS port (default is 443) is not supported for remotely-attached APICs (physical and virtual).

Procedure

Step 1

Log in to the APIC 1 using https://APIC1-IP.

1. For virtual APICs:

If you have completed the deployment of virtual APICs using ESXi (OVF template) or remote AWS (CFT), then you see output on the VM console similar to the following example:

System pre-configured successfully.
Use: https://172.31.1.2 to complete the bootstrapping.

The IP address to access the bootstrapping GUI (APIC Cluster Bringup) is explicitly indicated, as shown in the example. You can proceed to step 2.

After deploying APIC on AWS, keep the OOB management IP address handy to access the Cluster Bringup GUI. You can get the OOB management IP address from the Stacks Outputs tab on the AWS GUI.

2. For physical APICs:

Log in to the APIC 1 KVM console using the CIMC; you will see a screen as shown below:

APIC Version: 6.0(2a)
Welcome to Cisco APIC Setup Utility
Press Enter Or Input JSON string to bootstrap your APIC node.

If you see only a black screen on the KVM, connect to the CIMC using SSH and use serial over LAN (SoL) ("connect host") to connect to the console.

On APIC 1, press Enter and provide the requested information. The IP address to access the bootstrapping GUI (APIC Cluster Bringup) is explicitly indicated.

admin user configuration ...
Enter the password for admin [None]:
Reenter the password for admin [None]:
Out-of-band management configuration ...
Enter the IP Address [192.168.10.1/24]: 172.20.7.79/23
Enter the IP Address of default gateway [192.168.10.254]: 172.20.6.1
Would you like to edit the configuration? (y/n) [n]:
System pre-configured successfully.
Use: https://172.20.7.79 to complete the bootstrapping

The IP addresses displayed above are examples. The IP addresses will vary based on your deployment.

Step 2

Using the OOB address, log in to the APIC Cluster Bringup GUI.

Step 3

In the Select Workflow screen, choose New cluster and click Next.

Note

The workflow selection is only from Release 6.1(3) and later.

The GUI screen has four parts. Enter the details in the following screens:

- Connection Type

- Cluster Details

- Controller Registration

- Summary

Each of the above screens are discussed in detail in the subsequent steps. The screens are marked as steps with sequential numbers: 1, 2, 3, and 4; after you have entered and saved the required details in each of these screens, the number is replaced with a tick-mark.

Step 4

The first step is entering the Connection Type information. In the Connection Type screen, choose the type of connection between the APIC and the fabric.

The options are:

- Directly connected to leaf switches (APIC fabric)

- Remotely attached through a Layer 3 network

If it is virtual APIC using AWS, the system detects that the APIC is remotely-attached through a Layer 3 network and proceeds directly to the Cluster Details screen.

Click Next.

Step 5

The second step is entering the Cluster Details. Enter the fabric-level details in the Cluster Details screen.

- Fabric Name: Enter a name for the fabric.

- Cluster Size: The default cluster size displayed is "3", which is the recommended minimum cluster size. You can modify this value, based on your cluster size. The supported values are 1, 3, 4, 5, 6, 7, 8, and 9.

- GiPo Pool: Enter the IP address used for fabric multicast. The default address is 225.0.0.0/15. The range is from 225.0.0.0/15 to 231.254.0.0/15. The prefixlen must be 15 (128k IP addresses).

You cannot change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.

- Pod ID: (applicable only for directly connected APICs (virtual and physical)) the pod ID is displayed. If this is your first APIC, "1" is auto-populated. Subsequent APICs of the cluster can be associated with any pod number.

For a remotely-attached APICs, pod is 0.

- TEP Pool: (applicable only for directly connected APICs (ESXi virtual APIC and physical APIC)) enter the subnet of addresses used for internal fabric communication. The size of the subnet used will impact the scale of your pod.

You can not change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.

- Infrastructure VLAN: Enter the VLAN ID for fabric connectivity (infra VLAN). This VLAN ID should be allocated solely to APIC, and not used by any other legacy device(s) in your network. Default value is 3914. The range is from 0 to 4093.

You can not change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.

- Enable IPv6 on APICs (not applicable for virtual APIC on AWS): Put a check in this box if you want to enable IPv6 addresses for out-of-band management.

Click Next.

Step 6

The third step is entering the Controller Registration details. Click Add Controller to add the first APIC (of the cluster). Enter the following details:

- Controller Type: The bootstrapping procedure auto-detects the deployment for which the configuration is being carried out. Based on that, either Virtual or Physical is chosen. The options displayed for the virtual and physical controller types are discussed in substeps (a) and (b), respectively. Follow either of these substeps based on the controller type.

1. When the Controller Type is Virtual:

- Virtual Instance: The management IP used to access the APIC cluster bringup GUI. Only for the first APIC, this IP address is auto-populated. For the nodes that you subsequently add to the cluster, you will need to enter the management IP address and click Validate.

The management IP addresses are defined during the deployment of the VMs using ESXi/AWS. As mentioned in the prerequisites, keep all the required IP addresses handy while bringing up the cluster.

- General pane

- Name: User-defined name for the controller.

- Controller ID: The ID is auto-populated. If this is the first APIC of the cluster, the ID is "1". If you are adding the second controller of the cluster, "2" is auto-populated (and so on).

- Pod ID: (Applicable only for directly connected virtual APIC on ESXi) The pod ID is auto-populated for APIC 1 of the cluster. For subsequent controllers of the cluster, enter a value. The range is from 1 to 128.

- Serial Number: The serial number of the virtual machine is auto-populated.

- Out of Band Network pane

- IPv4 Address: The IP address is displayed (as defined during the deployment).

- IPv4 Gateway: The IP address is displayed (as defined during the deployment).

If you have enabled IPv6 addresses for OOB management earlier (step 5), enter the IPv6 address and gateway.

- Infra L3 Network pane (this pane is displayed only if the Connection Type that you chose earlier is Remotely attached through an L3 network.

- IPv4 Address: Enter the infra network address.

- IPv4 Gateway: Enter the IP address of the gateway.

- VLAN: (Applicable only for remotely attached virtual APIC- ESXi) Enter the interface VLAN ID to be used.

The Infra L3 Network pane does not display when you deploy the virtual APIC using AWS.

After you have entered and saved the first APIC details, click Add Controller on the Controller Registration screen to add another APIC to the cluster.

2. When the Controller Type is Physical:

- CIMC Details pane

- IP Address: The CIMC IP address. Only for the first APIC, this IP address is auto-populated. When you add more controllers to the cluster, you need to enter the CIMC IP addresses.

- Username: The username to access the CIMC. The username is auto-populated (for the first controller and subsequent controllers).

- Password: Enter the password to access CIMC. For the first controller, the password is auto-populated. For the subsequent controllers, enter the password.

- Click Validate. Validation success is displayed on successful authentication.

If the CIMC is unreachable from the APIC out of band management IP address due to the CIMC NIC mode settings, change the NIC mode or enter JSON strings to perform the bootstrap.

- General pane

- Name: Enter a name for the controller.

- Controller ID: If it is the first controller of the cluster, "1" is auto-populated. If it is the second controller, "2" is auto-populated, and so on (increasing order).

- Pod ID: (applicable only for a directly-connected APIC) the pod ID is auto-populated for APIC 1 of the cluster. For subsequent controllers of the cluster, enter a value. The range is from 1 to 128.

- Serial Number: The serial number is auto-populated (for APICs 1 to N, where N is the cluster size) after CIMC validation.

APIC 1 verifies the reachability of the CIMC IP addreses and also captures the serial number of the new APICs.

- Out of Band Network pane

- IPv4 Address: For APIC 1, the address is auto-populated. For subsequent APICs, enter the IP address (as defined during the deployment).

- IPv4 Gateway: For APIC 1, the gateway address is auto-populated. For subsequent APICs, enter the IP address (as defined during the deployment).

If you have enabled IPv6 addresses for OOB management earlier (step 5), enter the IPv6 address and gateway.

- Infra L3 Network pane (this pane is displayed only if the Connection Type that you chose earlier is remotely attached through a Layer 3 network).

- IPv4 Address: Enter the infra network IP address.

- IPv4 Gateway: Enter the infra network IP address of the gateway.

- VLAN: Enter a VLAN ID.

On the Controller Registration screen, after you have entered and saved the first APIC details, click Add Controller to add another APIC to the cluster.

(Optional, applicable only for virtual APICs) On the Controller Registration screen, put a check in the Import existing security certificates box to import existing security certificates for fabric recovery in virtual APICs. After putting a check in the box, enter the required details in the following fields:

- The Remote Server IP Address which contains the configuration file.

- The Remote Path which contains the configuration file.

- The configuration File Name.

- The AES Encryption Passphrase which was earlier used while backing up the configuration. The backup configuration file is linked to this key (passphrase).

- Choose the Protocol. The choices are:

- FTP

- SFTP

- SCP

- Remote Port

- (applicable only for SFTP and SCP Protocols) Choose the Authentication Type. The choices are:

- Use Password

- Use SSH Private Key Files

- The Username to access the remote server.

- The Password to authenticate access to the remote server.

- (applicable only for Use SSH Private Key Files Authentication Type) Enter the SSH Key Contents here.

- (applicable only for Use SSH Private Key Files Authentication Type) Specify the SSH Key Passphrase used for encrypting the private key.

For details about the Import/Export procedure, see the Cisco ACI Configuration Files: Import and Export document.

The Import existing security certificates is applicable only for virtual APICs (deployed using AWS/ ESXi). Physical APICs have in-built certificates. However, in case of virtual APICs, when you are restoring using backup configuration to recover the fabric, the existing security certificates can be re-used.

Step 7

Click Next.

The Next button is disabled until all the controllers for a cluster are added. This is defined by the value you have entered for Cluster Size in the Cluster Details screen.

You can use the Back button to navigate to an earlier screen. After adding an APIC, click Edit Details to edit the information for an APIC. Except the first APIC, you can delete the other controllers, if required, by clicking the delete icon.

Step 8

In the Summary screen, review the updates, and click Deploy.

The Cluster Status page is displayed, which shows the current status of the cluster formation. Wait for a few minutes after which you will be automatically redirected to the standard Cisco APIC GUI.

Step 9

(for virtual APICs on VMware and Nutanix only) The Change Password window is displayed. Enter the required details and click Submit.

IPv6 management addresses on Cisco APIC

IPv6 management addresses can be provisioned on the Cisco Application Policy Infrastructure Controller (APIC) at setup time or through a policy once the Cisco APIC is operational. Pure IPv4, pure IPv6, or dual stack (that is, both IPv6 and IPv4 addresses) are supported.

For the detailed cluster bring up procedure, see Bring up the APIC cluster using the GUI.

Access the APIC GUI

Use this procedure to access the APIC GUI.

APIC GUI is the interface which enables you to configure the APIC. The Cisco APIC can be configured using the GUI or using REST API or CLI.

Procedure

Step 1

Open one of the supported browsers:

- Chrome version 59 (at minimum)

- Firefox version 54 (at minimum)

- Internet Explorer version 11 (at minimum)

- Safari version 10 (at minimum)

Note

A known issue exists with the Safari browser and unsigned certificates. Read the information presented here before accepting an unsigned certificate for use with WebSockets. When you access the HTTPS site, the following message appears:

“Safari can’t verify the identity of the website APIC. The certificate for this website is invalid. You might be connecting to a website that is pretending to be an APIC, which could put your confidential information at risk. Would you like to connect to the website anyway?”

To ensure that WebSockets can connect, you must do the following:

Click Show Certificate.

Choose Always Trust in the three drop-down lists that appear.

If you do not follow these steps, WebSockets will not be able to connect.

Step 2

Enter the URL: https://mgmt_ip-address

Use the out-of-band management IP address that you configured during the initial setup. For example, https://192.168.10.1.

Note

Only https is enabled by default. By default, http and http-to-https redirection are disabled.

Note

If you see the following error message when logging into your Cisco APIC:

Need a valid webtoken cookie (named APIC-Cookie) or a signed request with signature in the cookie.

This is due to a known issue that occurs when you are logging into a Cisco APIC using both https and http. See the "Important Notes" section for more information on this issue and the workaround.

Step 3

When the login screen appears, enter the administrator name and password that you configured during the initial setup.

Step 4

In the Domain field, from the drop-down list, choose the appropriate domain that is defined.

If multiple login domains are defined, the Domain field is displayed. If the user does not choose a domain, the DefaultAuth login domain is used for authentication by default. This may result in login failure if the username is not in the DefaultAuth login domain.

Access REST API

The Cisco APIC can be configured using the REST API. Other options to configure the APIC are the GUI and CLI.

By using a script or a browser-based REST client, you can send an API POST or GET message of the form:

https://apic-ip-address/api/api-message-url

Use the out-of-band management IP address that you configured during the initial setup. You must send an authentication message to initiate an API session. Use the administrator login name and password that you configured during the initial setup.

Note

Only https is enabled by default. By default, http and http-to-https redirection are disabled.

Access the NX-OS style CLI

You can access the APIC NX-OS style CLI either directly from a terminal or through the APIC GUI.

Guidelines for the APIC NX-OS style CLI

- The CLI is supported only for users with administrative login privileges.

- The APIC NX-OS style CLI uses similar syntax and other conventions to the Cisco NX-OS CLI, but the APIC operating system is not a version of Cisco NX-OS software. Do not assume that a Cisco NX-OS CLI command works with or has the same function on the APIC CLI.

- If FIPS is enabled in the Cisco ACI setups, then SHA256 support is mandatory on the SSH Client. Additionally, to have the SHA256 support, the openssh-client must be running version 6.6.1 or higher.

- The object model CLI is available by typing the bash command at the initial CLI prompt.

- Access the NX-OS style CLI from a terminal

- Access the NX-OS style CLI from the GUI

Access the NX-OS style CLI from a terminal

Use this procedure to access the NX-OS style CLI from a terminal.

Procedure

Step 1

From a secure shell (SSH) client, open an SSH connection to APIC at username@ip-address .

Use the administrator login name and the out-of-band management IP address that you configured during the initial setup. For example, admin@192.168.10.1.

Step 2

When prompted, enter the administrator password.

What to do next

When you enter the NX-OS style CLI, the initial command level is the EXEC level. You can stay in EXEC mode or you can type configure to enter global configuration mode. In any mode, type ? to see the available commands.

For information about using the NX-OS style CLI commands, see the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide and the Cisco APIC NX-OS Style CLI Command Reference.

Access the NX-OS style CLI from the GUI

Use this procedure to access the NX-OS style CLI from the APIC GUI.

Procedure

Step 1

From the menu bar, choose System > Controllers.

Step 2

In the navigation pane, click Controllers.

Step 3

Right-click the desired APIC and choose Launch SSH.

Step 4

Follow the displayed instructions to open an SSH session to the selected controller.

What to do next

When you enter the NX-OS style CLI, the initial command level is the EXEC level. You can stay in EXEC mode or you can type configure to enter global configuration mode. In any mode, type ? to see the available commands.

For information about using the NX-OS style CLI commands, see the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide and the Cisco APIC NX-OS Style CLI Command Reference.

Access the object model CLI

Use this procedure to access the object model CLI. The object model CLI is available by typing the bash command at the initial CLI prompt.

Procedure

Step 1

From a secure shell (SSH) client, open an SSH connection to username@ip-address .

Use the administrator login name and the out-of-band management IP address that you configured during the initial setup. For example, ssh admin@192.168.10.1.

Step 2

When prompted, enter the administrator password that you configured during the initial setup.

You are now in the NX-OS style CLI for APIC.

Step 3

Type bash to enter the object model CLI.

Step 4

To return to the NX-OS style CLI, type exit .

This example shows how to enter the object model CLI and how to return to the NX-OS style CLI:

$ ssh admin@192.168.10.1
Application Policy Infrastructure Controller
admin@192.168.10.1's password: cisco123
apic#             <---- NX-OS style CLI prompt
apic# bash
admin@apic1:~>    <---- object model CLI prompt
admin@apic1:~> exit
apic#

What to do next

Every user must use the shared directory called /home. This directory gives permissions for a user to create directories and files; files created within /home inherit the default umask permissions and are accessible by the user and by root. We recommend that users create a /home/userid directory to store files, such as /home/jsmith, when logging in for the first time.

For detailed information about configuring the APIC CLI, see the Cisco APIC Object Model Command Line Interface User Guide.

Chapter 3. APIC GUI Overview

This chapter contains the following sections:

- The APIC GUI

- Menu bar and submenu bar

- Navigation pane

- Work pane

- Personalize the APIC GUI

- Deployment warning and policy usage information

- Graphical configuration of ports

- View an API interchange

- APIC GUI icons

- Next generation APIC GUI

- Preview the new APIC GUI

- Enhancements and improvements to the APIC GUI

The APIC GUI

The APIC GUI is a browser-based graphical interface for configuring and monitoring the ACI fabric. The GUI is organized to provide hierarchical navigation to all components, logical and physical, of the overall system. The primary control regions of the GUI are shown in the following figure.

The functions of these regions are described in the following links:

1. Menu bar tools—See Menu bar and submenu bar

2. Menu bar—See Menu bar and submenu bar

3. Submenu bar—See Menu bar and submenu bar

4. Work pane—See Work pane

5. Navigation pane—See Navigation pane

Below the Navigation pane, the Last Login is displayed, showing the date and time of the last instance of the current user's login.

As you operate the GUI to make configuration changes and retrieve information, the GUI communicates with the underlying operating system by exchanging REST API messages. You can observe these API messages using the API Inspector tool described in View an API interchange.

Menu bar and submenu bar

The menu bar provides access to the main configuration tabs, along with access to tools such as search, notifications, and preferences. Immediately below the menu bar is the submenu bar, which presents specific configuration areas for each selected menu bar tab. The submenu bar tabs are different for each menu bar tab and might also differ depending upon your specific configuration or privilege level.

Tip

In the APIC GUI configuration instructions, you will see notation such as Fabric > Fabric Policies. In this example, you are asked to click the Fabric tab in the menu bar followed by the Fabric Policies tab in the submenu bar.

The menu bar is displayed across the top of the APIC GUI. At the far right side of the menu bar are the menu bar tools (these are described in detail in the subsequent sections).

- username: the name of the currently logged in local user.

- : the Search icon.

- : the Multi-site manager icon to launch the multi-site manager.

- : the Feedback icon to send comments to Cisco.

- : the Alerts icon to view the list of active alerts.

- : the Tools icon to access the system tools.

- : the Help icon.

- : the Manage my profile icon. The logged-in use can configure settings and preferences.

- Menu bar tabs

- Menu bar tools

- Capacity dashboard

Menu bar tabs

The menu bar is displayed at the top of the APIC GUI. The menu bar serves as the vital navigation area, allowing you to access key configuration tabs. It also includes convenient tools such as search, notifications, and preferences, enabling you to manage settings. The menu bar tabs are discussed in detail in this section.

System tab

Use the System tab to collect and display a summary of the overall system health, its history, and a table of system-level faults.

In addition, the System tab provides the following functions:

- You can configure global system policies in the System Settings submenu.

- You can view your licensing status in the Smart Licensing submenu.

- You can view user sessions in the Active Sessions submenu.

Tenants tab

Use the Tenants tab in the menu bar to perform tenant management. The submenu bar provides a list of all tenants, an Add Tenant link, and links to three built-in tenants plus up to two of the most recently used tenants.

The built-in tenants are:

- The common tenant is preconfigured for defining policies that provide common behavior for all the tenants in the fabric. A policy defined in the common tenant is usable by any tenant.

- The infra tenant is preconfigured for configuration related to the fabric infrastructure

- The mgmt tenant is preconfigured for inband and out-of-band connectivity configurations of hosts and fabric nodes (leafs, spines, and controllers).

Fabric tab

The Fabric tab contains the following tabs in the submenu bar:

- Inventory tab: Displays the individual components of the fabric.

- Fabric Policies tab: Displays the monitoring and troubleshooting policies and fabric protocol settings or fabric maximum transmission unit (MTU) settings.

- Access Policies tab: Displays the access policies that apply to the edge ports of the system. These ports are on the leaf switches that communicate externally.

Virtual networking tab

Use the Virtual Networking tab to view and configure the inventory of the various virtual machine (VM) managers. You can configure and create various management domains under which connections to individual management systems (such as VMware vCenters or Nutanix hypervisors) can be configured.

Use the Inventory tab in the submenu bar to view the hypervisors and VMs that are managed by these VM management systems (also referred to as controllers in API).

Admin tab

Use the Admin tab to perform administrative functions such as authentication, authorization, and accounting functions, scheduling policies, retaining and purging records, upgrading firmware, and controlling features such as syslog, Call Home, and SNMP.

Operations tab

The Operations tab provides the following built-in tools for planning and monitoring fabric resources.

- Visibility & Troubleshooting: Shows the location of specified end points in the fabric and displays the traffic path, including any L4-L7 devices.

- Capacity Dashboard: Displays the available capacity of configurable resources such as end points, bridge domains, tenants, and contexts. For more details, see Capacity Dashboard.

- EP Tracker: Enables you to view virtual and bare metal endpoint connections and disconnections to leaf switches and FEXes.

- Visualization: Provides visualization of traffic maps.

Apps tab

The Apps tab displays all the applications installed or uploaded to APIC. The tab allows an APIC administrator to upload, enable, upgrade, install, or uninstall a packaged application in APIC.

Integrations tab

Use the Integrations tab to view all third-party integrations.

Menu bar tools

At the far right side of the menu bar are the menu bar tools. These are displayed as icons and the detailed descriptions for each of them are discussed in this section.

Search

The Search icon displays the search field to locate objects in the interface by name or other distinctive attributes.

The search function allows the use of wildcards (*).

Launch multi-site manager

The Multi-Site Manager icon launches the multi-site manager. The multi-site manager can launch multiple site APICs.

Feedback

The Feedback icon enables you to send comments directly to Cisco.

Alerts

The Alert icon indicates the alerts on the system. To view the alerts, click the icon. When system alerts are available, a numeric badge appears on the alert icon indicating the number of active alerts. When critical system notifications are available, the alert icon will blink red.

To disable blinking of the alert icon, remove all critical alerts from the alert list. A disabled Close button on a critical alert indicates that you must first resolve the underlying issue before the alert can be cleared.

Tools

To access the system tools, click the Tools icon and select an item from the drop-down list.

The following selections are available:

- ACI Fabric Setup: Guides initial APIC infrastructure setup.

- Show API Inspector: Provides visibility into internal API messages between the GUI and APIC operating system to execute tasks. For more information, see Viewing an API interchange section in this chapter.

- Start Remote Logging: Forwards logging information to a remote URL.

- Object Store Browser: Opens the Managed Object Browser, or Visore, which is a utility built into APIC that provides a graphical view of the managed objects (MOs) using a browser.

- Show Debug Info: Open a status bar at the bottom of the GUI to display information such as current managed object (MO) and system time. When the status bar is open, this selection changes to Hide Debug Info.

- Config Sync Issues: Opens the Configuration Objects Pending Resolution panel. This panel shows if there are any transactions involving user-configurable objects that have yet to take effect in APIC. You can use information in the panel to help with debugging.

Note

Global system settings are configured in System > System Settings.

Help

To access the help tools, click the Help icon and select an item from the drop-down list.

The following selections are available:

- Help: Displays links to API documentation and to the APIC

- What's new: Displays splash screen showing recent features.

- About: Displays the APIC version.

Manage my profile

Allows personalization and account management for the logged-in user.

The following selections are available:

- Bookmarks: Displays links to menus bookmarked by the user.

Menus that display the Favorites icon () can be bookmarked by clicking the icon.

- Change My Password: Change the password of the currently logged in local user.

- Change My SSH Keys: Change the user's public SSH key used for certificate-based login.

- Change My X509 Certificate: Change the user's X.509-format certificate for login.

- View My Permissions: Display the user's role-based read and write privileges for domains and accessible objects.

- Settings:Change general GUI settings.

- Remember Tree Selection: Enables the GUI to keep the navigation tree expanded when returning to a window. For example, if you enable this property and expand the navigation tree in the Tenants tab, click on the Fabric tab, then return to the Tenants tab, the tree will remain expanded.

- Preserve Tree Divider Position: Enables the GUI to keep the position of the tree divider after dragging the tree divider to the desired location.

- Disable Notification on Success: Suppresses the success dialog box notification.

- Disable Deployment Warning at Login: Disables the Deployment Warning dialog box when logging in. See the Deployment warning and policy usage information section later in this chapter.

- Default Page Size for Tables: Sets the GUI table size.

- Show All UI Sections: Displays hidden UI configuration options.

- Show What's New at Login: Displays splash screen at login, showing recent features.

- Enable Single-Browser Session (SBS): Allows logging in to the APIC GUI and then opening additional browser tabs or windows to the same APIC without being required to log in from each new tab or window. See Single-browser session management section later in this chapter.

- Change Deployment Settings: Enable and set the scope of the deployment notification. See the Deployment warning and policy usage information section later in this chapter.

- Logout: Exit the APIC configuration GUI.

Capacity dashboard

The capacity dashboard displays the available capacity of configurable resources such as endpoints, bridge domains, tenants, and contexts.

The dashboard contains the following tabs:

- Fabric Capacity: Displays the capacity of the managed objects within the fabric. Each tile provides the current and maximum capacity of each object, as well as the percentage of the maximum capacity that is used. You can hover your cursor over some of the tiles to see more information.

- Leaf Capacity: Displays the capacity of the managed objects for each leaf switch that the APIC manages. Beginning with APIC release 6.1(4), the name of the tab has been changed to Leaf Switch Capacity.

- For all of the objects, the GUI displays the current resource usage and maximum resource capacity, as well as the percentage of the maximum resource capacity that is used.

- The data for some of the objects is split into subcategories, such as local and remote for ESG MAC addresses.

- The data for MAC, IPv4, and IPv6 addresses shows the total number of local and remote addresses.

- (not applicable for APIC release 6.1(4) and after) The data for /32 routes and /128 routes provides the following information:

- UC: The total IPv4 /32 or /128 unicast routes. This value persists through each interval without resetting to zero.

- EP: The total IPv4 /32 or /128 endpoints. This value persists through each interval without resetting to zero.

- MCast: The total IPv4 /32 or /128 multicast routes. This value persists through each interval without resetting to zero.

- You can click the Configure Profile button in the Switch column to configure the forward scale profile for that switch. Beginning with APIC release 6.1(4), this button has been renamed to Change Forward Scale Profile.

You can change the forward scale profile by clicking the three dots at the end of the row for each entry.

- (not applicable for APIC release 6.1(4) and after) You can click any other part of a row to see detailed capacity usage information for that switch. For resources with the Absolute entry, this is the current resource usage. In the case of /32 and /128 routes, Absolute is the total of unicast routes, endpoints, and multicast routes being used. Percentage is the percentage of the maximum resource capacity that being used.

Beginning with APIC release 6.1(4), there is a considerable performance improvement to the Capacity Dashboard tab with faster display time of the various elements displayed here. The other significant enhancements include:

- Moving away from the tile layout. Prior to release 6.1(4), all the entries were available as tiles, now a table has been introduced for the Fabric Capacity and Leaf Switch Capacity sub-tabs. The metrics which were earlier displayed as tiles are available in the table. Some of the earlier tile entries, for the Fabric Capacity sub-tab, such as endpoint details, are now available only when you click Details in the MAC + IP Endpoints row.

- Status indication for each of the resources is dependent on the Resource Usage. Hover over the Resource Usage column to check the status of the usage of resources. The available status indicators are:

- Conformant (0 to 75% of resource usage)

- Approaching limits (76 to 90% of resource usage)

- Approaching high limits (91 to 100% of resource usage)

- Violated limits (above 100% of resource usage)

In the Leaf Switch Capacity tab, click the Resource Usage number to get more details about the switch parameters. You can also use the filtering option to filter using Metric, Status or Resource Usage.

- New entries added to the Fabric Capacity and Leaf Switch Capacity tabs are:

- Leaf Switch Capacity tab: Opflex agents, Opflex layer 2 endpoints, IPSec (for OSPFv3), Redirect Destination Monitoring.

- Fabric Capacity tab: Redirect Destination Monitoring (as part of the switch details).

Navigation pane

The Navigation pane in the APIC GUI enables you to access and manage hierarchical objects associated with submenu categories. It appears as a pane on the left side of the APIC GUI below the submenu bar and organizes both logical and physical objects relevant to each category.

Key attributes and functions

- The Navigation pane displays submenu categories as hierarchical trees of related objects.

- Objects may represent ports, policies, or groupings.

- Selecting an object in the Navigation pane displays its details in the Work pane.

- Right-clicking an object in the Navigation pane provides access to a context menu with the following actions:

- Delete: Deletes the object.

- Create <type of object>: Creates a new object.

- Save as...: Downloads the object and its properties in JSON or XML format to a local file.

- Post...: Exports the object and its properties to an existing local file.

- Share: Displays the URL of the object. You can copy the URL and send it to others.

- Open In Object Store Browser: Opens the object in Visore, a built-in utility that displays an object and its properties. This information may be useful in troubleshooting or for developing API tools.

- Clone: Creates a copy of the object. This action is useful for deriving a new contract or policy based on an existing contract or policy.

Note

When a container in the Navigation pane (such as Application Profiles under a Tenant) contains more than 40 profiles, direct expansion of profiles in the Navigation pane becomes unavailable. To expand a profile in this context, select the desired profile within the Work pane.

Work pane

The Work pane, which is on the right side of the APIC GUI, displays details about the component that you selected in the Navigation pane.

Key attributes and functions

- The Work pane includes a content area that displays tabs. These tabs provide information related to the component selected in the Navigation pane. The tabs displayed in the content area depend on the selected component.

- For some components in the Work pane, a link to conceptual information related to the component is indicated by a list icon in the upper right corner.

- You can bookmark almost any page, which enables you to go back to that page easily by choosing the bookmark from your list of bookmarks.

Bookmarked links are accessible from the User Profile and Preferences icon in the Menu bar.

- You can mark a tab as the "favorite" on a page. When you navigate to that page, that tab is displayed by default. This feature is available only for tabs in the Work pane. You cannot mark a menu bar tab as a favorite.

- Common pages in the work pane

Common pages in the work pane

In addition to displaying specific task menus, the Work pane also displays several types of special-purpose menus described in this section.

Quick Start Pages

Many APIC menu and submenu tabs open an initial Quick Start page, which summarizes the purpose of the tab, provides links to step-by-step instructions and videos for commonly-used procedures, and provides shortcut links to commonly-used subsections within the tab. An overall Quick Start page at System > QuickStart assists you in performing common and basic procedures, providing step-by-step instructions, available concept information, and links to main functional areas in the GUI.

Dashboard Pages

Dashboard pages provide at-a-glance summaries of the status of the ACI system and major system components, including health score trends, components with below-threshold health scores, and fault counts. You can configure health score thresholds to determine when components will appear in the dashboard. The system dashboard page at System > Dashboard summarizes the health of the overall ACI system, while switch dashboard pages at Fabric > Inventory > Pod n > component > Dashboard summarize the health and faults of each spine and leaf switch.

Summary Pages

Many top-level folders in the Navigation pane display tile-based Summary pages in the Work pane that link to subfolders. Some Summary pages, such as those in Fabric > Inventory > Pod n, contain tiles summarizing major components along with brief health and fault information for each component. Other Summary pages, such as those in Fabric > Fabric Policies > Policies, contain tiles that describe the configuration areas served by the contained folders.

Personalize the APIC GUI

The process includes pointers to personalize the Cisco APIC GUI.

Summary

Personalizing APIC GUIs enhances the usability and identification of APICs. The key components involved in the process are:

- Administrator: configures custom names and login banners.

- APIC GUI: displays the customized information (the name and banner).

Workflow

You can personalize the APIC GUI by

1. setting a name to each of the APICs in a cluster.

2. adding a customized banner at login (applicable for both CLI and GUI).

Benefits of personalization:

- Improved identification and usability: adding a custom name to the APIC GUI helps you identify which APIC you are managing in a cluster. This enhances operational clarity and reduces confusion when multiple APICs are in use.

- Custom login experience: you can add login banners to both the CLI and GUI interfaces. The CLI banner displays a text message before the password prompt, while the GUI banner can display a custom HTML page before user authentication.

- Assign a name to the APIC

- Add a login banner

Assign a name to the APIC

Set a custom name for the APIC interface to clearly indicate the controller you are viewing.

In an ACI cluster with three (the recommeneded number of controllers) APICs, naming each controller helps you to quickly identify the APIC you are logged in to.

Procedure

Step 1

On the APIC menu bar, choose System > System Settings.

Step 2

In the Navigation pane, click APIC Identification Preferences.

Step 3

In the work pane, type the desired APIC name in the GUI Alias box.

Step 4

Click Submit.

The custom APIC name appears at the top left of the interface in parentheses, allowing you to easily identify the controller.

Add a login banner

Display a custom login banner for users accessing the APIC system via the CLI or GUI interfaces.

Define banners that will be displayed when you log in to the CLI or GUI.

- The CLI banner is a simple text string displayed at the terminal before the password prompt. You can define a banner for the APIC CLI and a separate banner for the switch CLI.

- The GUI banner displays at the APIC URL before user login authentication. The GUI banner is defined as a URL of a site hosting the desired HTML.

Before you begin

- Ensure you have access privileges to modify APIC system identification preferences.

- (for GUI banner only) Confirm that the banner URL site allows embedding in an iFrame.

Note

The URL site owner must allow the site to be placed in an iFrame to display the informational banner. If the owner of the site sets the x-frame-option to deny or sameorigin, the site the URL points to will not appear.

Procedure

Step 1

On the APIC menu bar, choose System > System Settings.

Step 2

In the Navigation pane, click APIC Identification Preferences.

Step 3

In the work pane, complete the following fields:

1. To configure an APIC CLI banner, type the banner text into the Controller CLI Banner textbox.

2. To configure a switch CLI banner, type the banner text into the Switch CLI Banner textbox.

3. To configure an APIC GUI banner, type the URL of a site hosting the desired HTML into the GUI Banner (URL) textbox.

Step 4

Click Submit.

Deployment warning and policy usage information

Deployment Warning Settings enables you to automatically display policy usage information whenever you modify or delete policies that might affect other resources or policies. The policy usage information allows you to identify the resources and policies that are being used by the policy that you are currently modifying or deleting. Tables display the nodes where the given policy is used and other policies that use this policy. By default, usage information is displayed within a dialog box whenever you attempt to modify a policy. You can click the Show Usage button at the bottom of the screen to view the same information.

The Deployment Warning Settings dialog box allows you to enable and alter the scope of deployment notification that displays policy usage information. You can access this dialog box by selecting Change Deployment Settings in the menu bar tool User Settings and Preferences drop-down list or through a button on the Policy Usage Information dialog box.

When the Policy tab is selected in the upper right corner of the Deployment Warning Settings dialog box, you can configure the following policy options:

- (Global) Show Deployment Warning on Delete/Modify—Enable the Deployment Warning notification for every policy deletion or modification across the APIC.

- (Local) Show Deployment Warning on Delete/Modify: Set the rule for the Deployment Warning notification for specific policy configuration.

- Use Global Settings: Use the setting selected for (Global) Show Deployment Warning on Delete/Modify.

- Yes: Display the Deployment Warning notification before submitting configuration modifications on any policy change. Valid for this browser session only.

- No: Do not display the Deployment Warning notification before submitting configuration modifications on any policy change. Valid for this browser session only.

When the History tab is selected in the upper right corner of the Deployment Warning Settings dialog box, you can view tables of Events and Audit Log entries for previous deployment warnings.

Graphical configuration of ports

The APIC GUI provides a graphical method for configuring ports, port channels, and virtual port channels on the leaf switches in the fabric, configure ports for dynamic breakout, and link interfaces to FEX switches.

This configuration capability is present at these GUI locations:

- Fabric > Inventory > Topology

- Fabric > Inventory > Pod

- Fabric > Inventory > Pod > Leaf

- Fabric > Inventory > Pod > Spine

In the Work pane's Interface tab, click on the + button (at the top left), select one or more switches to configure, and click Add Selected. To select multiple switches, use Ctrl+Click or Shift+Click.

The switches are graphically displayed with their ports and links. If you have configured a breakout port, a block containing the sub ports is displayed below the leaf diagram.

Note

If you accessed the Interface tab from a leaf switch, the leaf switch is automatically added.

Select the interfaces to configure. When interfaces are selected, the available configuration buttons appear. Depending on the number of selected interfaces and where they are located, you can then click one of the these buttons:

- L2—Layer 2. Visible when you click one or more leaf interfaces on the switch diagrams.

- PC—Port Channel. Visible when you click one or more leaf interfaces on the switch diagrams.

- VPC—Virtual Port Channel. Visible when you click at least one interface on two switch diagrams.

- FEX—Fabric Extender. Visible when you click one or more leaf interfaces on the switch diagrams.

- Breakout—Breakout mode. Visible when you click one or more leaf interfaces on the switch diagrams.

- Fabric—Add policies to a fabric interface. Visible when you click a port that is eligible to be a fabric port.

- Uplink and Downlink—Convert eligible uplinks to downlinks and vice versa.

- Spine—Visible when you click one or more leaf interfaces on the switch diagrams.

View an API interchange

When you perform a task in the APIC graphical user interface (GUI), the GUI creates and sends internal API messages to the operating system to execute the task. By using the API Inspector, which is a built-in tool of the APIC, you can view and copy these API messages. A network administrator can replicate these messages in order to automate key operations, or you can use the messages as examples to develop external applications that will use the API.

Procedure

Step 1

Log in to the APIC GUI.

Step 2

In the upper right corner of the APIC window, click the System Tools icon to view the drop-down list.

Step 3

In the drop-down list, choose the Show API Inspector.

The API Inspector opens in a new browser window.

Step 4

In the Filters toolbar of the API Inspector window, choose the types of API log messages to display.

The displayed messages are color-coded according to the selected message types. This table shows the available message types:

Name

Description

trace

Displays trace messages.

debug

Displays debug messages. This type includes most API commands and responses.

info

Displays informational messages.

warn

Displays warning messages.

error

Displays error messages.

fatal

Displays fatal messages.

all

Checking this checkbox causes all other checkboxes to become checked. Unchecking any other checkbox causes this checkbox to be unchecked.

Step 5

In the Search toolbar, you can search the displayed messages for an exact string or by a regular expression.

This table shows the search controls:

Name

Description

Search

In this text box, enter a string for a direct search or enter a regular expression for a regex search. As you type, the first matched field in the log list is highlighted.

Reset

Click this button to clear the contents of the Search text box.

Regex

Check this checkbox to use the contents of the Search text box as a regular expression for a search.

Match case

Check this checkbox to make the search case sensitive.

Disable

Check this checkbox to disable the search and clear the highlighting of search matches in the log list.

Next

Click this button to cause the log list to scroll to the next matched entry. This button appears only when a search is active.

Previous

Click this button to cause the log list to scroll to the previous matched entry. This button appears only when a search is active.

Filter

Check this checkbox to hide nonmatched lines. This checkbox appears only when a search is active.

Highlight all

Check this checkbox to highlight all matched fields. This checkbox appears only when a search is active.

Step 6

In the Options toolbar, you can arrange the displayed messages.

This table shows the available options:

Name

Description

Log

Check this checkbox to enable logging.

Wrap

Check this checkbox to enable wrapping of lines to avoid horizontal scrolling of the log list.

Newest at the top

Check this checkbox to display log entries in reverse chronological order.

Scroll to latest

Check this checkbox to scroll immediately to the latest log entry.

Clear

Click this button to clear the log list.

Close

Click this button to close the API Inspector.

Example

This example shows two debug messages in the API Inspector window:

13:13:36 DEBUG - method: GET url: http://192.0.20.123/api/class/infraInfra.json
response: {"imdata":[{"infraInfra":{"attributes":{"instanceId":"0:0","childAction":"",
"dn":"uni/infra","lcOwn":"local","name":"","replTs":"never","status":""}}}]}

13:13:40 DEBUG - method: GET url: http://192.0.20.123/api/class/l3extDomP.json?
query-target=subtree&subscription=yes
response: {"subscriptionId":"72057598349672459","imdata":[]}

APIC GUI icons

This section will help you get a better understanding of the APIC GUI icons.

Standard APIC GUI icons

Frequently displayed icons

Icons

Description

Bookmark this page

Displays concept information for the current menu page

Quick Start

Plays a Quick Start video

Displays a Quick Start procedure

Link to related section

Topology

Pod

Collapse Tree View

Expand Tree View

Collapse All Nodes

Displays a drop-down list of actions

Refresh the displayed information

Download to a file

Upload a file

Faults, statistics and health level icons

Severity levels of faults

Icons

Description

Critical—This icon displays a fault level with critical severity.

Major—This icon displays a fault level with major severity.

Minor—This icon displays a fault level with minor severity.

Warning—This icon displays a fault level that requires a warning.

Next generation APIC GUI

This section outlines the changes to the Cisco APIC GUI beginning with Cisco APIC release 6.1(2).

The new layout is displayed here.

The new GUI

The areas of the new layout:

- 1 indicates the Tools area.

- 2 indicates the Navigation area.

- 3 indicates the Tabs area.

- 4 indicates the Work area.

Tools area

This area contains basic tools that appear regardless of the GUI page.

- Help: Opens a menu with these choices:

- About: Shows the Cisco APIC release, copyright year, and current system time.

- APIC Fabric Setup: Helps you determine what you have not yet configured for your fabric.

- Help Center: Provides links to some of the more commonly-used documentation.

- Open Object Store Browser: Opens the Managed Object Browser, or Visore, which is a utility built into Cisco APIC that provides a graphical view of the managed objects (MOs) using a browser.

- Show API Inspector: Opens the API Inspector, which is a built-in tool of the Cisco APIC that allows you to view the internal API messages between the GUI and the Cisco APIC operating system to execute tasks.

- Notifications: Shows notifications about key issues with the Cisco APIC. Using the drop-down menu, you can choose to show all notifications or only notifications of a specific severity. The menu includes only fault severities that are currently raised. You can also click View All to see all faults in a table.

- Profile: Opens a menu with these choices:

- Switch to existing UI: Switches back to the existing (old) GUI.

- User Preferences: Configures your user settings.

- Change Password: Changes your password.

- Change User Certificate: Changes your login certificate.

- Change SSH Key: Changes the public SSH key that you use for certificate-based login.

- Logout: Logs you out from the Cisco APIC.

Navigation area

This is the core navigation for the Cisco APIC, which is present on all GUI pages.

- Home: Enables you to view information about the fabric. This area has these tabs:

- Fabric View: Shows various information about your the fabric. You can click blue text as well as Controllers, Switches, and Pods to see more information about those things.

- Topology: Shows a graphical representation of your fabric's topology.

- Journey: Shows the main functionalities of Cisco APIC, displays the execution plan for the new GUI, and provides a preview of upcoming features and their functionality.

Most of the stages have a Tell Me More button, which opens a dialog that gives you more information about the part of the GUI that is associated with that stage, including a screenshot. For a stage that is labeled as "Coming Soon," the dialog outlines the upcoming development for that part of the GUI and the screenshot shows how the GUI might look.

User journey

- Manage: Enables you to view information about and configure these things:

- Inventory: Shows information about the controllers, switches, and pods.

- Tenants: Shows information about the tenants.

- Policy Groups: Shows information about the policy groups.

- Policies: Shows information about the configured policies.

- Domains and Pools: Shows information about the domains and pools.

- Software Management: Enables you to upgrade the firmware of the Cisco APICs and switches.

- Monitoring Destinations: Shows information about the monitoring destinations, such as for callhome, SNMP, or TACACS.

- Switches and Interfaces: Shows information about the switches and interfaces.

- Analyze: Enables you to view these things:

- Faults: Shows the faults that Cisco APIC raised due to issues with your fabric.

- History and Logs: Shows the various records and logs, such as fault records, event records, and audit logs.

- Tech Support: Enables you to export the tech support logs, which you provide when you contact the Cisco Technical Assistance Center (TAC).

- Admin: Enables you to use or configure these things:

- Backup and Restore: Enables you to backup and restore the Cisco APIC configurations.

- Authentication: Enables you to configure how users must authenticate to log into the Cisco APIC.

- Active Sessions: Shows the users who are logged into the Cisco APIC.

- Integrations: Shows information about third-party products that you have integrated with Cisco APIC.

- Licensing: Shows information about your active licenses for using Cisco APIC and its features.

- System Settings: Enables you to configure the Cisco APIC.

Tabs area

This area contains tabs that change depending on the core navigation that you chose. The tabs enable you to navigate to subsections of that core navigation.

Work area

This area changes depending on which page you are viewing. The work area shows status, contains links to view configured objects, and much more. Any text in blue is a link that you can click to see more information about or change the configuration of something.

Preview the new APIC GUI

The APIC GUI has been enhanced for a better look-and-feel, and customer experience. However, not all screen have moved to the new GUI. You can take a preliminary look at the new APIC GUI which is presently being developed. However, this is view-only; if you need to make any configuration changes, you will need to revert to the existing GUI and then update the required configurations.

Procedure

Step 1

Click the profile icon in the upper right of any page and choose Switch to new UI.

You cannot view anything in the new GUI that is grayed out. These pages will be added in a future release.

Step 2

To change back to the original GUI so that you can make configuration changes, click the profile icon in the upper right of any page and choose Switch to existing UI.

Enhancements and improvements to the APIC GUI

The Cisco APIC release 6.1(2) includes numerous UI revisions for enhanced ease of use, more detailed information, and quicker navigation with fewer clicks.

Updated GUI fields

Home

> Fabric View

Color coding, more cards, pill-shaped icons, and links to detailed information:

- Health Level > General information with Overview, Faults, and History tabs

- Recent Activity information with Audit Logs, Event Records, and Session Logs

- Inventory information with details on Controllers, Switches and Pods

- Connectivity information with Endpoints, L3 Neighbors, and Multicast Groups

- Interfaces information with Total and Physical detailed lists, dashlets, and cards

- Capacity hyperlinks to Fabric Capacity or Leaf Switch Capacity information

Manage

> Inventory

> Controllers. Controller screen displays the type of controller in the table (Physical or Virtual)

> Controller > select a controller

- > History > Historical Timeline. All health scores are displayed graphically for the past hour or day.

- Interfaces > Physical Interfaces, select an Interface ID. Physical Interfaces details appear, including the Interface ID and CDP/LLDP neighbor information.

> Switches > select a switch

- Switch view graphic appears. A pop-up window appears when hovering over a component.

- Click Rack at the bottom. Rack details appear that show the room, floor and building, a Summary card with Role, Fabric State, Version, and Health of the switch

- > History > Historical Timeline. All health scores are displayed graphically for the past hour or day.

> Pods > select a pod

- Health Summary information such as Health Level, General information, Recent Activity, and Inventory details appear.

- > History > Historical Timeline. All health scores are displayed graphically for the past hour or day.

Table Settings

Gear icon

- Click gear and table Configuration window slides from right.

- Select categories from list for each interface.

Settings persist only in the browser where the user made the changes, even after logging out.

Help

? icon

Help Center

You can access the Help menu in another way besides the "?" icon on the top right. From the Home menu, click Journey > What's New tab. The What's new window appears. Click Go to Help Center at the bottom.

GUI modifications

Chapter 4. Fabric Initialization and Switch Discovery

This chapter contains the following sections:

- Fabric initialization

- Multi-tier fabric topologies example

- Change the external routable subnet

- Switch discovery with the APIC

- Cisco NX-OS to Cisco ACI POAP Auto-conversion

- Cisco Nexus 9000 Switch Secure Erase

Fabric initialization

When you initialize the fabric, you add switches to be managed by the APIC and you validate the steps using the GUI, CLI, or API. Before you start this process, you must create an APIC cluster over the out-of-band network.

- Fabric Topology

Fabric Topology

An example of a fabric topology includes:

- Two spine switches (spine1, spine2)

- Two leaf switches (leaf1, leaf2)

- Three instances of APIC (apic1, apic2, apic3)

The figure shows an example of a fabric topology.

Fabric Topology Example

- Connections: fabric topology

Connections: fabric topology

An example of a fabric topology is as follows:

- Two spine switches (spine1, spine2)

- Two leaf switches (leaf1, leaf2)

- Three instances of APIC (apic1, apic2, apic3)

The following figure shows an example of a fabric topology.

Fabric Topology Example

Multi-tier fabric topologies example

A multi-tier fabric topology is a data center network design that uses an additional layer of leaf switches, creating a three-tier structure. This design is intended for large data center deployments where it separates aggregation and access network functions onto distinct switch layers, enabling flexible connectivity and port usage through configurable fabric ports.

3-tier Core-Aggregation-Access architectures are common in data center network topologies. As of the Cisco APIC 4.1(1) release, you can create a multi-tier ACI fabric topology that corresponds to the Core-Aggregation-Access architecture, mitigating the need to upgrade costly components such as rack space or cabling. The addition of a tier-2 leaf layer makes this topology possible. The tier-2 leaf layer supports connectivity to hosts or servers on the downlink ports and connectivity to the leaf layer (aggregation) on the uplink ports.

In the multi-tier topology, the leaf switches initially have uplink connectivity to the spine switches and downlink connectivity to the tier-2 leaf switches. To make the entire topology an ACI fabric, all ports on the leaf switches connecting to tier-2 leaf fabric ports must be configured as fabric ports (if not already using the default fabric ports). After APIC discovers the tier-2 leaf switch, you can change the downlink port on the tier-2 leaf to a fabric port and connect to an uplink port on the middle layer leaf.

Note

If you are not using the default fabric ports to connect leaf switches to tier-2 leaf, you must convert the leaf ports from downlink to uplink (leaf switch reload required). For more information about changing port connectivity, see the Access Interfaces chapter of the Cisco APIC Layer 2 Networking Configuration Guide.

This figure shows an example of a multi-tier fabric topology.

Multi-Tier Fabric Topology Example

While the topology in the image shows the Cisco APIC and L3Out/EPG connected to the leaf aggregation layer, the tier-2 leaf access layer also supports connectivity to APICs and L3Out/EPGs.

Note

Only Cisco Nexus 9000 Series switches with model numbers that end in EX, and later are supported as tier-2 leaf switches and as leaf switches, if there are tier-2 leaf switches attached to them. For more information, see the Supported Switches and Port Speeds for Multi-Tier Archictecture table.

Tier-2 leaf switches attached to remote leaf switches are not supported.

Supported Switches and Port Speeds for Multi-Tier Architecture

Switch

Maximum supported downlink port* (as tier-2 leaf)

Maximum supported fabric ports (as tier-2 leaf)

Maximum supported fabric ports (as tier-1 leaf)

Nexus 93180YC-EX

48x1/10/25-Gbps

4x40/100-Gbps

48 x 10/25-Gbps

6 x 40/100-Gbps

48 x 10/25-Gbps

6 x 40/100-Gbps

Nexus 93108TC-EX

48x100M/1/10G BASE-T

4x40/100-Gpbs

6 x 40/100-Gbps

6 x 40/100-Gbps

N9K-9348GC-FXP**

48 x 100M/1G BASE-T

4 x 10/25-Gbps

2 x 40/100-Gbps

4 x 10/25-Gbps

2 x 40/100-Gbps

N9K-93180YC-FX

48 x 1/10/25-Gbps

4x40/100-Gbps

48 x 10/25-Gbps

6 x 40/100-Gbps

48 x 10/25-Gbps

6 x 40/100-Gbps

N9K-93108TC-FX

48 x 100M/1/10G BASE-T

4x40/100-Gbps

6 x 40/100-Gbps

6 x 40/100-Gbps

N9K-93240YC-FX2

48x1/10/25-Gbps

10x40/100-Gbps

48x1/10/25-Gbps

12x40/100-Gbps

48x10/25-Gbps fiber ports

12x40/100-Gbps

N9K-C9336C-FX2

34 x 40/100-Gbps

36 x 40/100-Gbps

36 x 40/100-Gbps

N9K-C93216TC-FX2***

96 x 10G BASE-T

10 x 40/100-Gbps

12 x 40/100-Gbps

12 x 40/100-Gbps

N9K-C93360YC-FX2***

96 x 10/25-Gbps

10 x 40/100-Gbps

52 x 10/25Gbps

12 x 40/100Gbps

52 x 10/25Gbps

12 x 40/100Gbps

N9K-C9364C-GX

62 x 40/100-Gbps

62 x 40/100-Gbps

62 x 40/100-Gbps

* Last 2 original fabric ports cannot be used as downlink ports.

** If tier-2 leaf does not require much bandwidth, it can be used as tier-1 though it has fewer fiber ports. Copper port cannot be used as a fabric port.

*** Supported beginning with Cisco APIC Release 4.1(2).

Change the external routable subnet

If you need to make changes to the information in subnets or TEP table after you've configured them, follow these steps to change the external routable subnet.

Note

You cannot change an external routable subnet configuration using multiple subnets.

Follow these steps to change the external routable subnet:

Procedure

Step 1

Navigate to the area where you originally configured the external routable subnet.

1. On the menu bar, click Fabric > Inventory .

2. In the Navigation pane, click Pod Fabric Setup Policy .

3. On the Fabric Setup Policy panel, double-click the pod where you originally configured the external routable subnet.

The Fabric Setup Policy for a POD page for this pod appears.

4. Locate the information for the subnets or TEP table, depending on the release of your APIC software:

- For releases prior to 4.2(3), locate the Routable Subnets table.

- For 4.2(3) only, locate the External Subnets table.

- For 4.2(4) and later, locate the External TEP table.

Step 2

Locate the external routable subnet that you want to delete in the table and determine if the state of that subnet is set to active or inactive .

If the state is set to active , change the state to inactive :

1. Double-click on the entry in the subnets or TEP table for the existing external routable subnet that you want to delete.

2. Change the state for the subnet to inactive , then click Update .

Step 3

Delete the existing external routable subnet.

1. Click on the entry in the subnets or TEP table for the existing external routable subnet that you want to delete.

2. Click the trashcan icon at the top of the table, then click Yes in the pop-up confirmation window to delete the external routable subnet.

Step 4

Wait for at least 30 seconds, then configure a new external routable subnet.

1. Click + in the subnets or TEP table to configure a new external routable subnet.

2. Enter the IP address and Reserve Address, if necessary, and set the state to active or inactive .

- The IP address is the subnet prefix that you wish to configure as the routeable IP space.

- The Reserve Address is a count of addresses within the subnet that must not be allocated dynamically to the spine switches and remote leaf switches. The count always begins with the first IP in the subnet and increments sequentially. If you wish to allocate the Unicast TEP from this pool, then it must be reserved.

3. Click Update to add the new external routable subnet to the subnets or TEP table.

4. On the Fabric Setup Policy panel, click Submit .

Step 5

Verify that the new routable IP address is configured correctly.

Log into the APIC controller through the CLI and enter the following command:

apic1#
avread | grep routableAddress

Output similar to the following should appear:

routableAddress   14.3.0.228              14.3.0.229              14.3.1.228

Step 6

Check the NAT entries created on the spine switch.

Log into the spine switch through the CLI and enter the following command:

spine1#
show nattable

Output similar to the following should appear:

-----NAT TABLE---------
Private Ip  Routable Ip
----------  -----------
10.0.0.2    14.3.0.229
10.0.0.1    14.3.0.228
10.0.0.3    14.3.1.228

Switch discovery with the APIC

Switch discovery is a fabric management process that enables a controller to automatically detect and register switches in the network. The APIC is a central point of automated provisioning and management for all the switches that are part of the ACI fabric. A single data center might include multiple ACI fabrics; each data center might have its own APIC cluster and Cisco Nexus 9000 Series switches that are part of the fabric. To ensure that a switch is managed only by a single APIC cluster, each switch must be registered with that specific APIC cluster that manages the fabric.

The APIC discovers new switches that are directly connected to any switch it currently manages. Each APIC instance in the cluster first discovers only the leaf switch to which it is directly connected. After the leaf switch is registered with the APIC, the APIC discovers all spine switches that are directly connected to the leaf switch. As each spine switch is registered, that APIC discovers all the leaf switches that are connected to that spine switch. In a few simple steps, this cascaded discovery allows the APIC to discover the entire fabric topology.

- Switch Registration with the APIC Cluster

- Switch Role Considerations

- Registering an Unregistered Switch Using the GUI

- Add a switch before discovery using the GUI

- Switch discovery validation and switch management from the APIC

- Validate the registered switches using the GUI

- Validate the fabric topology

- Validate the fabric topology using the GUI

- Unmanaged switch connectivity in VM management

- Find your switch inventory using the GUI

- Troubleshoot Switch Discovery Issues

- Maintenance Mode

Switch Registration with the APIC Cluster

After you register a switch with the Cisco Application Policy Infrastructure Controller (APIC), the switch is part of the Cisco APIC-managed fabric inventory. With the Cisco Application Centric Infrastructure (ACI) fabric, the Cisco APIC is the single point of provisioning, management, and monitoring for switches in the infrastructure.

These guidelines and limitations apply:

- Before you begin registering a switch, make sure that all switches in the fabric are physically connected and booted in the desired configuration. For information about the installation of the chassis, see https://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

When the switch is running a different version than your APIC cluster, use Auto Firmware Update on Switch Discovery to automatically upgrade the switch during the discovery phase. See Auto Firmware Update on Discovery in the Cisco APIC Installation and ACI Upgrade and Downgrade Guide for details.

- The infrastructure IP address range must not overlap with other IP addresses used in the ACI fabric for in-band and out-of-band networks.

- When a switch is power cycled or upgraded, downlink interfaces will be in the admin-down state until the switch downloads the configurations again from the Cisco APICs. This process prevents external devices from sending traffic to a switch before it is ready. Fabric links and down links for Cisco APIC connectivity are exempt from being changed to the admin-down state. To achieve this exemption, the leaf switch remembers the downlink interface that was connected to the Cisco APICs prior to the power cycle or upgrade. Because of this, you must not change the Cisco APIC connectivity until the switches are fully operational again after the power cycle or upgrade.

Switch Role Considerations

A switch role is a fundamental part of network architecture design. It defines a switch's operational behavior in fabric topology, port connectivity, and discovery protocols. The role of a switch can be changed using the CLI based on the deployment scenario.

- The default fabric links must be used for initial switch discovery from another switch.

- If a default spine switch is connected to the Cisco Application Policy Infrastructure Controller (APIC) directly, the switch will be converted to a leaf switch automatically. During the conversion period, the fault will be present in the Cisco APIC, which is a normal behavior. The fault will be removed after the switch conversion is finished.

- For a leaf switch, you can configure a port profile to convert a port to be a downlink or fabric link after the port is registered to the Cisco APIC. For more information, see the Cisco APIC Layer 2 Networking Configuration Guide at the following site:

https://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html#Configuration_Guides

This table specifies the default role for the switches that support role changes:

Default Switch Roles

Switch Product ID

Default Role

First Release to Support a Role Change 1

N9K-C93600CD-GX

Leaf

5.2(1)

N9K-C9316D-GX

Spine

5.1(4)

N9K-C9364C-GX

Leaf

5.1(3)

N9K-C9332D-GX2B

Leaf

5.2(3)

N9K-C9364D-GX2A

Spine

5.2(3)

N9K-C9348D-GX2A

Spine

5.2(3)

N9K-C9408

Spine

6.0(2)

1 Specifies the first release to support the role change for the indicated switch. Role changing for that switch is supported in all subsequent releases.

The hybrid switches can change roles. The default role for hybrid switches is different for each model. In release 6.1(2), you can change the role of the switch by using a CLI command before the switch is discovered. The default role of these switches determines the role of interfaces during the discovering phase. This may cause some inconvenience when the switch is used in its non-default role. For instance, a switch whose default role is leaf is cabled up as a spine switch. Once the switch is discovered, its role is automatically converted to spine. However, by default, it boots up as a leaf switch and most of its interfaces are not configured as fabric links that can be used for spine discovery. As a result, you may have limited cabling options to ensure that the switch is discovered.

Another example: when the switch whose default role is spine is cabled up as a leaf switch and is supposed to be directly connected to an APIC. All the interfaces of a spine switch, however, are fabric links that cannot be used to connect to an APIC. As a result, the switch must be discovered through another switch which is connected to another APIC, so that it can be converted to a leaf switch–which now can be connected to an APIC.

Changing the default role of a switch is similar to assigning a new job function before an employee joins a company. Assigning the correct role ensures the switch is properly integrated into the network infrastructure from the outset, avoiding connectivity limitations and extra work.

To address these issues, use this CLI command on the new, Undiscovered switch to change its role before being discovered.

(none)# acidiag setrole <leaf/spine>
This command will reboot the switch, Proceed? [y/N]

Note

The acidiag setrole <leaf/spine> command will NOT work if the switch has been discovered. You will see an error message.

Registering an Unregistered Switch Using the GUI

Before you begin

Ensure that all switches in the fabric are physically connected and booted.

Note

The infrastructure IP address range must not overlap with other IP addresses used in the ACI fabric for in-band and out-of-band networks.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the Navigation pane, choose Fabric Membership.

Step 3

In the Work pane, click the Nodes Pending Registration tab.

Switches in the Nodes Pending Registration tab table may be in one of these conditions:

- A newly discovered but unregistered node has a node ID of 0 and has no IP address.

- A manually entered (in Cisco Application Policy Infrastructure Controller (APIC)) but unregistered switch has an original status of Undiscovered until it is physically connected to the network. Once connected, the status changes to Discovered.

Step 4

In the Nodes Pending Registration table, locate a switch with an ID of 0 or a newly connected switch with the serial number you want to register.

Step 5

(Optional) Double-click the row of that node if you want to see more information about the node.

A dialog appears that shows you various node properties, such as the ACI-mode switch release and information about LLDP neighbors.

Step 6

Right-click the row of that switch, choose Register, and perform the following actions:

1. Verify the displayed Serial Number to determine which switch is being added.

2. Configure or edit the following settings:

Field

Setting

Pod ID

Identifier of the pod where the node is located.

Node ID

A number greater than 100. The first 100 IDs are reserved for Cisco APIC appliance nodes.

Note

We recommend that you number leaf nodes and spine nodes be differently. Assign numbers to spine nodes in the 100 range (such as 101 and 102) and assign numbers to leaf nodes in the 200 range (such as 201 and 202).

After the node ID is assigned, it cannot be updated. After the node has been added to the Registered Nodes tab table, you can update the node name by right-clicking the table row and choosing Edit Node and Rack Name.

RL TEP Pool

Tunnel endpoint (TEP) pool identifier for the node.

Node Name

The node name, such as leaf1 or spine3.

Role

The assigned node role. The options are:

- spine

- leaf

- virtualleaf

- virtualspine

- remote leaf

- tier-2-leaf

If you choose a role other than the default role for the node, the node automatically reboots during the registration to change the role.

Rack Name

Choose Default, or choose Create Rack to add a name and description.

3. Click Register.

Cisco APIC assigns an IP address to the node and the node is added to the Registered Nodes tab table. Next and if applicable, other nodes that are connected to this node are discovered and appear in the Nodes Pending Registration tab table.

Step 7

Continue to monitor the Nodes Pending Registration tab table. As more nodes appear, repeat these steps to register each new node until all installed nodes are registered.

Add a switch before discovery using the GUI

You can add a switch description before the switch is physically connected to the network by following these steps:

Before you begin

Ensure you have the serial number of the switch.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the Navigation pane, choose Fabric Membership.

Step 3

On the Registered Nodes or Nodes Pending Registration work pane, click the Actions icon, then click Create Fabric Node Member.

The Create Fabric Node Member dialog appears.

Step 4

Configure the following settings:

Field

Setting

Pod ID

Identify the pod where the node is located.

Serial Number

Required: Enter the serial number of the switch.

Node ID

Required: Enter a number greater than one hundred. The first one hundred IDs are reserved for Cisco Application Policy Infrastructure Controller (APIC) appliance nodes.

Note

We recommend that you number leaf nodes and spine nodes differently. For example, number leaf nodes in the 100 range (such as 101, 102) and number spine nodes in the 200 range (such as 201, 202).

After you assign the node ID, you cannot update it. Once the node has been added to the Registered Nodes tab table, you can update the node name. To do this, right-click the table row and choose Edit Node and Rack Name.

Switch Name

The node name, such as leaf1 or spine3.

Node Type

Choose the type (role) for the node. The options are:

- leaf

Put a check in one of these boxes if applicable:

- Is Remote: Specifies that the node is a remote leaf switch.

- Is Virtual: Specifies that the node is virtual.

- Is Tier-2 Leaf: The fabric node member (leaf switch) being created will take on the characteristics of a tier-2 leaf switch in a multi-tier architecture.

- spine

Put a check in this box if applicable:

- Is Virtual: Specifies that the node is virtual.

- unknown

If you choose a role other than the default role for the node, the node automatically reboots during the registration to change the role.

VPC Pair

Optional. If the node is part of a vPC pair, choose the ID of the node with which to pair this node.

VPC Domain ID

Enter the vPC domain ID for the vPC pair. The range is from 1 to 1,000. This field only appears if you entered a value for VPC Pair, and is required in that case.

The Cisco APIC adds the new node to the Nodes Pending Registration tab table.

What to do next

Connect the physical switch to the network. After you connect the switch, the Cisco APIC matches the serial number of the physical switch to the new entry. Monitor the Nodes Pending Registration tab table until the Status for the new switch changes from Undiscovered to Discovered. Then, follow the steps in the Registering an Unregistered Switch Using the GUI section to complete the fabric initialization and discovery process for the new switch.

Switch discovery validation and switch management from the APIC

After you register the switches with the Cisco Application Policy Infrastructure Controller (APIC), the APIC automatically discovers the fabric topology so you can view and manage all the switches in the network.

You can configure, monitor, and upgrade each switch from the APIC without having to access the individual switches.

Validate the registered switches using the GUI

Confirm that switches are registered to the fabric and display their node IDs and assigned IP addresses.

Procedure

Step 1

On the menu bar, navigate to Fabric > Inventory > Fabric Membership.

Step 2

In the Fabric Membership work pane, click the Registered Nodes tab.

The Registered Nodes tab displays the switches in the fabric, along with their node IDs and assigned IP addresses.

Validate the fabric topology

After you register all the switches with the APIC cluster, the APIC automatically discovers all links and connectivity in the fabric and discovers the entire topology.

Validate the fabric topology using the GUI

Procedure

Step 1

On the menu bar, navigate to Fabric > Inventory > Pod number.

Step 2

In the Work pane, click the Topology tab.

The displayed diagram shows all attached switches, APIC instances, and links.

Step 3

(Optional) Hover over any component to view its health, status, and inventory information.

Step 4

(Optional) To view the port-level connectivity of a leaf switch or spine switch, double-click its icon within the topology diagram.

Step 5

(Optional) To refresh the topology diagram, click the icon in the upper right corner of the Work pane.

Unmanaged switch connectivity in VM management

An unmanaged switch in VM management is a Layer 2 device that can be connected between VM hosts and leaf ports. It must be configured with a management address which is advertised using LLDP or CDP. APIC uses that management address to discover and identify the unmanaged switch.

You can connect hosts managed by the VM controller, such as a vCenter, to the leaf port through a Layer 2 switch. APIC automatically discovers Layer 2 switches and identifies them by management address.

Find your switch inventory using the GUI

Use this procedure when you need to locate hardware information for your switches managed by Cisco APIC.

Before you begin

You must have access to the Cisco APIC GUI

Follow this procedure to find your switch inventory using the GUI:

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the navigation pane, click a Pod icon.

Your switch icons appear in the navigation pane.

Step 3

In the navigation pane, click on a switch icon.

A list of tabs appears at the top of the work pane.

Step 4

Click the General tab.

Your switch information appears in the work pane.

What to do next

Troubleshoot Switch Discovery Issues

The ACI-mode switch software includes a comprehensive leaf and spine switch discovery validation program. If your switch is stuck in discovery mode, use the switch CLI command to launch the leaf and spine switch discovery validation program.

The validation program performs the following tests:

1. System state—Checks the state of the topSystem managed object (MO).

1. If the state is "out-of-service," checks for any scheduled upgrades.

2. If the state is "downloading bootscript," a failure has occurred in the downloading bootscript. The failure is reported. If the switch is an L3out spine, the program also checks the bootstrap download state and reports any failure.

2. DHCP status—Checks for DHCP status and information, such as the TEP IP, node Id, and name assigned from the dhcpResp MO.

3. AV details—Checks whether the APICs are registered and whether they have valid IP addresses.

4. IP reachability—Uses the iping command to verify IP reachability to the address assigner APIC. To retest this condition, use the show discoveryissues apic ipaddress command.

5. infra VLAN received—Checks for the presence of the infra VLAN details in the lldpInst MO. If this switch belongs to a pod that has no APIC, no infra VLAN details are present, and this section of the test result can be ignored.

6. LLDP adjacency—Checks for the presence of LLDP adjacencies and for any wiring mismatch issues. LLDP issues can generate fault reports such as infra VLAN mismatch, chassis ID mismatch, or no connection to the front end ports.

7. Switch version—Reports the running firmware version of the switch. Also reports the version of the APIC, if available.

8. FPGA/BIOS—Checks for any FPGA/BIOS version mismatch on the switch.

9. SSL validation—Checks for validity of the SSL certificate details using the acidiag verifyssl -s serialNumber command.

10. Policy downloads—Checks the pconsBootStrap MO to see whether registration to APIC (policy managment shards) is complete and whether all policies were downloaded successfully.

11. Time—Reports the current time on the switch.

12. Hardware status—Checks the module, power, and fan status from the eqptCh, eqptFan, eqptPsu, eqptFt, and eqptLC MOs.

Running the Test Manually

To run the switch discovery validation program, log in to the spine or leaf switch CLI console and run this command:

show discoveryissues [apic ipaddress]

Example of a Successful Test

This output is an example of a successful switch discovery validation test.

spine1# show discoveryissues

Checking the platform type................SPINE!
Check01 - System state - in-service                [ok]
Check02 - DHCP status                [ok]
TEP IP: 10.0.40.65 Node Id: 106 Name: spine1
Check03 - AV details check                [ok]
Check04 - IP reachability to apic                [ok]
Ping from switch to 10.0.0.1 passed
Check05 - infra VLAN received                [ok]
infra vLAN:1093
Check06 - LLDP Adjacency                [ok]
Found adjacency with LEAF
Check07 - Switch version                [ok]
version: n9000-14.2(0.167)  and apic version: 5.0(0.25)
Check08 - FPGA/BIOS out of sync test                [ok]
Check09 - SSL check                [check]
SSL certificate details are valid
Check10 - Downloading policies                [ok]
Check11 - Checking time                [ok]
2019-08-21 17:15:45
Check12 - Checking modules, power and fans                [ok]

Example of a Failed Test

This ouptput shows an example of a switch discovery validation program with discovery issues.

spine1# show discoveryissues

Checking the platform type................SPINE!
Check01 - System state - out-of-service                [FAIL]
Upgrade status is notscheduled
Node upgrade is notscheduled state
Check02 - DHCP status                [FAIL]
ERROR: discover not being sent by switch
Ignore this, if the IP is already known by switch
ERROR: node Id not configured
ERROR: IP not assigned by dhcp server
ERROR: Address assigner's IP not populated
TEP IP: unknown Node Id: unknown Name: unknown
Check03 - AV details check                [ok]
Check04 - IP reachability to apic                [FAIL]
please rerun the CLI with argument apic Ip
(show discoveryissues apic <ip>) to check its reachability from switch
Check05 - infra VLAN received                [FAIL]
Please ignore if this switch is part of a pod with no apic
Check06 - LLDP Adjacency                [FAIL]
Error: spine not connected to any leaf
Check07 - Switch version                [ok]
version: n9000-14.2(0.146)  and apic version: unknown
Check08 - FPGA/BIOS out of sync test                [ok]
Check09 - SSL check                [ok]
SSL certificate details are valid
Check10 - Downloading policies                [FAIL]
Registration to all PM shards is not complete
Policy download is not complete
Pcons booststrap is in triggered state
Check11 - Checking time                [ok]
2019-07-17 19:26:29
Check12 - Checking modules, power and fans                [FAIL]
Line card state is testing

Maintenance Mode

These terms are helpful to understand when using maintenance mode:

- Maintenance mode: Used to isolate a switch from user traffic for debugging purposes. You can put a switch in maintenance mode by enabling the Maintenance (GIR) field in the Fabric Membership page in the APIC GUI, located at Fabric > Inventory > Fabric Membership (right-click on a switch and choose Maintenance (GIR)).

If you put a switch in maintenance mode, that switch is no longer considered part of the operational ACI fabric infra and it will not accept regular APIC communications.

You can use maintenance mode to gracefully remove a switch and isolate it from the network in order to perform debugging operations. When you enable maintenance mode, the switch is removed from the regular forwarding path with minimal traffic disruption.

In graceful removal, all external protocols except the fabric protocol (IS-IS) are gracefully brought down, and the switch is isolated from the network. During maintenance mode, the maximum metric is advertised in IS-IS within the Cisco Application Centric Infrastructure (Cisco ACI) fabric, so the leaf switch in maintenance mode does not attract traffic from the spine switches. In addition, all front-panel interfaces on the switch are shut down except for the fabric interfaces. To return the switch to its fully operational (normal) mode after the debugging operations, recommission the switch. This action triggers a stateless reload of the switch.

During graceful insertion, the system automatically decommissions, reboots, and recommissions the switch. After recommissioning is completed, the system restores all external protocols and resets maximum metric in IS-IS 10 minutes later.

These protocols are supported:

- Border Gateway Protocol (BGP)

- Enhanced Interior Gateway Routing Protocol (EIGRP)

- Intermediate System-to-Intermediate System (IS-IS)

- Open Shortest Path First (OSPF)

- Link Aggregation Control Protocol (LACP)

Protocol Independent Multicast (PIM) is not supported.

Important Notes

- If you place a border leaf switch with static route in maintenance mode, the route might remain in the routing tables of other switches in the ACI fabric. This can cause routing issues.

To work around this issue, perform one of these actions:

- Configure the same static route with the same admnistrative distance on the other border leaf switch, or

- Use IP SLA or BFD for track reachability to the next hop of the static route

- While the switch is in maintenance mode, the Ethernet port module stops propagating the interface related notifications. As a result, if the remote switch is rebooted or the fabric link is flapped during this time, the fabric link will not come up afterward unless the switch is manually rebooted (using the acidiag touch clean command), decommissioned, and recommissioned.

- While the switch is in maintenance mode, CLI 'show' commands on the switch show the front panel ports as being in the up state and the BGP protocol as up and running. The interfaces are actually shut and all other adjacencies for BGP are brought down, but the displayed active states allow you to to debug.

- For multi-pod / multi-site, set the IS-IS metric for redistributed routes to less than 63 to minimize the traffic disruption when bringing the node back into the fabric. To set the IS-IS metric for redistributed routes, choose Fabric > Fabric Policies > Pod Policies > IS-IS Policy.

- When you reboot a spine or leaf and after the IS-IS adjacency comes up the IS-IS metric for redistributed routes is advertised as high, which is, 34 and will not be available as an ECMP next hop.

- Existing GIR supports all Layer 3 traffic diversion. With LACP, all the Layer 2 traffic is also diverted to the redundant node. After a node enters maintenance mode, LACP immediately informs neighboring switches that aggregation is no longer possible in the port-channel. Then, all traffic is diverted to the vPC peer node.

- You cannot perform these operations in maintenance mode:

- Upgrade: Upgrading the network to a newer version to isolate a switch from user traffic

- Stateful Reload: Restarting the GIR node or its connected peers

- Stateless Reload: Restarting with a clean configuration or power cycle of the GIR node or its connected peers

- Link Operations: Shut / no-shut or optics OIR on the GIR node or its peer node

- Configuration Change: Any configuration change (such as clean configuration, import, or snapshot rollback)

- Hardware Change: Any hardware change (such as adding, swapping, removing FRU's or RMA)

- Set a switch to maintenance mode using the GUI

- Set a switch to operational mode using the GUI

Set a switch to maintenance mode using the GUI

Use this procedure to set a switch to maintenance mode using the GUI to safely perform upgrades or diagnostics. Out-of-band management interfaces will remain up and accessible during this process.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the navigation pane, click Fabric Membership.

Step 3

In the work pane, click Actions > Maintenance (GIR).

Step 4

Click OK.

The gracefully removed switch displays Debug Mode in the Status column.

The switch enters the maintenance mode and displays Debug Mode in the Status column.

Set a switch to operational mode using the GUI

Use this procedure to insert a switch to operational mode using the GUI.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the navigation pane, click Fabric Membership.

Step 3

In the Registered Nodes table in the work pane, right-click the row for the switch you want to set to operational mode and select Commission.

Step 4

Click Yes.

The switch transitions into operational mode and is ready to use.

What to do next

Verify the switch connectivity and status in the operational dashboard.

Cisco NX-OS to Cisco ACI POAP Auto-conversion

Cisco NX-OS to Cisco Application Centric Infrastructure (ACI) power-on auto-provisioning (POAP) auto-conversion feature automates the process of upgrading software images and installing configuration files on nodes that are being deployed in the network for the first time. If a Cisco NX-OS node with the POAP auto-conversion feature boots without a startup configuration, the node enters the POAP mode and starts DHCP discovery on all ports. The node locates a DHCP server and bootstraps itself with its interface IP address, gateway, and DNS server IP addresses. The device also obtains the IP address of a TFTP server and downloads a configuration script that enables the node to download and install the appropriate software image and configuration file. This process converts the Cisco NX-OS node from standalone mode to Cisco ACI-mode.

To auto-convert a Cisco NX-OS node to a Cisco ACI node using POAP, you need to specify an interface on a Cisco ACI switch node that is connected to the Cisco NX-OS node that needs to be auto-converted. The interface specified on the Cisco ACI switch enables the handling of POAP and allows the Cisco NX-OS node to use the Cisco Application Policy Infrastructure Controller (APIC) as its DHCP server for auto-conversion. The Cisco ACI switch node must be already registered with the Cisco ACI fabric and be active. This means the node is reachable from the Cisco APIC cluster. This auto-conversion can be used when adding a new switch to the fabric or when replacing an existing Cisco ACI switch.

- Guidelines and limitations for Cisco NX-OS to Cisco ACI POAP auto-conversion

- Convert a Cisco NX-OS node to Cisco ACI with POAP Auto-conversion using the GUI

Guidelines and limitations for Cisco NX-OS to Cisco ACI POAP auto-conversion

Supported hardware for Cisco NX-OS to Cisco ACI POAP auto-conversion

Cisco NX-OS to Cisco ACI POAP auto-conversion supports these switches:

- Switches with -EX, -FX, -GX, or newer PIDs

- Cisco N9K-C9364C and N9K-C9332C switches

Guidelines for Cisco NX-OS to Cisco ACI POAP auto-conversion

These guidelines apply when using Cisco NX-OS to Cisco Application Centric Infrastructure (ACI) power-on auto-provisioning (POAP) auto-conversion:

- Because a Cisco NX-OS node that is being converted starts to send discover packets on all interfaces including management, any external DHCP server (apart from the Cisco Application Policy Infrastructure Controller's (APIC's) server) should be removed, as they may intercept POAP discover packets and disrupt the conversion.

- After you auto-convert a spine or leaf node, the show system reset-reason CLI command does not display any information regarding conversion. The output only states this:

reset-requested-by-cli-command-reload

Limitations for Cisco NX-OS to Cisco ACI POAP auto-conversion

These limitations apply when using Cisco NX-OS to ACI POAP auto-conversion:

- Cisco NX-OS to Cisco ACI POAP auto-conversion is supported only when the NX-OS device to be converted is connected to an existing Cisco ACI switch node that has reachability to the Cisco APIC cluster. Due to this reason, these scenarios are not supported:

- When discovering the first Cisco ACI switch from APIC 1.

- When replacing a Cisco ACI leaf node when a Cisco APIC is single-homed to the leaf node.

- When adding or replacing a Cisco ACI switch that reaches to the Cisco APIC cluster only through an IPN device. That is, when adding a Cisco NX-OS node as a new remote leaf node, adding a Cisco NX-OS node as a first spine node in a new pod, replacing a remote leaf node, or replacing a spine node in a Cisco ACI Multi-Pod setup with only one spine node in the pod. This scenario is supported beginning with the Cisco APIC 5.2(4) release with the required configurations on the IPN device.

- Modular spine node supervisor replacement is not supported.

- You must use optical cables between Cisco ACI switches and Cisco NX-OS switches. You cannot use copper cables in this case.

- The Cisco ACI switch image that needs to be used for auto-conversion must be present on the Cisco APIC cluster's firmware repository. You can use the GUI to check that the image is present by going to Admin > Firmware > Images.

- Cisco NX-OS to Cisco ACI auto-conversion using POAP is not supported when the target switch release is 16.0(3) or later while the current release running on the switch is Cisco NX-OS 9.3(12) or earlier. If you attempt to use the Cisco NX-OS to ACI auto-conversion by using POAP under this condition, the switch may get stuck indefinitely. To convert Cisco NX-OS to Cisco ACI under these conditions, you must perform the upgrade manually.

Convert a Cisco NX-OS node to Cisco ACI with POAP Auto-conversion using the GUI

Use this task when migrating a standalone Cisco NX-OS node to Cisco ACI mode without decommissioning the node.

Before you begin

Enable Auto Firmware Update on Switch Discovery with the target Cisco ACI firmware version. For more information, see the Cisco APIC Getting Started Guide.

Follow these steps to convert a Cisco NX-OS node to Cisco ACI with POAP Auto-conversion using the GUI.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the Navigation pane, choose Fabric Membership.

Step 3

In the Work pane, choose the Registered Nodes tab.

Step 4

(Optional) When replacing an existing Cisco ACI switch node with a new switch that may be running NX-OS, right-click the node to be replaced and choose Remove From Controller as you would do for a regular replacement scenario.

Step 5

In the action menu at the top right of the table, choose Add with NXOS to ACI Conversion.

In the replacement scenario, if the switch node to be replaced is decommissioned or inactive, you can alternatively right-click the node and choose Replace with NXOS to ACI Conversion. This will perform actions Remove From Controller from step 4 and Add with NXOS to ACI Conversion from step 5 at the same time.

Step 6

In the dialog, fill out the fields as follows:

- Node ID: Choose the ID of a node that is connected to the node that you want to convert. You can click the trash can to delete a node or + to add another node. Specify at least one node. You can click Hide Interfaces to hide the interface information if you need more space in the GUI when configuring additional nodes.

- Interface ID: Choose the ID of one of the node's interfaces that is connected to the node that you want to convert. You can click the trash can to delete an interface or + to add another interface. Configure only one interface in each node to handle POAP for POAP auto-conversion.

Step 7

Click Submit.

Step 8

Choose the Nodes Pending Registration tab.

After the node appears in this tab, the node registration procedure is the same as for regular Cisco ACI switches.

Step 9

(Optional) After the switch is registered and has joined the fabric with the Active status, you may delete the POAP auto-conversion setting on the interface that you configured in step 6. After the conversion completes, delete the POAP settings from the connected node:

1. Choose the Registered Nodes tab.

2. Double-click the row of the node from which you want to delete the POAP settings.

3. In the dialog, choose the NXOS Conversion Policy tab.

4. Select the pathname that you want to delete, then click the delete icon (the trashcan).

The Cisco NX-OS node is converted to Cisco ACI mode and registered in the fabric. You can now remove POAP auto-conversion settings from associated interfaces.

What to do next

- Verify the node’s operational status in the fabric.

- Reconfigure any necessary POAP settings on remaining interfaces.

Cisco Nexus 9000 Switch Secure Erase

Cisco Nexus 9000 switches utilize persistent storage to maintain system software images, switch configuration, software logs, and operational history. Each of these areas can contain information specific to the user, including network architecture details, design data, and security vulnerabilities. The secure erase feature enables you to comprehensively remove information during situations such as returning a switch for return merchandise authorization (RMA), upgrading or replacing a switch, or decommissioning a system that has reached its end of life.

Cisco APIC release 6.0(1) and later support secure erase. All the leaf and spine switches in the fabric must use APIC release 6.0(1) or later.

This feature erases user data from the following storage devices:

- SSD

- EMMC

- MTD

- CMOS

- NVRAM

Note

Not every switch model has all these storage devices.

- Securely Erase user data from a Cisco Nexus 9000 switch using the GUI

- Securely Erase user data from a module of a Cisco Nexus 9000 modular switch line card using the GUI

- Securely Erase user data from a Cisco Nexus 9000 switch using the switch's CLI

Securely Erase user data from a Cisco Nexus 9000 switch using the GUI

Use this task to decommission a switch and ensure no user data or configuration remains.

Before you begin

Use this procedure to securely erase user data from a Cisco Nexus 9000 switch using the GUI.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the Navigation pane, choose Fabric Membership.

Step 3

In the Work pane, right-click the switch (node) that you want to securely erase and choose Decommission.

Step 4

In the Decommission dialog, choose Decommission & Secure Remove.

Step 5

Click OK.

The decommission process takes from 2 to 8 hours, depending on the switch and SSD type. The process securely erases the switch and removes the switch configuration from the Cisco Application Policy Infrastructure Controller (APIC). However, the secure erase process does not remove the NX-OS image from the bootflash. The switch cannot join the fabric until you manually re-register the switch.

The switch reboots after the secure erase operation completes. To connect to the switch, you must use the terminal console because the IP address is not reachable.

Securely Erase user data from a module of a Cisco Nexus 9000 modular switch line card using the GUI

Before you begin

Use this procedure to securely erase user data from a module of a Cisco Nexus 9000 modular switch line card using the GUI.

Procedure

Step 1

On the menu bar, choose Fabric > Inventory.

Step 2

In the Navigation pane, choose pod_id > node_id > Chassis > Line Modules > slot_id.

Step 3

Right-click the slot ID and choose Disable.

Step 4

In the Disable dialog, click Secure Erase.

The process securely erases the data from the module of the switch and removes its configuration from the Cisco Application Policy Infrastructure Controller (APIC). The decommission process takes from 30 minutes to 2 hours, depending on the switch and SSD type, and leaves the module powered on. The NX-OS image from the bootflash remains unaffected.

What to do next

To connect to the switch, you must use the terminal console because the IP address is not reachable.

Securely Erase user data from a Cisco Nexus 9000 switch using the switch's CLI

Before you begin

Before you perform the secure erase operation using the CLI, decommission the switch or physically disconnect the switch from the fabric. If you do not decommission the switch or physically disconnect the switch from the fabric, the Cisco APICpushes configuration back to the switch after process.

Use this procedure to securely erase user data from a Cisco Nexus 9000 switch using the switch's CLI. You cannot use the Cisco Application Policy Infrastructure Controller (APIC)'s CLI for this procedure.

Procedure

Step 1

Log in to the switch’s CLI.

Step 2

Enter the virtual shell on your switch.

leaf1# vsh

Step 3

Disable the session timeout on your terminal.

leaf1# terminal session-timeout 0

If you do not disable the timeout, the VSH session can time out and exit before the secure erase completes and can provide status.

Step 4

Reset the switch to the factory settings to securely erase your data from the switch.

leaf1# factory-reset [preserve-image] [module module_number]

- preserve-image: Specify this flag to retain the NX-OS image in the bootflash of the switch. If you do not specify this flag, the NX-OS image will also be erased and the switch will boot to the loader prompt.

- module module_number: For modular switch line cards and fabric modules, you must specify the number of the module on which to perform the secure erase.

Depending on the switch and SSD type, the decommission process for nonmodular switches takes from 2 to 8 hours. The process securely erases the switch and removes the switch configuration from the Cisco Application Policy Infrastructure Controller (APIC). The secure erase process does not remove the NX-OS image from the bootflash. The switch cannot join the fabric until you manually register it again.

The switch reboots after the secure erase operation completes. To connect to the switch, you must use the terminal console because the IP address is not reachable.

For modular switch line cards or fabric modules, the decommission process takes from 30 minutes to 2 hours, depending on the switch and SSD type. The process securely erases the data from the module of the switch and removes the module's configuration from the Cisco APIC. The process does not remove the NX-OS image from the bootflash.

After the secure erase operation completes, the module will be in the powered-down state. To connect to the switch, you must use the terminal console because the IP address is not reachable.

Chapter 5. Cisco APIC Cluster Management

- APIC clusters

- Cisco Fabric Controller cluster expansion

- Cisco APIC Cluster contractions

- Best practices for cluster management

- Expand the Fabric Controller cluster using the GUI

- Expand the APIC cluster using the Add Node option

- Contracting the APIC Cluster Using the GUI

- Contract APIC nodes using the Delete Node option

- APIC migration types

- Manage APIC cluster at boot using the GUI

- Nexus Dashboard cluster from APIC GUI

APIC clusters

An APIC cluster is a deployment of Cisco Application Policy Infrastructure Controller (APIC) appliances that:

- consists of a minimum of three controllers to provide control of the Cisco ACI fabric,

- scales in size according to transaction-rate requirements and the overall ACI deployment size, and

- allows any controller in the cluster to service any user operation. Controllers can be transparently added or removed from the cluster.

This section provides guidelines and examples related to expanding, contracting, and recovering APIC clusters. Refer to these resources for procedures and best practices in managing cluster size and resilience.

Cisco Fabric Controller cluster expansion

Expanding the Cisco Fabric Controller clusters is an administrative operation that

- increases the cluster size from N to N+1 within approved limits,

- requires the operator to set the desired administrative cluster size and connect controllers with the correct cluster IDs, and

- enables the cluster to automatically perform the expansion.

Expanding clusters accommodates network growth and helps maintain redundancy and performance by including additional controllers as the needs of the organization change.

Expanding the Cisco APIC cluster is the operation to increase any size mismatches, from a cluster size of N to size N+1, within the boundaries. The operator sets the administrative cluster size and connects the APICs with the appropriate cluster IDs, and the cluster performs the expansion.

During cluster expansion, regardless of in which order you physically connect the APIC controllers, the discovery and expansion takes place sequentially based on the APIC ID numbers. For example, APIC 2 is discovered after APIC 1, and APIC 3 is discovered after APIC 2 and so on until you add all the desired APICs to the cluster. As each sequential APIC is discovered, a single data path or multiple data paths are established, and all the switches along the path join the fabric. The expansion process continues until the operational cluster size reaches the equivalent of the administrative cluster size.

Cisco APIC Cluster contractions

A cluster contraction is a cluster management operation that

- reduces a cluster’s size by removing one node at a time within legal boundaries,

- increases computational and memory load on the remaining nodes, and

- makes the decommissioned cluster slot unavailable until re-enabled by operator input.

When contracting the cluster, always decommission the last APIC in the sequence first and proceed in reverse order. For example:

- Decommission APIC4 before APIC3.

- Decommission APIC3 before APIC2.

Best practices for cluster management

- Always verify the health of all controllers before making any changes to the cluster. Confirm that every controller is fully fit and resolve any health issues before proceeding.

- Ensure that all controllers in the cluster run the same firmware version before adding, configuring, or clustering devices. Do not cluster controllers running different firmware versions.

- Maintain at least three active controllers in your cluster, and add standby controllers as needed. For scalability requirements, consult the Verified Scalability Guide to determine the required number of active controllers for your deployment.

- Ignore cluster information from controllers that are not currently active in the cluster, as their data may be inaccurate.

- Know that once you configure a cluster slot with a controller’s ChassisID, you must decommission that controller to make the slot available for reassignment.

- Wait for all ongoing firmware upgrades to complete and verify the cluster is fully fit before making additional changes.

- When moving a controller, always ensure the cluster is healthy. Select the controller you intend to move, shut it down, physically move and reconnect it, and then power it on. After the move, verify through the management interface that all controllers return to a fully fit state.

- Move only one controller at a time to maintain cluster stability.

- When transferring a controller to a different set of leaf switches or to a different port within the same leaf switch, ensure the cluster is healthy first. Decommission the controller before moving it, and then recommission it after the move.

- Before configuring the cluster, confirm that all controllers run the same firmware version to prevent unsupported operations and cluster issues.

- Delete any unused OOB EPGs associated with a controller. Assigning multiple EPGs to a controller is not supported and can cause the cluster workflow IP address to be overridden by policy.

- Remember that log record objects are stored only in one shard on a single controller. If you decommission or replace that controller, those logs are permanently lost.

- When decommissioning a controller, be aware that all fault, event, and audit log history stored on it is deleted. If you replace all controllers, all log history is lost. Before migrating a controller, manually back up its log history to prevent data loss.

- Fabric Controller cluster size expansion

- Cluster size reductions

- Controller replacements in the cluster

Fabric Controller cluster size expansion

A Fabric Controller cluster size is a configuration parameter that

- defines the total number of controllers operating together within a Fabric Controller cluster,

- determines the level of redundancy and fault tolerance achievable by the cluster, and

- influences the cluster’s ability to scale and handle increasing workloads.

Follow these guidelines to expand the APIC cluster size:

- Schedule the cluster expansion at a time that ensures the fabric workload is not impacted.

- If the health status of one or more APIC controllers in the cluster is not "fully fit," remedy the situation before proceeding.

- Stage the new APIC controllers according to the instructions in their hardware installation guide. Verify in-band connectivity using a ping test.

- Increase the cluster target size to match the sum of the existing and new controller counts. For example, if the existing controller count is three and you are adding three controllers, set the new cluster target size to six. The cluster adds each new controller sequentially until expansion is complete. If an existing controller becomes unavailable during the expansion process, cluster expansion will stop. Address the issue before continuing with cluster expansion.

- The expansion may require more than ten minutes per appliance because the controllers must synchronize data when a new appliance is added. After the cluster successfully expands, the operational size and the target size will be equal. Allow the controllers to complete the cluster expansion before making additional changes to the cluster.

Cluster size reductions

A cluster size reduction is a system management operation that

- decreases the number of controllers in a cluster,

- requires the orderly decommissioning and removal of selected controllers, and

- triggers cluster synchronization processes to maintain system stability.

Follow these guidelines to reduce the Cisco Application Policy Infrastructure Controller (APIC) cluster size and decommission the Cisco APICs that are removed from the cluster:

Note

Failure to follow an orderly process to decommission and power down Cisco APICs from a reduced cluster can lead to unpredictable outcomes. Do not allow unrecognized Cisco APICs to remain connected to the fabric.

- Reducing the cluster size increases the load on the remaining Cisco APICs. Schedule the Cisco APIC size reduction at a time when the demands of the fabric workload will not be impacted by the cluster synchronization.

- If one or more of the Cisco APICs' health status in the cluster is not "fully fit," remedy that situation before proceeding.

- Reduce the cluster target size to the new lower value. For example if the existing cluster size is 6 and you will remove 3 controllers, reduce the cluster target size to 3.

- Starting with the highest numbered controller ID in the existing cluster, decommission, power down, and disconnect the Cisco APIC one by one until the cluster reaches the new lower target size.

Upon the decommissioning and removal of each controller, the Cisco APIC synchronizes the cluster.

Note

After decommissioning a Cisco APIC from the cluster, promptly power it down and disconnect it from the fabric to prevent its rediscovery. Before returning it to service, do a wiped clean back to factory reset.

If the disconnection is delayed and a decommissioned controller is rediscovered, follow these steps to remove it:

1. Power down the Cisco APIC and disconnect it from the fabric.

2. In the list of Unauthorized Controllers, reject the controller.

3. Erase the controller from the GUI.

- Cluster synchronization stops if an existing Cisco APIC becomes unavailable. Resolve this issue before attempting to proceed with the cluster synchronization.

- Depending on the amount of data the Cisco APIC must synchronize upon the removal of a controller, the time required to decommission and complete cluster synchronization for each controller could be more than 10 minutes per controller.

Example:

If a cluster originally contains six controllers and three are to be removed, administrators should set the cluster target size to three. Remove controllers one at a time, starting with the highest numbered controller ID, and follow established procedures to ensure reduction and synchronization are successful.

Note

Complete the entire necessary decommissioning steps, allowing the Cisco APIC to complete the cluster synchronization accordingly before making additional changes to the cluster.

Controller replacements in the cluster

A controller replacement is a cluster maintenance operation that

- substitutes a failed or decommissioned controller with a new or spare unit,

- requires using the same initial provisioning parameters and software image as the controller being replaced, and

- maintains cluster synchronization and operational continuity when performed according to established procedures.

Additional reference information

When replacing a Cisco controller in a cluster, observe the following guidelines to ensure a safe and successful process:

- Verify that all controllers have a Fully Fit health status before beginning the replacement.

- Schedule the Cisco APIC controller replacement at a time when the demands of the fabric workload will not be impacted by the cluster synchronization.

- Make note of the initial provisioning parameters and image used on the Cisco APIC controller that will be replaced. The same parameters and image must be used with the replacement controller. The Cisco APIC proceeds to synchronize the replacement controller with the cluster.

Note

Cluster synchronization stops if an existing Cisco APIC controller becomes unavailable. Resolve this issue before attempting to proceed with the cluster synchronization.

- You must choose a Cisco APIC controller that is within the cluster and not the controller that is being decommissioned. For example: Log in to Cisco APIC1 or APIC2 to invoke the shutdown of APIC3 and decommission APIC3.

- CIMC policy configuration: Delete the CIMC policy for the standby and active APIC when replacing the standby APIC. If you do not delete the CIMC policy, ensure to update the CIMC policy for the active APIC after the replacement of the standby APIC is complete.

- Perform the replacement procedure in the following order:

1. Make note of the configuration parameters and image of the APIC being replaced.

2. Decommission the APIC you want to replace (see Decommission a Cisco APIC in the cluster using the GUI)

3. Commission the replacement APIC using the same configuration and image of the APIC being replaced (see Commission a Cisco APIC in the cluster using the GUI)

- Stage the replacement Cisco APIC controller according to the instructions in its hardware installation guide. Verify in-band connectivity with a PING test.

Note

Failure to decommission Cisco APIC controllers before attempting their replacement will preclude the cluster from absorbing the replacement controllers. Also, before returning a decommissioned Cisco APIC controller to service, do a wiped clean back to factory reset.

- Depending on the amount of data the Cisco APIC must synchronize upon the replacement of a controller, the time required to complete the replacement could be more than 10 minutes per replacement controller. Upon successful synchronization of the replacement controller with the cluster, the Cisco APIC operational size and the target size will remain unchanged.

Note

Allow the Cisco APIC to complete the cluster synchronization before making additional changes to the cluster.

- The UUID and fabric domain name persist in a Cisco APIC controller across reboots. However, a clean back-to-factory reboot removes this information. If a Cisco APIC controller is to be moved from one fabric to another, a clean back-to-factory reboot must be done before attempting to add such an controller to a different Cisco ACI fabric.

Expand the Fabric Controller cluster using the GUI

Use this procedure to add one or more APICs to an existing Fabric Controller cluster using the GUI. This process applies to software releases prior to Cisco APIC release 6.0(2). If you are using release 6.0(2) or later, use the Add Node option as described in the relevant procedure.

Before you begin

Set up any new APICs to be added to the cluster. For details, refer to the Set Up Cisco ACI.

Follow these steps to expand the Fabric Controller cluster using the GUI:

Procedure

Step 1

On the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_name > Cluster as Seen by Node.

For apic_name, you must choose a Cisco APIC that is within the cluster that you wish to expand.

The Cluster as Seen by Node window appears in the Work pane with the APIC Cluster and Standby APIC tabs. In the APIC Cluster tab, the controller details appear. This includes the current cluster target and current sizes, the administrative, operational, and health states of each controller in the cluster.

Step 3

Verify that the health state of the cluster is Fully Fit before you proceed with contracting the cluster.

Step 4

In the Work pane, click Actions > Change Cluster Size.

Step 5

In the Change Cluster Size dialog box, in the Target Cluster Administrative Size field, choose the target number to which you want to expand the cluster. Click Submit.

Note

You cannot have a cluster size of two Cisco APICs. You can have a cluster of one, three, or more Cisco APICs.

Step 6

In the Confirmation dialog box, click Yes.

In the Work pane, under Properties, the Target Size field must display your target cluster size.

Step 7

Physically connect all the Cisco APICs that are being added to the cluster.

In the Work pane, in the Cluster > Controllers area, the Cisco APICs are added one by one and displayed in the sequential order starting with N + 1 and continuing until the target cluster size is achieved.

Step 8

Verify that the Cisco APICs are in operational state, and the health state of each controller is Fully Fit.

The Fabric Controller cluster expands to the specified target size. All APICs are added, operational, and healthy.

What to do next

Monitor the cluster and ensure all controllers remain in the Fully Fit health state.

Optionally, update documentation with the new cluster configuration.

Expand the APIC cluster using the Add Node option

Use this procedure on an existing Cisco Application Policy Infrastructure Controller (APIC) cluster to expand the cluster using the Add Node option, which was introduced in Cisco APIC release 6.0(2). To expand a cluster in Cisco APIC releases prior to 6.0(2), see the previous procedure.

The Add Node option is a simpler and direct method to add a Cisco APIC to a cluster.

Before you begin

- Ensure the node to-be-added is a clean node or is in factory-reset state.

- Check the current Cluster Size in the General pane. If it is N, after successful node addition, the size will be N+1.

Procedure

Step 1

On the menu bar, choose System > Controllers. In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

Step 2

In the Active Controllers pane, click the Actions button and select the Add Node option.

The Add Node screen is displayed.

Step 3

Enter the following details in the Add Node screen:

Select the Controller Type. Based on your selection, proceed to the relevant substep.

Put a check in the Enabled box if you need to support IPv6 addresses.

1. When the Controller Type is Physical:

- CIMC details pane

- IP Address: Enter the CIMC IP address.

- Username: Enter the username to access CIMC.

- Password: Enter the password to access CIMC.

- Click Validate. Validation success is displayed on successful authentication.

This pane appears only if you configured CIMC. If you did not configure CIMC, instead perform the physical APIC login step of the Bring up the APIC cluster using the GUI procedure (step 1b) on the new node to configure out-of-band management.

- General pane

- Name: Enter a name for the controller.

- Admin Password: Enter the admin password for the controller.

- Controller ID: This is auto-populated based on the existing cluster size. If the current cluster size is N, the controller ID is displayed as N+1.

- Serial Number: This is auto-populated after CIMC validation.

- Force Add: Put a check in the Enabled box to add a Cisco APIC that has a release earlier than 6.0(2).

- Out of Band Network pane

- IPv4 Address: The address is auto-populated.

- IPv4 Gateway: The gateway address is auto-populated.

Note

If you put a check in the Enabled box for IPv6 earlier, enter the IPv6 address and gateway.

- Infra Network pane

- IPv4 Address: Enter the infra network IP address.

- IPv4 Gateway: Enter the infra network IP address of the gateway.

- VLAN: Enter a VLAN ID.

2. When the Controller Type is Virtual:

- Management IP pane

- IP Address: Enter the management IP address.

Note

The management IP addresses are defined during the deployment of the virtual machines using ESXi/AWS.

- Enter the username for the virtual APIC.

- Enter the password for the virtual APIC.

- Click Validate. Validation success is displayed on successful authentication.

- General pane

- Name: User-defined name for the controller.

- Controller ID: This is auto-populated based on the existing cluster size. If the current cluster size is N, the controller ID is displayed as N+1.

- Serial Number: The serial number of the virtual machine is auto-populated.

- Force Add: Put a check in the Enabled box to add a Cisco APIC that has a release earlier than 6.0(2).

- Out of Band Network pane

- IPv4 Address: The IP address is auto-populated.

- IPv4 Gateway: The gateway IP address is auto-populated.

Note

If you put a check in the Enabled box for IPv6 earlier, enter the IPv6 address and gateway.

- Infra Network pane

- IPv4 Address: Enter the infra network address.

- IPv4 Gateway: Enter the IP address of the gateway.

- VLAN: (Applicable only for remotely attached virtual APIC- ESXi) Enter the interface VLAN ID to be used.

Note

The Infra L3 Network pane is not displayed when the virtual APIC is deployed using AWS.

Step 4

Click Apply.

What to do next

The newly added controller appears in the Unauthorized Controllers pane. Wait for a few minutes for the latest controller to appear with the other controllers of the cluster, under the Active Controllers pane.

Also, check the Current Size and the Target Size in the General pane. The number displayed is updated with the latest node addition.

Contracting the APIC Cluster Using the GUI

Perform this task for releases prior to Cisco APIC release 6.0(2). For release 6.0(2) and later, use the Delete Node option described in the subsequent procedure.

Follow these steps to contract the Fabric Controller cluster:

Before you begin

Ensure the cluster health state is Fully Fit.

Identify which controllers should remain in the cluster.

Procedure

Step 1

On the menu bar, choose System > Controllers. In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

You must choose an apic_name that is within the cluster and not the controller that is being decommissioned.

The Cluster as Seen by Node window appears in the Work pane with the APIC Cluster and Standby APIC tabs. In the APIC Cluster tab, the controller details appear. This includes the current cluster target and current sizes, the administrative, operational, and health states of each controller in the cluster.

Step 2

Verify that the health state of the cluster is Fully Fit before you proceed with contracting the cluster.

Step 3

In the Work pane, click Actions > Change Cluster Size.

Step 4

In the Change Cluster Size dialog box, in the Target Cluster Administrative Size field, choose the target number to which you want to contract the cluster. Click Submit.

Note

It is not acceptable to have a cluster size of two APICs. A cluster of one, three, or more APICs is acceptable.

Step 5

From the Active Controllers area of the Work pane, choose the APIC that is last in the cluster.

Example:

In a cluster of three, the last in the cluster is three as identified by the controller ID.

Step 6

Right-click on the controller you want to decommission and choose Decommission. When the Confirmation dialog box displays, click Yes.

The decommissioned controller displays Unregistered in the Operational State column. The controller is then taken out of service and not visible in the Work pane any longer.

Step 7

Repeat the earlier step to decommission the controllers one by one for all the APICs in the cluster in the appropriate order of highest controller ID number to the lowest.

Note

The operation cluster size shrinks only after the last appliance is decommissioned, and not after the administrative size is changed. Verify after each controller is decommissioned that the operational state of the controller is unregistered, and the controller is no longer in service in the cluster.

You should be left with the remaining controllers in the APIC cluster that you desire.

What to do next

- After deleting an APIC from the cluster, power the controller down and disconnect it from the fabric.

- Wait for a few minutes, and confirm that the Health State of the remaining nodes of the cluster is displayed as Fully fit before further action.

Contract APIC nodes using the Delete Node option

Use this procedure to contract a cluster using the Delete Node option which has been introduced in Cisco APIC release 6.0(2). To contract a cluster in APIC releases prior to 6.0(2), see the previous procedure.

The Delete Node option includes two operations— reduces the cluster size, and decommissions the node.

Note

A two-node cluster is not supported. You cannot delete one node from a three-node cluster. The minimum recommended cluster size is three.

Note

Starting from Cisco APIC 6.1(2), you can delete a standby node from a cluster. Once you delete a standby node, you must perform a clean reboot before you add it back to the cluster.

Before you begin

- Ensure the cluster contains at least three nodes; a two-node cluster is not supported.

- You cannot delete a node from a three-node cluster.

- Delete nodes in decreasing order by Node ID (for example, delete Node ID 6 before Node ID 5).

- If you delete a standby node, which is available starting from APIC 6.1(2), perform a clean reboot before adding the node back to the cluster.

Procedure

Step 1

On the menu bar, choose System > Controllers. In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

Step 2

In the Active Controllers pane, select the controller which you want to delete by selecting the required check-box..

Step 3

Click the Actions button and select the Delete Node option.

Step 4

Click OK on the pop-up screen to confirm the deletion.

Selecting the force option has no effect. It is a no operation option, as it is not supported on Cisco APIC release 6.0(2).

Note

You need to delete the nodes in decreasing order, that is, for example, you can not delete a node with ID 5 before deleting the node with ID 6.

Check the Current Size and Target Size in the General pane. The size indicated will be one lesser than it was before. If the earlier cluster size was N, now it will be N-1.

Note

If you are deleting more than node from the cluster, the last node of the cluster is deleted first, followed by the other nodes. Shrink In Progress in the General pane is set to Yes until all the selected nodes are deleted.

The selected nodes are removed from the cluster, and the remaining nodes display the updated cluster size.

What to do next

- After deleting an APIC from the cluster, power the controller down and disconnect it from the fabric.

- Wait for a few minutes, and confirm that the Health State of the remaining nodes of the cluster is displayed as Fully fit before further action.

Commission a Cisco APIC in the cluster using the GUI

You typically commission a Cisco APIC when adding a new or previously decommissioned APIC to your cluster. This procedure applies to Cisco APIC releases before Cisco APIC release 6.0(2). For newer releases, refer to the subsequent section for the updated commissioning workflow.

Follow these steps to commission a Cisco APIC in the cluster:

Before you begin

- Ensure you have an APIC controller in decommissioned state and physical access if required.

- Confirm you have Administrator access to the Cisco APIC GUI.

Procedure

Step 1

From the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

The Cluster as Seen by Node window appears in the Work pane with the APIC Cluster and Standby APIC tabs. In the APIC Cluster tab, the controller details appear. This includes the current cluster target and current sizes, the administrative, operational, and health states of each controller in the cluster.

Step 3

From the APIC Cluster tab of the Work pane, verify in the Active Controllers summary table that the cluster Health State is Fully Fit before continuing.

Step 4

From the Work pane, right-click the decommissioned controller that is displaying Unregistered in the Operational State column and choose Commission.

The controller is highlighted.

Step 5

In the Confirmation dialog box, click Yes.

Step 6

Verify that the commissioned Cisco APIC is in the operational state and the health state is Fully Fit.

The Cisco APIC is commissioned and actively participates in the cluster with a healthy operational state.

What to do next

Monitor the cluster to ensure it maintains a healthy state. If the controller does not reach 'Active' and 'Fully Fit' status, review the configuration and attempt commissioning again.

Commissioning a Cisco APIC in the cluster

Use this procedure when you need to commission an APIC in an existing cluster running release 6.0(2) or later. This may be performed for normal provisioning or as part of an RMA (Return Material Authorization) workflow.

Follow these steps to commission a Cisco APIC in the cluster:

Before you begin

- Verify that the APIC is connected to the network and powered on.

- Obtain required admin credentials for the APIC cluster.

Procedure

Step 1

From the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

Step 3

Select a decommissioned Cisco APIC from the Active Controllers table.

Step 4

In the Active Controllers table, click the Actions icon (three dots), which is displayed at the end of the row for each Cisco APIC. From the displayed options, click Commission.

The Commission dialog box is displayed.

Step 5

Enter the following details in the Commission screen:

Choose the Controller Type. Based on your choice, proceed to the relevant substep.

Put a check in the Enabled check box if you need to support IPv6 addresses.

1. When the Controller Type is Physical:

- CIMC details pane

- IP Address: Enter the CIMC IP address.

- Username: Enter the username to access CIMC.

- Password: Enter the password to access CIMC.

- Click Validate. Validation success is displayed on successful authentication.

This pane appears only if you configured CIMC. If you did not configure CIMC, instead perform the physical APIC login step of the Bring up the APIC cluster using the GUI procedure (step 1b) on the new node to configure out-of-band management.

- General pane

- Name: The name of the controller. The name is entered automatically after the CIMC validation.

- Admin Password: Enter the admin password for the controller.

- Controller ID: This is auto-populated based on the Cisco APIC that was decommissioned. The ID of the decommissioned node is assigned.

- Serial Number: This is auto-populated after CIMC validation.

- Pod ID: Enter the ID number of the pod for the Cisco APIC.

- Out of Band Network pane

- IPv4 Address: Enter the IPv4 address of the out-of-band network.

- IPv4 Gateway: Enter the IPv4 gateway address of the out-of-band network.

Note

If you have selected the Enabled check box for IPv6 earlier, enter the IPv6 address and gateway.

2. When the Controller Type is Virtual :

- Virtual Instance: Enter the management IP and click Validate.

Note

The management IP addresses are defined during the deployment of the VMs using ESXi/ AWS.

- General pane

- Name: A user-defined name for the controller.

- Controller ID: This is auto-populated based on the Cisco APIC that was decommissioned. The ID of the decommissioned node is assigned.

- Serial Number: The serial number of the VM is auto-populated.

- Out of Band Network pane

- IPv4 Address: The IP address is auto-populated.

- IPv4 Gateway: The gateway IP address is auto-populated.

Note

If you have selected the Enabled check box for IPv6 earlier, enter the IPv6 address and gateway.

- Infra Network pane

- IPv4 Address: Enter the infra network address.

- IPv4 Gateway: Enter the IP address of the gateway.

- VLAN: (applicable only for remotely attached virtual APIC- ESXi) enter the interface VLAN ID to be used.

Note

The Infra L3 Network pane is not displayed when the virtual APIC is deployed using AWS.

Step 6

Click Apply.

Step 7

Verify that the commissioned Cisco APIC is in the operational state and the health state is Fully Fit.

The APIC is successfully commissioned and ready for role in the controller cluster.

What to do next

Review APIC status and connectivity. Update your records or monitoring tools as needed.

Decommission a Cisco APIC in the cluster using the GUI

Remove a Cisco APIC controller from the cluster using the graphical user interface (GUI) for APIC releases prior to 6.0(2).

Note

Log record objects are stored only on one Cisco APIC. Decommissioning or replacing that APIC permanently deletes its associated log record objects.

Follow these steps to decommission a Cisco APIC in the cluster:

Before you begin

- Make sure you are running a Cisco APIC release prior to 6.0(2).

- Ensure you are not logged in to the controller that you plan to decommission.

Procedure

Step 1

Go to System > Controllers in the main menu.

Step 2

Select an APIC node that is in the cluster (but not the one you want to decommission), and expand Cluster as Seen by Node.

Step 3

In the APIC Cluster tab, confirm that the Health State for Active Controllers is Fully Fit.

Step 4

In the Active Controllers table, right-click the controller you want to remove and select Decommission.

Step 5

In the confirmation dialog box, click Yes.

The controller’s operational state shows Unregistered. It is removed from cluster service and is no longer visible.

What to do next

- Power down and disconnect the decommissioned controller from the fabric.

- Before returning the controller to service, perform a factory reset.

- Cluster size only decreases after all appliances are decommissioned.

- For Layer 4 to Layer 7 services, reboot the controller before re-commissioning it.

Decommission a Cisco APIC in the cluster

Use this procedure for Cisco APIC release 6.0(2) and later versions. Refer to the Decommission a Cisco APIC in the cluster for releases prior to release 6.0(2).

Note

Any log record objects held solely on the decommissioned controller are lost permanently after this operation.

Follow these steps to decommission a Cisco APIC in the cluster:

Before you begin

- Ensure you are not logged into the controller you plan to decommission.

- Any log record objects stored only on the decommissioned APIC’s database shard are permanently lost

- Confirm that the controller is running release version 6.0(2).

- Confirm the cluster health state is Fully Fit.

Procedure

Step 1

Go to System > Controllers.

Step 2

In the Controllers list, select an APIC within the cluster (not the one to be decommissioned) and access Cluster as Seen by Node.

Step 3

Verify the cluster health state is Fully Fit in the Active Controllers summary.

Step 4

In the Active Controllers table, for the controller you wish to remove, choose the Actions menu and select Decommission.

Step 5

Confirm the decommission action.

After the controller is decommissioned, its operational state changes to Unregistered, and it is removed from cluster management.

Step 6

Starting with Cisco ACI release 6.2(1), APIC clean reload is no longer supported. In addition, the acidiag touch clean command will be deprecated. Use the steps below to add an APIC back into the cluster.

1. Decommission the APIC.

2. Use the acidiag touch setup command and the acidiag reboot command to reboot the APIC.

3. From the existing cluster, perform the recommission workflow to recommission the APIC back into the cluster.

A clean reload of the entire cluster is not supported. To recover the cluster, run acidiag reboot command to reboot each APIC individually. Afterward, perform ID recovery.

For Cisco ACI Release 6.2(1) or later, contact Cisco TAC to recover the Cisco Application Policy Infrastructure Controller (APIC) password. You cannot recover the password on your own.

The selected APIC controller is removed from the cluster and marked as unregistered.

What to do next

- Power down the decommissioned controller and disconnect it from the fabric.

- Perform a factory reset before repurposing or recommissioning the controller.

- For Layer 4 to Layer 7 services, reboot the controller before recommissioning.

- If you decommission all controllers, verify the cluster size updates accordingly.

Shut down all APICs in a cluster

Use this procedure to power down all APIC controllers in a cluster. Before beginning, verify that the cluster is healthy. Do not make configuration changes during the shutdown process.

Follow these steps to shut down all APICs in a cluster:

Before you begin

- Verify the APIC cluster is in a healthy state and all APICs are fully fit.

- Ensure no configuration changes are made during this procedure.

Procedure

Step 1

Log in to Cisco APIC with appliance ID 1.

Step 2

On the menu bar, choose System > Controllers.

Step 3

In the Navigation pane, expand Controllers > apic_controller_name.

You must select the third APIC in the cluster.

Step 4

Right-click the controller and click Shutdown.

Step 5

Repeat the steps to shutdown the second APIC in the cluster.

Step 6

Log in to Cisco IMC of the first APIC in the cluster to shutdown the APIC.

Step 7

Choose Server > Server Summary > Shutdown Server.

You have now shutdown all the three APICs in a cluster.

All APICs in the cluster are safely shut down.

What to do next

Before performing maintenance or shutting down related infrastructure, verify that all APICs are powered off.

Bring back the APICs in a cluster

This procedure ensures all APICs are powered on and ready for configuration changes.

Follow these steps to bring back the APICs in a cluster:

Before you begin

- Confirm hardware checks are complete.

- Ensure you have access credentials for Cisco IMC for each APIC.

Procedure

Step 1

Log in to Cisco IMC of the first APIC in the cluster.

Step 2

Choose Server > Server Summary > Power On to power on the first APIC.

Step 3

Repeat the steps to power on the second APIC and then the third APIC in the cluster.

After all the APICs are powered on, ensure that all the APICs are in a fully fit state. Only after verifying that the APICs are in a fully fit state, you must make any configuration changes on the APIC.

All APICs in the cluster are powered on and in a fully fit state. You can now make configuration changes on the APICs.

Cold standby clusters

A cold standby cluster is a high availability deployment that

- operates with some controllers as active and others as standby,

- enables standby controllers to quickly take over when an active controller fails, and

- allows administrators to manually initiate a switchover to maintain cluster services.

Additional information

In a Cisco APIC cluster, administrators typically configure at least three active controllers and one or more standby controllers for resilience. The active controllers manage traffic under normal conditions, while standby controllers remain ready to replace any active controller if needed. The switch to a standby controller is performed manually to ensure controlled recovery during failures. This arrangement maintains network stability and reduces downtime.

Guidelines for standby Cisco APICs

These guidelines apply to standby Cisco Application Policy Infrastructure Controllers (APICs) in a cluster:

- Prior to Cisco APIC release 6.1(3), only clusters with physical APIC nodes supported standby APICs. Beginning with release 6.1(3), both physical and virtual APIC nodes support standby APICs. Standby APICs must be the same form factor (physical or virtual) as the active APICs in the cluster.

- A minimum of three active APICs is required before adding a standby APIC.

- The standby APIC must be running the same firmware version as the cluster when it joins during initial setup.

- After all active APICs are upgraded during a software upgrade process, the standby APICs are upgraded automatically.

- During initial setup, IDs are assigned to standby APICs. If a standby APIC is promoted to active status, it adopts the ID of the APIC it replaces.

- Administrative login is not enabled on standby APICs. To troubleshoot a standby APIC, log in using SSH as the `rescue-user`.

- During a switchover, the replaced active APIC must be powered down to prevent connectivity issues.

- Switchover may fail in these cases.

- There is no connectivity to the standby APIC.

- The standby APIC firmware version does not match the active cluster.

- After switching over a standby APIC to active, you can set up another standby APIC if necessary.

- If you check the option 'Retain OOB IP address for Standby (new active),' the standby APIC promoted to active retains its original out-of-band (OOB) management IP address.

- If only one active APIC is down, the standby (new active) APIC uses the old active APIC's OOB management IP address.

- If more than one active APIC is down, the standby (new active) APIC attempts to use the old active APIC’s OOB management IP address. However, this may fail if the IP configuration data is in the minority state.

- For Cisco ACI Multi-Pod deployments, if the old active and standby APICs use different OOB management IP subnets, be sure to check the option to retain the original standby OOB management IP. Otherwise, OOB connectivity may be lost. If connectivity is lost due to this or if multiple active APICs are down, create a new static node management OOB IP and make sure your cluster is not in the minority state.

- The standby APIC does not participate in policy configuration or management.

- The system does not replicate any information, including administrator credentials, to standby APICs.

- A standby APIC does not retain its in-band management IP address when promoted to active. You must manually reconfigure the in-band management IP address on the newly active APIC.

Verify cold standby status using the GUI

Use this task to view and validate the status of standby controllers in your system.

Follow these steps:

1. On the menu bar, choose System > Controllers.

2. In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

3. In the Work pane, the standby controllers are displayed under Standby Controllers.

The GUI displays the standby controllers, confirming their current status.

Switch over an active APIC with a standby APIC

Use this procedure to switch over an active APIC with a standby APIC.

Before you begin

- Confirm that you have access rights to the APIC system.

- Identify the active controller you want to replace and its standby counterpart.

- Verify that the cluster's health status is normal.

Procedure

Step 1

On the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

The apic_controller_name should be other than the name of the controller that you are replacing.

Step 3

In the Work pane, verify that the Health State in the Active Controllers summary table indicates the active controllers other than the one being replaced are Fully Fit before continuing.

Step 4

Click an apic_controller_name that you want to switch over.

Step 5

In the Work pane, click ... in the row of the contoller that you are replacing, then choose Replace.

The Replace dialog box displays.

Step 6

Choose the Backup Controller from the drop-down list and click Submit.

It may take several minutes to switch over an active APIC with a standby APIC and for the system to be registered as active.

Step 7

Verify the progress of the switch over in the Failover Status field in the Active Controllers summary table.

Note

We recommend that you use a standby APIC in the same pod to replace an active APIC because each pod might use a different out of band management IP subnet.

If you can’t use the recommended approach (for example, if active APIC (ID:2) in Pod1 is replaced by standby APIC (ID:21) in Pod2), and the out of band management IP subnets are different between pods, an additional procedure is required to have the standby Cisco APIC (new active) retain its original out of band management IP address after the failover.

- Check the Retain OOB IP address for Standby (new active) box at Step 6.

- After the failover, delete the static Node Management Address configuration for the replaced (old active) Cisco APIC and read the static Node Management Address configuration for the new active (previously standby) Cisco APIC.

The standby controller becomes the new active APIC. The system updates to reflect the change and resumes normal operation.

What to do next

If you replaced controllers between different pods with distinct out-of-band subnets, confirm OOB management configuration as previously advised.

Warm standby for a Cisco APIC cluster

A Warm Standby APIC is a disaster recovery solution for Cisco APIC clusters that

- continuously synchronizes all data from active APIC nodes,

- enables restoration and rebuilding of the APIC cluster even when some or all database shards are lost, and

- allows for faster and more effective replacement of failed APIC nodes compared to Cold Standby APICs.

Starting from Cisco APIC 6.1(2), Standby APIC can be setup as a Warm Standby APIC as opposed to a Cold Standby APIC. Unlike Cold Standby APIC which does not contain any data until it’s promoted to active, Warm Standby APIC constantly synchronizes all data from the active APIC nodes while it’s still in the standby role. This enables you to rebuild the APIC cluster by using the Warm Standby APIC when some or all piece of the database is distributed across the APIC cluster is lost forever. Some such scenarios are explained below.

APIC Cluster uses a database technology called sharding and replica. The data of the ACI fabric is divided into smaller, separate parts called shards and distributed across active APIC nodes. Each shard is replicated up to three replicas regardless of the size of your cluster. For instance, if you have a cluster of 5 APIC nodes, one shard is replicated on APIC 1, 2, and 3 while another shard is replicated on APIC 3, 4, and 5. This means that if you loose three or more APIC nodes in the cluster, data for some shard(s) will be completely lost even if you still have some active APIC nodes. In such a case, the Cold Standby APIC cannot replace those lost APIC nodes because the Cold Standby APIC does not contain any data by itself and cannot restore those lost shards from any of the remaining active APIC nodes. Similarly, if you lost all APIC nodes in the cluster, Cold Standby APIC cannot replace them either regardless of the number of APIC nodes you lost.

For these scenarios, a Warm Standby APIC can be used. Some practical examples of such data loss scenarios are as follows.

Data loss scenario 1:

In a multi-pod deployment where you have APIC 1, 2, and 3 in Pod 1 and APIC 4 and 5 in Pod 2, if Pod 1 goes down because of a disaster, such as flood, fire, earthquake, and so on, three APIC nodes are lost. This means some database shards are completely lost.

Data loss scenario 2:

In a multi-pod deployment with Pod 1 and 2 in the same location while Pod 3 and 4 in another location where you have APIC 1 and 2 in Pod 1, APIC 3 and 4 in Pod 2, APIC 5 and 6 in Pod 3 and APIC 7 in Pod 4, if the location with Pod 1 and 2 has a disaster, four APICs (APIC 1, 2, 3 and 4) are lost. This means some database shards are completely lost.

Data loss scenario 3:

In a multi-pod deployment where you have APIC 1 and 2 in Pod 1, APIC 3 in Pod 2 and no active APIC in Pod 3, if Pod 1 and 2 goes down because of a disaster, all data of the fabric is lost in the cluster as you lose all active APIC nodes.

For these scenarios, if you have a Warm Standby APIC in the healthy pod/site, you can restore the lost data shard and restore the fabric because the Warm Standby APIC had all shards synchronized from all the active APIC nodes while they were still operational. This is not possible with the Cold Standby APIC.

These examples are all multi-pod deployments because it’s unlikely for a single pod deployment to lose more than three APIC nodes or all APIC nodes in the cluster while the standby APIC nodes are intact. Nevertheless, a Warm Standby APIC is supported and functions in the same way for both multi-pod and single pod deployments.

As shown with these examples, the new capability introduced with the Warm Standby APIC is disaster recovery where some or all the database shards are lost and the APIC cluster needs to be re-built in addition to the capability of a faster and easier replacement of an active APIC node which is supported by both Warm and Cold Standby APIC.

When you need to replace one active APIC node with a Warm or Cold Standby APIC node, the replacement operation is triggered from one of the remaining healthy active APIC nodes. However, the promotion of the Warm Standby APIC to rebuild the cluster in the case of data loss, is not performed via the remaining healthy active APIC nodes because there may be no active APIC nodes left. It can be performed via the GUI or REST API on one of the Warm Standby APIC nodes. This always promotes the Warm Standby APIC to APIC 1 such that it can be the starting point of the disaster recovery. See the Recover an APIC cluster using Warm Standby and the GUI section for details.

For Warm Standby APIC to restore the fabric from disastrous events, it is recommended to have one Warm Standby APIC node at a minimum on each failure domain which may be a pod or a geographical site.

- Guidelines for using Warm Standby Cisco APICs

- Change the standby APIC type using the GUI

- Add a standby APIC

- Deleting a Standby from the Cluster

- Recover an APIC cluster using Warm Standby and the GUI

Guidelines for using Warm Standby Cisco APICs

Follow these requirements when configuring and managing Warm Standby Cisco APICs:

- In Cisco APIC 6.1(2), a warm standby APIC supported APIC clusters only on physical APIC nodes. Beginning with Cisco APIC 6.1(3), standby APICs for APIC clusters with virtual nodes are also supported. Standby APICs must be of the same form factor (physical or virtual) as the active APICs in the cluster.

- A Warm Standby APIC is supported for both types of APIC connectivity – directly attached and remotely attached via a L3 network.

- Before Cisco APIC release 6.2(1), you could configure only one type of standby APIC—either cold or warm—per cluster. Cold and warm standby APICs could not coexist in the same cluster, and you had to set or modify the standby type before or after adding standby nodes.

Starting with Cisco APIC release 6.2(1), only the warm standby controller type is supported. Cold standby controllers are deprecated and are not available in this or later releases.

- Limit the number of Warm Standby APIC nodes to a maximum of three per cluster.

- Do not change the standby APIC type to Warm if four or more Cold Standby APIC nodes exist in the cluster.

- In Cisco ACI 6.2(1) and later, upgrading fails if more than three cold standby nodes are present. Reduce the number of standby nodes to three before upgrading; these nodes are converted to warm standby by default during the upgrade.

- Disaster recovery with Warm Standby APIC to rebuild the entire cluster is allowed only when there is data loss in the cluster, in other words only when three or more active APIC nodes are lost and as a result all three replicas of some shard are lost forever.

-

- When three or more active APICs are lost temporarily because of a network issue in the Inter-Pod Network (IPN), you should not promote the Warm Standby APIC node to APIC 1 as you must initialize all other APICs and rebuild the cluster even when the APIC nodes are healthy in each pod.

- You must change the Standby APIC Type of the cluster to Cold before you downgrade to a version older than 6.1(2) that does not support the Warm Standby APIC.

- Prior to Cisco APIC 6.1(2) which only had the Cold Standby APIC, the upgrade (or downgrade) of standby APIC nodes was not visible. You did not have to wait before you proceeded with a switch upgrade. The standby APIC nodes were initialized and booted up with a new version after the APIC upgrade for the active nodes were completed.

Starting from Cisco APIC 6.1(2), the APIC upgrade process may take a little longer than before when there are standby APIC nodes. The upgrade process explicitly includes standby APIC nodes for both Warm and Cold Standby APICs. This is to ensure that the database is backed up in the Warm Standby APIC and is updated to match the new version models. Although, the Cold Standby APIC does not contain any data to be updated, the same process is applied to the Cold Standby APIC, but this process completes much faster than the Warm Standby APIC.

- You can delete a Standby node from the cluster. Refer to Deleting a Standby from the Cluster for more information.

- If any switch is unavailable during disastor recovery due to reboot, interface down, process crash, power failure etc., then Cisco recommends you do a clean reboot and rediscover the switch back into fabric.

- If a Cisco APIC cluster is running in Strict mode on Cisco APIC 6.1(2) or later and the switches are running versions earlier than Cisco APIC 6.1(1), disaster recovery operations fail. The cluster must be switched to Permissive mode before performing disaster recovery.

Change the standby APIC type using the GUI

Complete this procedure to change the standby type on Cisco APIC.

Before you begin

Ensure you have administrator access to the Cisco APIC interface.

Procedure

Step 1

Navigate to the System > System Settings sub menu.

Step 2

In the Fabric Wide Settings Policy page, select Warm or Cold as the Standby Type.

Step 3

Click Submit.

Step 4

To verify the Warm Standby status:

1. On the menu bar, choose System > Controllers.

2. In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

3. In the Work pane, the standby controllers are displayed under Standby Controllers.

4. The Standby Type is displayed in the Cluster As Seen by Node pane.

The selected standby APIC type is changed and the status is confirmed in the controllers view.

Add a standby APIC

Adding a standby APIC allows your Application Policy Infrastructure Controller (APIC) cluster to remain operational if a primary controller fails, ensuring continuous management of your fabric.

Follow these steps to add a standby APIC:

Before you begin

- Make sure you have administrator access to the APIC system.

- Gather CIMC or management credentials and network information for the APIC to be added.

- Confirm your network policies allow communication between controllers.

Procedure

Step 1

On the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_name > Cluster as Seen by Node.

The Cluster as Seen by Node window appears in the Work pane.

Step 3

In the Work pane, click Actions > Add Standby Node.

Step 4

In the Controller Type field, either Physical or Virtual is pre-selected as the same type as the active controllers in the cluster.

Step 5

In the Connectivity Type field, select CMIC or OOB For virtual APICs, OOB is pre-selected as the only option.

Step 6

In the CMIC Details pane or the Management IP pane, enter the following details:

1. IP Address: Enter the CIMC IP address.

2. Username: The username to access the CIMC

3. Password: Enter the password to access CIMC.

Step 7

In the General pane, enter the following details:

1. Name: Enter a name for the controller.

2. Controller ID:Enter a value for the Controller ID. We recommend that you add a value in the range of 21 to 29 for this ID.

3. Pod ID: Enter the pod ID for APIC. The range is from 1 to 128.

4. Serial Number: The serial number is auto-populated (for APICs 1 to N, where N is the cluster size) after CIMC validation.

APIC 1 verifies the reachability of the CIMC IP address and also captures the serial number of the new APIC.

Step 8

In the Out of Band Network pane, enter the following details:

1. IPv4 Address: Enter the IPv4 address.

2. IPv4 Gateway: Enter the IPv4 gateway address.

If you have enabled IPv6 addresses for OOB management, enter the IPv6 address and gateway.

1. IPv6 Address: Enter the IPv6 address.

2. IPv6 Gateway: Enter the IPv6 gateway address.

Step 9

Click Apply.

The standby APIC is added to the cluster and is ready to serve as a backup controller if needed.

What to do next

- Verify the new standby APIC appears in the cluster and monitor its health status.

- Perform any additional configuration as required by your environment.

Deleting a Standby from the Cluster

Complete this procedure to select and delete the warm standby from Cisco APIC.

Before you begin

Before you begin this procedure, see Guidelines and Limitations for Warm Standby on Cisco APICS.

Procedure

Step 1

On the menu bar, choose System > Controllers.

Step 2

In the Navigation pane, expand Controllers > apic_controller_name > Cluster as Seen by Node.

Step 3

In the Controllers pane, select the node and click Actions>Delete Nodes.

Note

Shutdown the node that must be deleted and then delete the node. Once you have deleted the node, it must be disconnected from the fabric. You cannot add the deleted standby node back to the cluster until you have performed a factory reset for this node.

Recover an APIC cluster using Warm Standby and the GUI

As explained in the “Warm Standby” section, one of the use cases of Warm Standby APIC is to rebuild the APIC cluster even when some or all the database information (shard) is lost along with the active APIC nodes. See the Warm standby for a Cisco APIC cluster section for details of the data loss scenarios that need recovery with a Warm Standby APIC.

To rebuild the APIC cluster to restore the fabric from a disastrous event that caused the data loss in the APIC cluster, you can access the GUI or REST API of one of the Warm Standby APIC nodes and follow the procedure shown below.

The procedure in this section will promote the Warm Standby APIC node to APIC 1 using the database information in the standby node itself. Once the Warm Standby APIC node is successfully promoted to APIC 1, initialize the remaining active and/or standby APIC nodes and discover them as new active APIC 2, APIC 3 and so on. As the new APIC nodes are discovered, the data stored in APIC 1, which used to be the Warm Standby APIC node will be distributed to those new nodes as the new replica of each shard.

Note

When the Warm Standby APIC node is promoted to APIC 1, the standby APIC node shuts down the infra interfaces on the remaining active or Standby APICs that are still reachable to ensure that the ACI switches can only see the standby node, soon to be new APIC 1, to avoid any conflict with the remaining active APIC nodes

Complete this procedure to setup the Cisco APIC Disaster Recovery for Cisco APIC.

Before you begin

- Confirm you have administrator credentials for the Warm Standby APIC node.

- Identify which APIC nodes are available and which have been lost.

Procedure

Step 1

Login into one of the Warm Standby APIC by accessing https://<standby APIC OoB IP>. Admin user password is required.

Step 2

Click Promote to start re-building the APIC cluster by promoting the Warm Standby APIC to APIC 1.

Note

If a Cisco APIC cluster does not need disaster recovery, you are redirected to an active APIC UI.

Step 3

The Initiation Progress status is displayed. If successful, the active Cisco APIC is displayed. The GUI will transition to the regular APIC GUI with the former standby node as new APIC 1. You will use this GUI to add a new APIC 2, APIC 3 and so on in the following steps.

Step 4

Initialize the remaining APIC nodes with acidiag touch setup then acidiag reboot via the CLI on each node.

Step 5

Add the initialized APIC nodes as the new APIC 2, APIC 3, and so on via the APIC GUI on the new APIC 1. See Expand the APIC cluster using the Add Node optionfor details.

After rebuilding, the promoted Warm Standby node serves as APIC 1. Additional nodes join as APIC 2, APIC 3, and synchronize data. The fabric is restored.

APIC migration types

APIC migrations are controller transition processes that:

- enable movement between physical and virtual APIC clusters,

- support migration among clusters of different APIC types: physical, virtual, or mixed

- facilitate the migration of one or more standby nodes from a source cluster to a destination cluster.

Beginning with Cisco APIC release 6.1(1), APIC migrations support transitioning from a physical APIC cluster to a virtual APIC cluster deployed on ESXi hosts (using VMware vCenter) and from a virtual APIC cluster (on ESXi hosts) to a physical APIC cluster.

Starting with Cisco APIC release 6.2(1), APIC migrations support transitions between clusters composed of any mix of physical and virtual controllers hosted on VMware ESXi or Nutanix AHV platforms. This release also introduces support for migrating one or more standby nodes from the source cluster to the destination cluster.

- Guidelines and limitations for migrating physical APICs to virtual APICs

- Migrating APIC clusters

- Migrate an APIC cluster between physical and virtual deployments

- Migration status

- Operations in case of a migration failure

- Basic troubleshooting

Guidelines and limitations for migrating physical APICs to virtual APICs

The following guidelines and limitations apply when migrating physical APICs to virtual APICs and vice versa:

Guidelines

- Physical APICs in layer 2 (directly attached to the fabric) can be migrated to layer 2 virtual APICs, and layer 2 virtual APICs can be migrated to layer 2 physical APICs. Physical APICs in layer 3 (remotely attached to the fabric) can be migrated to layer 3 virtual APICs, and virtual APICs in layer 3 can be migrated to layer 3 physical APICs. Migration between layer 2 APICs and layer 3 APICs is not supported.

- Do not initiate the migration process if an upgrade is in progress.

- Do not initiate an upgrade if migration is in progress.

- Update any configuration that uses APIC out-of-band (OOB) management after migration is completed.

- If NDO (Network Domain Orchestrator) is configured, update connection details in NDO after migration, as the process changes OOB IP and subnet addresses.

- If an SMU (Software Maintenance Update) is installed on the physical APIC, migration from physical to virtual APIC is not recommended for Cisco APIC release 6.1(1). Upgrade the cluster to an image that contains the SMU fix before migrating.

- For app-infra, stop any running ELAM/FTRIAGE jobs prior to migration and restart them after migration is complete.

Limitations

- Migration of standby nodes is not supported. Remove all standby nodes from the cluster before migration.

- Migration is not supported for mini ACI fabric.

Migrating APIC clusters

The APIC cluster migration process transitions a three-node Cisco APIC cluster from physical (source) nodes to virtual (target) nodes to maintain continuous network operations.

The key components involved in the migration process are:

- Source APIC nodes: The physical nodes currently running in the cluster (APIC 1, APIC 2, APIC 3).

- Target APIC nodes: The virtual nodes that will form the new cluster after migration.

- Administrator: The user who initiates and manages the migration workflow.

In this section, a high-level explanation of the migration process is provided. For the detailed steps, see the Migrate an APIC cluster between physical and virtual deployments procedure in the subsequent section.

Consider a three-node cluster; three source nodes and correspondingly three target nodes (after-migration nodes). The APIC with controller ID 1 is considered as APIC 1. Login to APIC 1 (IP address 172.16.1.1) and initiate the migration process.

Sample APIC nodes

APIC

Source Node

Target Node

APIC 1

172.16.1.1

172.16.1.11

APIC 2

172.16.1.2

172.16.1.12

APIC 3

17.16.1.3

172.16.1.13

Stages in the migration process

1. Log in to source APIC 1 (172.16.1.1), and initiate the migration process.

2. Migration of source node APIC 3 (172.16.1.3) is initiated.

3. Migration of APIC 3 is completed (to target node 172.16.1.13).

4. Migration of source node APIC 2 (172.16.1.2) is initiated.

5. Migration of APIC 2 is completed (to target node 172.16.1.12).

6. Target APIC 2 takes control to enable the migration of APIC 1. This is called the handover process where in the control is passed on from source APIC 1 (172.16.1.1) to target APIC 2 (172.16.1.12). At this stage, a new window is displayed (URL redirected to target APIC 2). This is because, after successful migration, source APIC 1 is no longer part of the cluster (which now has the migrated target APICs).

The migration process is completed in the reverse order, that is, APIC N (APIC 3 in the example) is migrated first, followed by APIC N-1 (APIC 2 in the example), so on, and then finally APIC 1.

Migrate an APIC cluster between physical and virtual deployments

Use this procedure to migrate the nodes of a physical APIC cluster to a virtual APIC cluster (or vice-versa).

Before you begin

Following are the required prerequisites before you start with the migration process:

Cluster health

Confirm that the current APIC cluster is Fully fit.

Generic

- Ensure that the source and destination APICs’ date and time are synchronized.

- Ensure that all the controllers are on Cisco APIC release 6.1(1), and all the switches are running the same version as the controller.

Source and target nodes

- For directly connected APIC migration, ensure both source and target nodes are on the same layer 2 network.

- For remotely connected APIC migration, ensure both source and target nodes have infra network connectivity between them. This means the new target APIC should have the correct IPN configuration such that it can interact with the infra network of the fabric.

- Target nodes have the same admin password as the source cluster.

- Target nodes’ OOB IP address should be different while all other fields can be same or different from the source node. Infra addresses will remain same for layer 2 (directly attached); for layer 3 (remotely attached) cluster, they can be same or different based on deployments.

- Source cluster and target cluster OOB networking stacks should match. For example, if source cluster is using dual stack (IPv4 and IPv6) for OOB, dual stack (IPv4 and IPv6) address details should be provided for target nodes too.

- Ensure OOB connectivity between the source and destination APICs.

- Ensure the OOB contracts and reachability for the new APIC are configured correctly; the migration process uses the OOB IP address to communicate between the APICs.

For virtual APIC to physical APIC migration

- Ensure the physical APIC nodes are factory reset; use the acidiag touch setup and acidiag reboot commands.

- For migration with/ without CIMC (applicable for physical APICs):

If....

Then....

Using CIMC

ensure that the physical APIC CIMC addresses are reachable from the OOB network of the virtual APIC.

Not using CIMC

ensure that the OOB IP address is configured manually on the physical APIC after factory-reset and use the OOB option for connectivity.

For Physical to virtual migration

- Ensure that you have deployed the virtual APIC nodes as per the procedure in the Deploying Cisco Virtual APIC Using VMware vCenter guide.

- If virtual APICs are deployed on a vCenter that is part of a VMM domain, ensure that Infrastructure VLAN is enabled on the AEP configured on the interfaces connected to the ESXi host(s) where the virtual APIC is deployed.

Procedure

Step 1

On the Cluster as Seen by Node screen, click Migrate (displayed in the Cluster Overview area).

All the available controllers in the cluster are displayed.

Note

The Migrate button is displayed only on APIC 1 (of the cluster).

Step 2

Click the pencil icon next to the Validate column, to start the migration process of the selected controller.

The Migrate Node screen is displayed.

Step 3

Enter the following details in the Migrate Node screen:

1. For the Controller Type, select Virtual or Physical, as the case may be (migration from physical APIC to virtual APIC and vice-versa is supported).

2. For the Connectivity Type, select OOB, if you are migrating a physical APIC to a virtual APIC. If you are migrating a virtual APIC to a physical APIC, you can either select the OOB option or the CIMC option.

It is recommended to select the CIMC option for the virtual to physical migration. To use the OOB option, connect to the CIMC address of physical APICs and configure the OOB IP addresses manually before starting the migration process.

Controller Type and Connectivity Type are auto selected based on the source controller type. If required, you can modify them.

3. In the Management IP pane, enter the following (target APIC details) — Management IP address, Username, Password.

or

(applicable only for virtual to physical APIC migration) In the CIMC Details pane, enter the following details of the physical APIC— CIMC IP address, username and password of the node.

4. Click Validate.

After you click Validate, the details displayed in the General and Out of Band management panes change to match the details of the controller. The only editable fields are the Name and Pod ID (applicable only for layer 2), the other fields cannot be modified. For virtual to physical APIC migration, confirm the Admin Password too.

Note

If dual stack is supported, fill in the IPv4 and IPv6 addresses.

5. In the Infra Network pane (applicable only for Layer 3, APIC is remotely attached to the fabric), enter the following:

- IPv4 Address: the infra network address.

- IPv4 Gateway: the IP address of the gateway.

- VLAN: the interface VLAN ID to be used.

The OOB gateway and IP addresses are auto-populated in the table (based on the validation); click Apply. The validation status is displayed as Complete (on the Migrate Nodes screen).

Repeat the same process for the other APICs in the cluster by clicking the pencil icon (next to the Validation column). After providing all the controller details, click the Migrate button at the bottom of the Migrate screen.

Migration status

The migration process involves a series of activities, and this is displayed in stages. Each stage is indicated with a color-coded bar.

Migration Status

The Migrate Cluster Status screen displays the overall migration status, followed by the status of the apisever. The apiserver is the process that orchestrates the whole migration process. Below the apiserver status, the controller-wise migration status is displayed. The source IP address and the target IP address of the nodes are also indicated.

The apiserver status is indicated as 100% done (green bar) after the handover to APIC 2 is completed. At this stage, a new window is displayed (URL redirected to target APIC 2). Login to target APIC 2. A banner indicating that Migration is in progress is displayed at the top of the GUI until the migration is complete. After the handover process, the banner which was displayed on source APIC 1 is displayed on the target APIC 2. Click the View Status link on the banner to check the migration status.

You can also abort the migration process from source APIC 1 by clicking the Abort button available on the Migrate Cluster Status screen. The Abort button is displayed only after a certain period of time after initiating the migration.

After successful migration:

- the migration status is no longer displayed. If the migration has failed, then a failure message is explicitly displayed.

- to confirm if the target cluster is healthy and fully fit, navigate to System > Controllers > Expand Controllers. Expand Controller 1 > Cluster as seen by Node page.

- verify if all the fabric nodes are in active state; navigate to Fabric > Fabric Membership.

- if the Pod ID of the target APIC has changed, inband address for the node needs to be reconfigured on the Tenant Management screen. Navigate to Tenants > Mgmt > Node Management Addresses page.

Operations in case of a migration failure

The migration process may be interrupted due to a failure, or you may choose to abort the migration. In case the migration is not successful, it is recommended to revert or resume migration to either the source or target controller type. It is not recommended to have a APIC cluster in a migration failed state with a mix of physical and virtual controllers. Before attempting revert or resume, follow the steps in the next section, Basic Troubleshooting, to get the cluster to a healthy state.

If you choose to resume— the migration process is continued. On the Migrate Node screen (source APIC 1):

1. Enter the details of all the target nodes based on the controller type you want to migrate to.

2. Click Migrate.

If you choose to revert— the migration process is restarted. Migration process needs to be restarted after getting the controllers of the cluster to the initial (source) IP addresses.

1. Factory reset each of the source APIC nodes which are being migrated using acidiag touch setup and acidiag reboot commands.

2. On the Migrate Node screen, enter the source APIC details for all the nodes, as the migration process reverts the previously migrated APICs to the source controller type.

3. Click Migrate.

Note

If the migration process fails after the handover process (control is passed on to target APIC 2 from source APIC 1), the migration cannot be resumed or reverted.

As mentioned earlier, the various sub-stages of migration with their completion progress are indicated as bars, for each controller. In case of failure at any stage, collect the relevant tech-support details and contact Cisco TAC for further assistance. To collect the logs for tech support, navigate to Admin > Import/Export > Export Policies > On-demand Tech Support > migration_techsuppport.

Basic troubleshooting

Consider a three-node cluster where in two nodes have migrated successfully, and a failure is detected during the migration of the third node. Check the status of the failed node. If the controller is not in the Fully fit state, the migration could fail.

Use this procedure to get the cluster to a healthy state:

Procedure

Step 1

(for migration failures with APIC 1) Check the cluster health from target APIC 2 by navigating to, System > Controllers. Select Controller 2 > Cluster as seen by Node screen.

or

(for migration failures with APIC 2 to N) Check the cluster health from source APIC 1 by navigating to, System > Controllers. Select Controller 1 > Cluster as seen by Node screen.

Step 2

If APIC 1 (or any other node of the cluster) is not Fully fit, click the three dots adjacent to the serial number of the controller. Select Maintenance > Decommission. Click Force Decommission as the node is not in a Fully fit state. Connect to the source APIC node N using SSH and factory-reset the node using the following commands - acidiag touch setup , acidiag reboot .

Step 3

From source APIC 1 navigate to, System > Controllers. Click Controllers > Controller 1 > Cluster as seen by Node screen.

or

From target APIC 2 navigate to, System > Controllers. Click Controllers > Controller 2 > Cluster as seen by Node screen.

Step 4

To commission a controller, click the three dots adjacent to the serial number of the controller. Select Maintenance > Commission. Enter the details, as required. Refer to the Commissioning a Node procedure (described earlier in this chapter). The only difference here is that the controller ID is pre-populated with the number corresponding to the ID of the controller in the cluster.

After the controller is commissioned, the cluster is indicated as Fully fit.

Step 5

Check the status of the cluster after commissioning the failed node. If the cluster is in a healthy state, resume the migration by clicking Migrate on the Cluster As Seen By Node screen. If the migration fails again, contact Cisco TAC for further assistance.

Manage APIC cluster at boot using the GUI

Follow these steps to build a new cluster, add a node to an existing cluster, and replace one of the nodes in the existing cluster with a new node at boot using the GUI.

Procedure

Step 1

Log in to the APIC using https://APIC-IP and enter the password.

1. For virtual APICs:

If you have completed the deployment of virtual APICs using ESXi (OVF template) or remote AWS (CFT), then you see the output on the VM console similar to this example:

System pre-configured successfully.
Use: https://172.31.1.2 to complete the bootstrapping.

The IP address to access the bootstrapping GUI (APIC Cluster Bringup) is explicitly indicated, as shown in the example. You can proceed to step 2.

After deploying Cisco APIC on AWS, keep the OOB management IP address handy to access the Cluster Bringup GUI. You can get the OOB management IP address from the Stacks Outputs tab on the AWS GUI.

2. For physical APICs:

Log in to the APIC KVM console using the CIMC; you will see a screen as shown:

APIC Version: 6.0(2a)
Welcome to Cisco APIC Setup Utility
Press Enter Or Input JSON string to bootstrap your APIC node.

If you see only a black screen on the KVM, connect to the CIMC using SSH and use serial over LAN (SoL) ("connect host") to connect to the console.

On APIC, press Enter and provide the requested information. The IP address to access the bootstrapping GUI (APIC Cluster Bringup) is explicitly indicated.

admin user configuration ...
Enter the password for admin [None]:
Reenter the password for admin [None]:
Out-of-band management configuration ...
Enter the IP Address [192.168.10.1/24]: 172.20.7.79/23
Enter the IP Address of default gateway [192.168.10.254]: 172.20.6.1
Would you like to edit the configuration? (y/n) [n]:
System pre-configured successfully.
Use: https://172.20.7.79 to complete the bootstrapping

Note

The admin password must be the same as the password of the existing cluster.

The IP addresses displayed above are examples. The IP addresses will vary based on your deployment.

Step 2

In the Select Workflow screen of the APIC Cluster Bringup wizard, choose one of these workflows.

- New cluster: Use this option to start a new cluster.

- Cluster expansion: Use this option to add a node to an existing clsuter.

- APIC replacement: Use this option to replace one of the nodes in the existing cluster with a new node.

1. To start a new cluster, see Bring up the APIC cluster using the GUI.

2. To add a node to the existing cluster, perform these steps:

- In the Select Workflow screen, choose Cluster expansion and click Next.

- In the Cluster Verification screen, enter the OOB IP address of the active APIC node and click Validate.

The fabric and controllers information are displayed.

Click Next after verifying the information.

- In the Enter Node Details screen, enter a name for the controller and the Pod ID in the Controller Name field and Pod ID fields respectively.

Note

Pod ID is not required for Layer3 APIC.

- (Optional) Check the Standby checkbox and enter a value in the Controller ID field.

The range of standby controller ID is from 21 to 29.

The Controller ID field is auto-populated with a value and disabled by default.

- (Optional) Check the Force check box to force this configuration.

- (Optional) Check the Enable IPv6 check box if you want to enable IPv6 addresses for out-of-band management and enter the IPv6 address and gateway.

The IPv4 address and IPv4 gateway are auto-populated under the Out-of- band Network pane.

- (For Layer3 APIC) Enter these details under the Infra L3 Network pane.

- IPv4 Address: Enter the infra network address.

- IPv4 Gateway: Enter the IP address of the gateway.

- VLAN: Enter the interface VLAN ID to be used.

Click Next.

- In the Summary screen, review the information and click Deploy.

The progress of operation is displayed in a screen.

3. To replace one of the nodes in the existing cluster with a new node, perform these steps:

- In the Select Workflow screen, choose APIC replacement and click Next.

- In the Cluster Verification screen, enter the OOB IP address of the active APIC node and click Validate.

The fabric and controllers information are displayed.

Click Next after verifying the information.

- In the Enter Node Details screen, choose a controller ID from the Controller ID drop-down list.

Only the IDs of the decommissioned controllers are available in the Controller ID drop-down list.

The controller name is auto-populated based on the selected controller ID.

- Enter the Pod ID in the Pod ID field.

Note

Pod ID is not required for Layer3 APIC.

- (Optional) Check the Force check box to force this configuration.

- (Optional) Check the Enable IPv6 check box if you want to enable IPv6 addresses for out-of-band management and enter the IPv6 address and gateway.

The IPv4 address and IPv4 gateway are auto-populated under the Out-of- band Network pane.

- (For Layer3 APIC) Enter these details under the Infra L3 Network pane.

- IPv4 Address: Enter the infra network address.

- IPv4 Gateway: Enter the IP address of the gateway.

- VLAN: Enter the interface VLAN ID to be used.

Click Next.

- In the Summary screen, review the information and click Deploy.

The progress of operation is displayed in a screen.

Note

The user must manually reload the UI after this step to navigate to the main APIC UI.

Nexus Dashboard cluster from APIC GUI

Starting with APIC Release 6.1.4, users can directly navigate to a registered Nexus Dashboard cluster from the APIC GUI without the need for a separate login. The navigation is supported only if the Nexus Dashboard is running Release 4.1or later and the APIC is running Release 6.1.4 or later.

Only remote users with admin-write privileges on APIC can navigate from the APIC GUI to the Nexus Dashboard. A remote user refers to an APIC user authenticated through LDAP, RADIUS, or TACACS+. The logged-in user is granted super admin privileges on the Nexus Dashboard.

- Navigate to Nexus Dashboard cluster from APIC GUI

Navigate to Nexus Dashboard cluster from APIC GUI

Follow these steps to directly navigate to a registered Nexus Dashboard cluster from the APIC GUI without the need for a separate login.

See Connecting Clusters article to navigate to APIC from Nexus Dashboard.

Before you begin

- Ensure that the NTP servers on both the APIC and Nexus Dashboard are configured to synchronize with the same time source.

- APIC must be onboarded as a site within the Nexus Dashboard.

- The user must log in to the APIC as a remote user.

- The remote user must have admin-write privileges on the APIC. This user will have super admin privileges on the Nexus Dashboard.

- Ensure that the client system is configured with the DNS domain for the Nexus Dashboard, as APIC uses the Nexus Dashboard's hostname for navigation.

Procedure

Step 1

Log in to the APIC GUI.

Step 2

Navigate to Nexus Dashboard cluster using one of these methods:

- In the upper right corner of the APIC window, click the Launch the ND Cluster icon.

The registered Nexus Dashboard clusters are displayed.

Click the desired Nexus Dashboard cluster to navigate.

APIC starts the active node in the Nexus Dashboard cluster.

- On the menu bar, choose Integrations > Nexus Dashboard.

The registered Nexus Dashboard clusters are displayed in a table.

Check the desired cluster, click …, and choose Open in New Tab to navigate to the Nexus Dashboard cluster.

Configuring the Cisco APIC Using the CLI

- Best practices for cluster management

- Replacing a Cisco APIC in a Cluster Using the CLI

- Cluster size reductions

- Cisco APIC Cluster contractions

- Switching Over Active APIC with Standby APIC Using CLI

- Verifying Cold Standby Status Using the CLI

- Registering an Unregistered Switch Using the CLI

- Adding a Switch Before Discovery Using the CLI

- Removing a Switch to Maintenance Mode Using the CLI

- Inserting a Switch to Operation Mode Using the CLI

- Configuring a Remote Location Using the NX-OS Style CLI

- Finding Your Switch Inventory Using the NX-OS CLI

- Verifying the Cisco APIC Cluster Using the CLI

Best practices for cluster management

- Always verify the health of all controllers before making any changes to the cluster. Confirm that every controller is fully fit and resolve any health issues before proceeding.

- Ensure that all controllers in the cluster run the same firmware version before adding, configuring, or clustering devices. Do not cluster controllers running different firmware versions.

- Maintain at least three active controllers in your cluster, and add standby controllers as needed. For scalability requirements, consult the Verified Scalability Guide to determine the required number of active controllers for your deployment.

- Ignore cluster information from controllers that are not currently active in the cluster, as their data may be inaccurate.

- Know that once you configure a cluster slot with a controller’s ChassisID, you must decommission that controller to make the slot available for reassignment.

- Wait for all ongoing firmware upgrades to complete and verify the cluster is fully fit before making additional changes.

- When moving a controller, always ensure the cluster is healthy. Select the controller you intend to move, shut it down, physically move and reconnect it, and then power it on. After the move, verify through the management interface that all controllers return to a fully fit state.

- Move only one controller at a time to maintain cluster stability.

- When transferring a controller to a different set of leaf switches or to a different port within the same leaf switch, ensure the cluster is healthy first. Decommission the controller before moving it, and then recommission it after the move.

- Before configuring the cluster, confirm that all controllers run the same firmware version to prevent unsupported operations and cluster issues.

- Delete any unused OOB EPGs associated with a controller. Assigning multiple EPGs to a controller is not supported and can cause the cluster workflow IP address to be overridden by policy.

- Remember that log record objects are stored only in one shard on a single controller. If you decommission or replace that controller, those logs are permanently lost.

- When decommissioning a controller, be aware that all fault, event, and audit log history stored on it is deleted. If you replace all controllers, all log history is lost. Before migrating a controller, manually back up its log history to prevent data loss.

Replacing a Cisco APIC in a Cluster Using the CLI

Note

- For more information about managing clusters, see Best practices for cluster management.

- When you replace an Cisco APIC, the password will always be synced from the cluster. When replacing APIC 1, you will be asked for a password but it will be ignored in favor of the existing password in the cluster. When replacing Cisco APIC 2 or 3, you will not be asked for a password.

Before you begin

Before replacing a Cisco Application Policy Infrastructure Controller (APIC), ensure that the replacement Cisco APIC is running the same firmware version as the Cisco APIC to be replaced. If the versions are not the same, you must update the firmware of the replacement Cisco APIC before you begin. Initial clustering of Cisco APICs running differing versions is an unsupported operation and may cause problems within the cluster.

Procedure

Step 1

Identify the Cisco APIC that you want to replace.

Step 2

Note the configuration details of the Cisco APIC to be replaced by using the acidiag avread command.

Step 3

Decommission the Cisco APIC using the decommission controller controller-id command in config mode.

Decommissioning the Cisco APIC removes the mapping between the APIC ID and Chassis ID. The new Cisco APIC typically has a different APIC ID, so you must remove this mapping in order to add a new Cisco APIC to the cluster.

Beginning with Cisco APIC release 6.0(2), an optional argument (force) is added to the decommission command to allow forcing the decommission operation. The revised command is decommission controller controller-id [force] , with the following behaviors:

- When force is not declared, the decommission proceeds only if the cluster is not in an unhealthy or upgrade state, where a decommission may not be proper.

- When force is declared, the decommission proceeds regardless of the cluster state.

For example, decommission controller 3 force decommissions APIC3 regardless of the cluster state.

Step 4

To commission the new Cisco APIC, follow these steps:

1. Disconnect the old Cisco APIC from the fabric.

2. Connect the replacement Cisco APIC to the fabric.

The new Cisco APIC appears in the Cisco APIC GUI menu System > Controllers > apic_controller_name > Cluster as Seen by Node in the Unauthorized Controllers list.

3. Commission the new Cisco APIC using the controller controller-id commission command.

4. Boot the new Cisco APIC.

5. Allow several minutes for the new Cisco APIC information to propagate to the rest of the cluster.

The new Cisco APIC appears in the Cisco APIC GUI menu System > Controllers > apic_controller_name > Cluster as Seen by Node in the Active Controllers list.

What to do next

For each decommissioned controller, verify that the operational state of the controller is unregistered and that the controller is no longer in service in the cluster.

Note

If a decommissioned Cisco APIC is not promptly removed from the fabric, it might be rediscovered, which could cause problems. In that case, follow the instructions in Cluster size reductions to remove the controller.

Cluster size reductions

A cluster size reduction is a system management operation that

- decreases the number of controllers in a cluster,

- requires the orderly decommissioning and removal of selected controllers, and

- triggers cluster synchronization processes to maintain system stability.

Follow these guidelines to reduce the Cisco Application Policy Infrastructure Controller (APIC) cluster size and decommission the Cisco APICs that are removed from the cluster:

Note

Failure to follow an orderly process to decommission and power down Cisco APICs from a reduced cluster can lead to unpredictable outcomes. Do not allow unrecognized Cisco APICs to remain connected to the fabric.

- Reducing the cluster size increases the load on the remaining Cisco APICs. Schedule the Cisco APIC size reduction at a time when the demands of the fabric workload will not be impacted by the cluster synchronization.

- If one or more of the Cisco APICs' health status in the cluster is not "fully fit," remedy that situation before proceeding.

- Reduce the cluster target size to the new lower value. For example if the existing cluster size is 6 and you will remove 3 controllers, reduce the cluster target size to 3.

- Starting with the highest numbered controller ID in the existing cluster, decommission, power down, and disconnect the Cisco APIC one by one until the cluster reaches the new lower target size.

Upon the decommissioning and removal of each controller, the Cisco APIC synchronizes the cluster.

Note

After decommissioning a Cisco APIC from the cluster, promptly power it down and disconnect it from the fabric to prevent its rediscovery. Before returning it to service, do a wiped clean back to factory reset.

If the disconnection is delayed and a decommissioned controller is rediscovered, follow these steps to remove it:

1. Power down the Cisco APIC and disconnect it from the fabric.

2. In the list of Unauthorized Controllers, reject the controller.

3. Erase the controller from the GUI.

- Cluster synchronization stops if an existing Cisco APIC becomes unavailable. Resolve this issue before attempting to proceed with the cluster synchronization.

- Depending on the amount of data the Cisco APIC must synchronize upon the removal of a controller, the time required to decommission and complete cluster synchronization for each controller could be more than 10 minutes per controller.

Example:

If a cluster originally contains six controllers and three are to be removed, administrators should set the cluster target size to three. Remove controllers one at a time, starting with the highest numbered controller ID, and follow established procedures to ensure reduction and synchronization are successful.

Note

Complete the entire necessary decommissioning steps, allowing the Cisco APIC to complete the cluster synchronization accordingly before making additional changes to the cluster.

Cisco APIC Cluster contractions

A cluster contraction is a cluster management operation that

- reduces a cluster’s size by removing one node at a time within legal boundaries,

- increases computational and memory load on the remaining nodes, and

- makes the decommissioned cluster slot unavailable until re-enabled by operator input.

When contracting the cluster, always decommission the last APIC in the sequence first and proceed in reverse order. For example:

- Decommission APIC4 before APIC3.

- Decommission APIC3 before APIC2.

Switching Over Active APIC with Standby APIC Using CLI

Use this procedure to switch over an active APIC with a standby APIC.

Procedure

Step 1

replace-controller replace ID number Backup serial number

Replaces an active APIC with an standby APIC.

Example:

apic1#replace-controller replace 2 FCH1804V27L
Do you want to replace APIC 2 with a backup? (Y/n): Y

Step 2

replace-controller reset ID number

Resets fail over status of the active controller.

Example:

apic1# replace-controller reset 2
Do you want to reset failover status of APIC 2? (Y/n): Y

Verifying Cold Standby Status Using the CLI

Procedure

To verify the Cold Standby status of APIC, log in to the APIC as admin and enter the command show controller .

apic1# show controller
Fabric Name          : vegas
Operational Size     : 3
Cluster Size         : 3
Time Difference      : 496
Fabric Security Mode : strict

ID    Pod   Address          In-Band IPv4     In-Band IPv6               OOB IPv4         OOB IPv6                   Version             Flags  Serial Number     Health
----  ----  ---------------  ---------------  -------------------------  ---------------  -------------------------  ------------------  -----  ----------------  ------------------
1*    1     10.0.0.1         0.0.0.0          fc00::1                    172.23.142.4     fe80::26e9:b3ff:fe91:c4e0  2.2(0.172)          crva-  FCH1748V0DF       fully-fit
2     1     10.0.0.2         0.0.0.0          fc00::1                    172.23.142.6     fe80::26e9:bf8f:fe91:f37c  2.2(0.172)          crva-  FCH1747V0YF       fully-fit
3     1     10.0.0.3         0.0.0.0          fc00::1                    172.23.142.8     fe80::4e00:82ff:fead:bc66  2.2(0.172)          crva-  FCH1725V2DK       fully-fit
21~         10.0.0.21                                                                                                                    -----  FCH1734V2DG

Flags - c:Commissioned | r:Registered | v:Valid Certificate | a:Approved | f/s:Failover fail/success
(*)Current (~)Standby

Registering an Unregistered Switch Using the CLI

Use this procedure to register a switch from the Nodes Pending Registration tab on the Fabric Membership work pane using the CLI.

Note

This procedure is identical to "Adding a Switch Before Discovery Using the CLI". When you execute the command, the system determines if the node exists and, if not, adds it. If the node exists, the system registers it.

Procedure

[no] system switch-id serial-number switch-id name pod id role leaf node-type tier-2-leaf

Adds the switch to the pending registration list.

Adding a Switch Before Discovery Using the CLI

Use this procedure to add a switch to the Nodes Pending Registration tab on the Fabric Membership work pane using the CLI.

Note

This procedure is identical to "Registering an Unregistered Switch Using the CLI". When you execute the command, the system determines if the node exists and, if not, adds it. If the node does exist, the system registers it.

Procedure

[no] system switch-id serial-number switch-id name pod id role leaf node-type tier-2-leaf

Adds the switch to the pending registration list.

Removing a Switch to Maintenance Mode Using the CLI

Use this procedure to remove a switch to maintenance mode using the CLI.

Note

While the switch is in maintenance mode, CLI 'show' commands on the switch show the front panel ports as being in the up state and the BGP protocol as up and running. The interfaces are actually shut and all other adjacencies for BGP are brought down, but the displayed active states allow for debugging.

Procedure

[no]debug-switch node_id or node_name

Removes the switch to maintenance mode.

Inserting a Switch to Operation Mode Using the CLI

Use this procedure to insert a switch to operational mode using the CLI.

Procedure

[no]no debug-switch node_id or node_name

Inserts the switch to operational mode.

Configuring a Remote Location Using the NX-OS Style CLI

In the ACI fabric, you can configure one or more remote destinations for exporting techsupport or configuration files.

Procedure

SUMMARY STEPS

1. configure

2. [no] remote path remote-path-name

3. user username

4. path {ftp | scp | sftp} host [:port] [remote-directory ]

DETAILED STEPS

Step 1

configure

Enters global configuration mode.

Example:

apic1# configure

Step 2

[no] remote path remote-path-name

Enters configuration mode for a remote path.

Example:

apic1(config)# remote path myFiles

Step 3

user username

Sets the user name for logging in to the remote server. You are prompted for a password.

Example:

apic1(config-remote)# user admin5

Step 4

path {ftp | scp | sftp} host [:port] [remote-directory ]

Sets the path and protocol to the remote server. You are prompted for a password.

Example:

apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic

Examples

This example shows how to configure a remote path for exporting files.

apic1# configure
apic1(config)# remote path myFiles
apic1(config-remote)# user admin5
You must reset the password when modifying the path:
Password:
Retype password:
apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic
You must reset the password when modifying the path:
Password:
Retype password:

Finding Your Switch Inventory Using the NX-OS CLI

This section explains how to find your switch model and serial numbers using the NX-OS CLI.

Procedure

Find your switch inventory as follows:

Example:

switch# show hardware
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Software
BIOS:      version 07.56
kickstart: version 12.1(1h) [build 12.1(1h)]
system:    version 12.1(1h) [build 12.1(1h)]
PE:        version 2.1(1h)
BIOS compile time:       06/08/2016
kickstart image file is: /bootflash/aci-n9000-dk9.12.1.1h.bin
kickstart compile time:  10/01/2016 20:10:40 [10/01/2016 20:10:40]
system image file is:    /bootflash/auto-s
system compile time:     10/01/2016 20:10:40 [10/01/2016 20:10:40]

Hardware
cisco N9K-C93180YC-EX ("supervisor")
Intel(R) Xeon(R) CPU  @ 1.80GHz with 16400384 kB of memory.
Processor Board ID FDO20101H1W

Device name: ifav41-leaf204
bootflash:    62522368 kB

Kernel uptime is 02 day(s), 21 hour(s), 42 minute(s), 31 second(s)

Last reset at 241000 usecs after Sun Oct 02 01:27:25 2016
Reason: reset-by-installer
System version: 12.1(1e)
Service: Upgrade

plugin
Core Plugin, Ethernet Plugin
--------------------------------
Switch hardware ID information
--------------------------------

Switch is booted up
Switch type is : Nexus C93180YC-EX Chassis
Model number is N9K-C93180YC-EX
H/W version is 0.2010
Part Number is 73-15298-01
Part Revision is 1
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-15298-01

--------------------------------
Chassis has one slot
--------------------------------

Module1 ok
Module type is : 48x10/25G
1 submodules are present
Model number is N9K-C93180YC-EX
H/W version is 0.2110
Part Number is 73-17776-02
Part Revision is 11
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-17776-02

GEM ok
Module type is : 6x40/100G Switch
1 submodules are present
Model number is N9K-C93180YC-EX
H/W version is 0.2110
Part Number is 73-17776-02
Part Revision is 11
Manufacture Date is Year 20 Week 10
Serial number is FDO20101H1W
CLEI code is 73-17776-02

---------------------------------------
Chassis has  2 PowerSupply Slots
---------------------------------------

PS1 shut
Power supply type is : 54.000000W 220v AC
Model number is NXA-PAC-650W-PE
H/W version is 0.0
Part Number is 341-0729-01
Part Revision is A0
Manufacture Date is Year 19 Week 50
Serial number is LIT19500ZEK
CLEI code is 341-0729-01

PS2 ok
Power supply type is : 54.000000W 220v AC
Model number is NXA-PAC-650W-PE
H/W version is 0.0
Part Number is 341-0729-01
Part Revision is A0
Manufacture Date is Year 19 Week 50
Serial number is LIT19500ZEA
CLEI code is 341-0729-01

---------------------------------------
Chassis has  4 Fans
---------------------------------------

FT1 ok

Fan1(sys_fan1)(fan_model:NXA-FAN-30CFM-F)                                    is inserted but info is not available

FT2 ok

Fan2(sys_fan2)(fan_model:NXA-FAN-30CFM-F)                                    is inserted but info is not available

FT3 ok

Fan3(sys_fan3)(fan_model:NXA-FAN-30CFM-F)                                    is inserted but info is not available

FT4 ok

Fan4(sys_fan4)(fan_model:NXA-FAN-30CFM-F)                                    is inserted but info is not available

====================================================================================

Verifying the Cisco APIC Cluster Using the CLI

Cisco Application Policy Infrastructure Controller (APIC) release 4.2.(1) introduces the cluster_health command, which enables you to verify the Cisco APIC cluster status step-by-step. The following output example demonstrates a scenario where everything is fine except for one node (ID 1002), which is inactive.

Note

To use the cluster_health command, you must be logged in as admin.

Procedure

To verify the cluster status:

F1-APIC1# cluster_health
Password:

Running...

Checking Wiring and UUID: OK
Checking AD Processes: Running
Checking All Apics in Commission State: OK
Checking All Apics in Active State: OK
Checking Fabric Nodes: Inactive switches: ID=1002(IP=10.1.176.66/32)
Checking Apic Fully-Fit: OK
Checking Shard Convergence: OK
Checking Leadership Degration: Optimal leader for all shards
Ping OOB IPs:
APIC-1: 172.31.184.12 - OK
APIC-2: 172.31.184.13 - OK
APIC-3: 172.31.184.14 - OK
Ping Infra IPs:
APIC-1: 10.1.0.1 - OK
APIC-2: 10.1.0.2 - OK
APIC-3: 10.1.0.3 - OK
Checking APIC Versions: Same (4.2(0.261a))
Checking SSL: OK

Done!

Cluster_Health Verification Steps

Step

Description

Checking Wiring and UUID

Leaf switches provide infra connectivity between each Cisco APIC by detecting the Cisco APICs using LLDP. This step checks wiring issues between a leaf and a Cisco APIC that is detected during LLDP discovery.

Any issues in here implies a leaf switch cannot provide infra connectivity for a Cisco APIC as it doesn’t have a valid information. For example, a Cisco APIC UUID mismatch means the new APIC2 has a different UUID than the previously known APIC2.

UUID – Universally Unique ID, or chassis ID in some outputs

Checking AD Processes

Cisco APIC clustering is handled by the Appliance Director process on each Cisco APIC. This step checks if the process is running correctly.

Checking All APICs in Commission State

To complete the Cisco APIC clustering, all Cisco APICs need to be commissioned.

Checking All APICs in Active State

To complete the Cisco APIC clustering, all commissioned Cisco APICs need to be active. If it is not active, the Cisco APIC may not be up yet.

Checking Fabric Nodes: Inactive switches

The Cisco APIC's communication are through infra connectivity provided by leaf and spine switches. This step checks inactive switches to ensure switches are providing infra connectivity.

Checking APIC Fully-Fit

When Cisco APICs have established IP reachability to each other through infra network, it will synchronize its database to each other. When the synchronization completes, the status of all Cisco APICs become "Fully-Fit." Otherwise, the status will be "Data Layer Partially Diverged," and so on.

Checking Shard Convergence

When Cisco APICs are not fully-fit, database shards need to be checked to see which service is not fully synchronized. If there is any service that has problems in synchronization, you may reach out to Cisco TAC for further troubleshooting.

Checking Leadership Degration

In ACI, each database shard has one leader shard distributed to each Cisco APIC in the cluster. This step shows if all shards have an optimal leader. If there is an issue in here when all Cisco APICs are up, you may reach out to Cisco TAC for further troubleshooting.

Ping OOB IPs

This step is to check if all Cisco APICs are up and operational by pinging the OOB IP which is configured separately from clustering.

Ping Infra IPs

This step is to check if there is infra connectivity between each Cisco APIC. Cisco APIC clustering is performed through infra connectivity instead of OOB.

Checking APIC Versions

All Cisco APICs should be on a same version to complete clustering.

Checking SSL

All Cisco APICs need to have a valid SSL that should be built-in when a Cisco APIC is shopped as an appliance. Without a valid SSL, the server cannot operate the Cisco APIC OS correctly.

Configuring the Cisco APIC Using the REST API

- Expanding the APIC Cluster Using the REST API

- Contracting the APIC Cluster Using the REST API

- Cluster size reductions

- Switching Over Active APIC with Standby APIC Using REST API

- Registering an Unregistered Switch Using the REST API

- Adding a Switch Before Discovery Using the REST API

- Removing a Switch to Maintenance Mode Using the REST API

- Inserting a Switch to Operational Mode Using the REST API

- Configuring a Remote Location Using the REST API

- Sending an On-Demand Tech Support File Using the REST API

- Finding Your Switch Inventory Using the REST API

Expanding the APIC Cluster Using the REST API

The cluster drives its actual size to the target size. If the target size is higher than the actual size, the cluster size expands.

Procedure

Step 1

Set the target cluster size to expand the APIC cluster size.

Example:

POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=3/>

Step 2

Physically connect the APIC controllers that you want to add to the cluster.

Contracting the APIC Cluster Using the REST API

Use this procedure to shrink the cluster size by removing controllers. For more information about contracting the cluster size, see Cisco APIC Cluster contractions.

Note

Beginning with Cisco APIC Release 6.0(2), two additional properties are added to the API command to allow forcing the decommission operation. The new object properties are:

- infraClusterPol:shrink

- false: (default) If the target cluster size (infraClusterPol:size) is less than the current operational cluster size, the APICs to be removed must be decommissioned manually, as in earlier releases.

- true: If the target cluster size is less than the current operational cluster size, a cluster shrink decommission is triggered. The APICs to be removed are decommissioned automatically, beginning with the APIC with the highest controller ID number.

- infraWiNode:force

- false: (default) The decommission proceeds only if the cluster is not in an unhealthy or upgrade state, where a decommission may not be proper.

- true: The decommission proceeds regardless of the cluster state.

This example shows how to contract the cluster from three APIC controllers to a single controller. To achieve a target size of one, APIC3 and APIC2 must be decommissioned, in that order.

Procedure

Step 1

Set the target cluster size so as to contract the APIC cluster size.

When the cluster size is reduced with shrink='true', the APICs to be removed are decommissioned automatically. Otherwise, they must be decommissioned manually.

Example:

Cisco APIC Release 6.0(1) and earlier:

POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=1 />

You must now manually decommission the APICs to be removed, as shown in the next steps.

Example:

Cisco APIC Release 6.0(2) and later, using 'shrink' property:

POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=1 shrink='true' />

With shrink='true', the following steps can be skipped. The APICs to be removed are decommissioned automatically.

POST
https://<IP address>/api/node/mo/uni/controller.xml
<infraClusterPol name='default' size=1 shrink='false' />

With shrink='false', you must now manually decommission the APICs to be removed, as shown in the next steps.

Step 2

Decommission APIC3 on APIC1 for cluster contraction.

Example:

Cisco APIC Release 6.0(1) and earlier:

POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=3 adminSt='out-of-service'/>

The decommission proceeds only if the cluster is in a healthy state.

Example:

Cisco APIC Release 6.0(2) and later, using 'force' property:

POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=3 adminSt='out-of-service' force='true' />

With force='true', the decommission proceeds regardless of the cluster state.

Step 3

Decommission APIC2 on APIC1 for cluster contraction.

Example:

Cisco APIC Release 6.0(1) and earlier:

POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=2 adminSt='out-of-service'/>

The decommission proceeds only if the cluster is in a healthy state.

Example:

Cisco APIC Release 6.0(2) and later, using 'force' property:

POST
https://<IP address>/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=2 adminSt='out-of-service' force='false' />

With force='false', the decommission proceeds only if the cluster is in a healthy state.

The operation cluster size shrinks only after the last appliance is decommissioned, and not after the administrative size is changed. Verify after each controller is decommissioned that the operational state of the controller is unregistered and that the controller is no longer in service in the cluster.

Note

If a decommissioned APIC controller is not promptly removed from the fabric, it might be rediscovered, which could cause problems. In that case, follow the instructions in Cluster size reductions to remove the controller.

Cluster size reductions

A cluster size reduction is a system management operation that

- decreases the number of controllers in a cluster,

- requires the orderly decommissioning and removal of selected controllers, and

- triggers cluster synchronization processes to maintain system stability.

Follow these guidelines to reduce the Cisco Application Policy Infrastructure Controller (APIC) cluster size and decommission the Cisco APICs that are removed from the cluster:

Note

Failure to follow an orderly process to decommission and power down Cisco APICs from a reduced cluster can lead to unpredictable outcomes. Do not allow unrecognized Cisco APICs to remain connected to the fabric.

- Reducing the cluster size increases the load on the remaining Cisco APICs. Schedule the Cisco APIC size reduction at a time when the demands of the fabric workload will not be impacted by the cluster synchronization.

- If one or more of the Cisco APICs' health status in the cluster is not "fully fit," remedy that situation before proceeding.

- Reduce the cluster target size to the new lower value. For example if the existing cluster size is 6 and you will remove 3 controllers, reduce the cluster target size to 3.

- Starting with the highest numbered controller ID in the existing cluster, decommission, power down, and disconnect the Cisco APIC one by one until the cluster reaches the new lower target size.

Upon the decommissioning and removal of each controller, the Cisco APIC synchronizes the cluster.

Note

After decommissioning a Cisco APIC from the cluster, promptly power it down and disconnect it from the fabric to prevent its rediscovery. Before returning it to service, do a wiped clean back to factory reset.

If the disconnection is delayed and a decommissioned controller is rediscovered, follow these steps to remove it:

1. Power down the Cisco APIC and disconnect it from the fabric.

2. In the list of Unauthorized Controllers, reject the controller.

3. Erase the controller from the GUI.

- Cluster synchronization stops if an existing Cisco APIC becomes unavailable. Resolve this issue before attempting to proceed with the cluster synchronization.

- Depending on the amount of data the Cisco APIC must synchronize upon the removal of a controller, the time required to decommission and complete cluster synchronization for each controller could be more than 10 minutes per controller.

Example:

If a cluster originally contains six controllers and three are to be removed, administrators should set the cluster target size to three. Remove controllers one at a time, starting with the highest numbered controller ID, and follow established procedures to ensure reduction and synchronization are successful.

Note

Complete the entire necessary decommissioning steps, allowing the Cisco APIC to complete the cluster synchronization accordingly before making additional changes to the cluster.

Switching Over Active APIC with Standby APIC Using REST API

Use this procedure to switch over an active APIC with standby APIC using REST API.

Procedure

Switch over active APIC with standby APIC.

URL for POST: https://ip address/api/node/mo/topology/pod-initiator_pod_id/node-initiator_id/av.xml
Body: <infraWiNode id=outgoing_apic_id targetMbSn=backup-serial-number/>
where initiator_id = id of an active APIC other than the APIC being replaced.
pod-initiator_pod_id = pod ID of the active APIC
backup-serial-number = serial number of standby APIC

Example:

https://ip address/api/node/mo/topology/pod-1/node-1/av.xml
<infraWiNode id=2 targetMbSn=FCH1750V00Q/>

Registering an Unregistered Switch Using the REST API

Use this procedure to register a switch from the Nodes Pending Registration tab on the Fabric Membership work pane using the REST API.

Note

This procedure is identical to "Adding a Switch Before Discovery Using the REST API". When you apply the code, the system determines if the node exists and, if not, adds it. If the node does exist, the system registers it.

Procedure

Add a switch description.

Example:

POST
https://<IP address>/api/policymgr/mo/uni.xml

<!-- /api/policymgr/mo/uni.xml -->
<polUni>
<ctrlrInst>
<fabricNodeIdentPol>
<fabricNodeIdentP  nodeType="tier-2-leaf" podId="1" serial="XXXXXXXXX"
name="tier-2-leaf-leaf1" nodeId="101"/>

</fabricNodeIdentPol>
</ctrlrInst>
</polUni>

Adding a Switch Before Discovery Using the REST API

Use this procedure to add a switch to the Nodes Pending Registration tab on the Fabric Membership work pane using the REST API.

Note

This procedure is identical to "Registering an Unregistered Switch Using the REST API". When you apply the code, the system determines if the node exists and, if not, adds it. If the node does exist, the system registers it.

Procedure

Add a switch description.

Example:

POST
https://<IP address>/api/policymgr/mo/uni.xml

<!-- /api/policymgr/mo/uni.xml -->
<polUni>
<ctrlrInst>
<fabricNodeIdentPol>
<fabricNodeIdentP  nodeType="tier-2-leaf" podId="1" serial="XXXXXXXXX"
name="tier-2-leaf1" nodeId="101"/>

</fabricNodeIdentPol>
</ctrlrInst>
</polUni>

Removing a Switch to Maintenance Mode Using the REST API

Use this procedure to remove a switch to maintenance mode using the REST API.

Procedure

Remove a switch to maintenance mode.

Example:

POST
https://<IP address>/api/node/mo/uni/fabric/outofsvc.xml

<fabricOOServicePol
descr=""
dn=""
name="default"
nameAlias=""
ownerKey=""
ownerTag="">
<fabricRsDecommissionNode
debug="yes"
dn=""
removeFromController="no"
tDn="topology/pod-1/node-102"/>
</fabricOOServicePol>

Inserting a Switch to Operational Mode Using the REST API

Use this procedure to insert a switch to operational mode using the REST API.

Procedure

Insert a switch to operational mode.

Example:

POST
https://<IP address>/api/node/mo/uni/fabric/outofsvc.xml

<fabricOOServicePol
descr=""
dn=""
name="default"
nameAlias=""
ownerKey=""
ownerTag="">
<fabricRsDecommissionNode
debug="yes"
dn=""
removeFromController="no"
tDn="topology/pod-1/node-102"
status="deleted"/>
</fabricOOServicePol>

Configuring a Remote Location Using the REST API

This procedure explains how to create a remote location using the REST API.

<fileRemotePath name="local" host=“host or ip" protocol=“ftp|scp|sftp" remotePath=“path to folder" userName=“uname" userPasswd=“pwd" />

Sending an On-Demand Tech Support File Using the REST API

Procedure

Step 1

Set the remote destination for a technical support file using the REST API, by sending a POST with XML such as the following example:

Example:

<fileRemotePath userName="" remotePort="22" remotePath="" protocol="sftp" name="ToSupport" host="192.168.200.2"
dn="uni/fabric/path-ToSupport" descr="">

<fileRsARemoteHostToEpg tDn="uni/tn-mgmt/mgmtp-default/oob-default"/>

</fileRemotePath>

Step 2

Generate an on-demand technical support file using the REST API by sending a POST with XML such as the following:

Example:

<dbgexpTechSupOnD upgradeLogs="no" startTime="unspecified" name="Tech_Support_9-20-16"
exportToController="no" endTime="unspecified" dn="uni/fabric/tsod-Tech_Support_9-20-16" descr=""
compression="gzip" category="forwarding" adminSt="untriggered">
<dbgexpRsExportDest tDn="uni/fabric/path-ToSupport"/>
<dbgexpRsTsSrc tDn="topology/pod-1/node-102/sys"/>
<dbgexpRsTsSrc tDn="topology/pod-1/node-103/sys"/>
<dbgexpRsTsSrc tDn="topology/pod-1/node-101/sys"/>
<dbgexpRsData tDn="uni/fabric/tscont"/>
</dbgexpTechSupOnD>
<fabricFuncP>
<fabricCtrlrPGrp name="default">
<fabricRsApplTechSupOnDemand tnDbgexpTechSupOnDName=" Tech_Support_9-20-16"/>
</fabricCtrlrPGrp>
</fabricFuncP>

Finding Your Switch Inventory Using the REST API

This section explains how to find your switch model and serial numbers using the REST API

Procedure

Find your switch inventory as follows:

Example:

GET https://192.0.20.123/api/node/mo/topology/pod-1.json?query-target=children&target-subtree-class=fabricNode

The following response is returned:

response:
{
"totalCount":"8",
"imdata":
[{
"fabricNode":{
"attributes":{
"adSt":"on",
"childAction":"",
"delayedHeartbeat":"no",
"dn":"topology/pod-1/node-103",
"fabricSt":"active",
"id":"103",
"lcOwn":"local",
"modTs":"2016-10-08T14:49:35.665+00:00",
"model":"N9K-C9396PX",
"monPolDn":"uni/fabric/monfab-default",
"name":"leaf3",
"nameAlias":"",
"role":"leaf",
"serial":"TEP-1-103",
"status":"","uid":"0",
"vendor":"Cisco Systems, Inc",
"version":""}
}
},{
"fabricNode":{
"attributes":{
"adSt":"on",
"childAction":"",
"delayedHeartbeat":"no",
"dn":"topology/pod-1/node-105",
"fabricSt":"active",
"id":"105",
"lcOwn":"local",
"modTs":"2016-10-08T14:47:52.011+00:00",
"model":"N9K-C9508",
"monPolDn":"uni/fabric/monfab-default",
"name":"spine2",
"nameAlias":"",
"role":"spine",
"serial":"TEP-1-105","status":"",
"uid":"0",
"vendor":"Cisco Systems, Inc",
"version":""
...
[TRUNCATED]
...
}

Trademarks

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS REFERENCED IN THIS DOCUMENTATION ARE SUBJECT TO CHANGE WITHOUT NOTICE. EXCEPT AS MAY OTHERWISE BE AGREED BY CISCO IN WRITING, ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS DOCUMENTATION ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

The Cisco End User License Agreement and any supplemental license terms govern your use of any Cisco software, including this product documentation, and are located at: https://www.cisco.com/c/en/us/about/legal/cloud-and-software/software-terms.html. Cisco product warranty information is available at https://www.cisco.com/c/en/us/products/warranty-listing.html. US Federal Communications Commission Notices are found here https://www.cisco.com/c/en/us/products/us-fcc-notice.html.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any products and features described herein as in development or available at a future date remain in varying stages of development and will be offered on a when-and if-available basis. Any such product or feature roadmaps are subject to change at the sole discretion of Cisco and Cisco will have no liability for delay in the delivery or failure to deliver any products or feature roadmap items that may be set forth in this document.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: https://www.cisco.com/c/en/us/about/legal/trademarks.html. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)