This topic explains the process of switch discovery in a fabric management system, detailing how the APIC automatically detects and registers switches to manage the ACI fabric efficiently, ensuring each switch is managed by a single APIC cluster.

Switch discovery is a fabric management process that enables a controller to automatically detect and register switches in the network. The APIC is a central point of automated provisioning and management for all the switches that are part of the ACI fabric. A single data center might include multiple ACI fabrics; each data center might have its own APIC cluster and Cisco Nexus 9000 Series switches that are part of the fabric. To ensure that a switch is managed only by a single APIC cluster, each switch must be registered with that specific APIC cluster that manages the fabric.

The APIC discovers new switches that are directly connected to any switch it currently manages. Each APIC instance in the cluster first discovers only the leaf switch to which it is directly connected. After the leaf switch is registered with the APIC, the APIC discovers all spine switches that are directly connected to the leaf switch. As each spine switch is registered, that APIC discovers all the leaf switches that are connected to that spine switch. In a few simple steps, this cascaded discovery allows the APIC to discover the entire fabric topology.

Switch Registration with the APIC Cluster

After you register a switch with the Cisco Application Policy Infrastructure Controller (APIC), the switch is part of the Cisco APIC-managed fabric inventory. With the Cisco Application Centric Infrastructure (ACI) fabric, the Cisco APIC is the single point of provisioning, management, and monitoring for switches in the infrastructure.

These guidelines and limitations apply:

Before you begin registering a switch, make sure that all switches in the fabric are physically connected and booted in the desired configuration. For information about the installation of the chassis, see https://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

When the switch is running a different version than your APIC cluster, use Auto Firmware Update on Switch Discovery to automatically upgrade the switch during the discovery phase. See Auto Firmware Update on Discovery in the Cisco APIC Installation and ACI Upgrade and Downgrade Guide for details.
The infrastructure IP address range must not overlap with other IP addresses used in the ACI fabric for in-band and out-of-band networks.
When a switch is power cycled or upgraded, downlink interfaces will be in the admin-down state until the switch downloads the configurations again from the Cisco APICs. This process prevents external devices from sending traffic to a switch before it is ready. Fabric links and down links for Cisco APIC connectivity are exempt from being changed to the admin-down state. To achieve this exemption, the leaf switch remembers the downlink interface that was connected to the Cisco APICs prior to the power cycle or upgrade. Because of this, you must not change the Cisco APIC connectivity until the switches are fully operational again after the power cycle or upgrade.

Switch Role Considerations

A switch role is a fundamental part of network architecture design. It defines a switch's operational behavior in fabric topology, port connectivity, and discovery protocols. The role of a switch can be changed using the CLI based on the deployment scenario.

The default fabric links must be used for initial switch discovery from another switch.
If a default spine switch is connected to the Cisco Application Policy Infrastructure Controller (APIC) directly, the switch will be converted to a leaf switch automatically. During the conversion period, the fault will be present in the Cisco APIC, which is a normal behavior. The fault will be removed after the switch conversion is finished.
For a leaf switch, you can configure a port profile to convert a port to be a downlink or fabric link after the port is registered to the Cisco APIC. For more information, see the Cisco APIC Layer 2 Networking Configuration Guide at the following site:

https://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html#Configuration_Guides

This table specifies the default role for the switches that support role changes:

Table 1. Default Switch Roles
Switch Product ID	Default Role	First Release to Support a Role Change ¹
N9K-C93600CD-GX	Leaf	5.2(1)
N9K-C9316D-GX	Spine	5.1(4)
N9K-C9364C-GX	Leaf	5.1(3)
N9K-C9332D-GX2B	Leaf	5.2(3)
N9K-C9364D-GX2A	Spine	5.2(3)
N9K-C9348D-GX2A	Spine	5.2(3)
N9K-C9408	Spine	6.0(2)

¹ Specifies the first release to support the role change for the indicated switch. Role changing for that switch is supported in all subsequent releases.

The hybrid switches can change roles. The default role for hybrid switches is different for each model. In release 6.1(2), you can change the role of the switch by using a CLI command before the switch is discovered. The default role of these switches determines the role of interfaces during the discovering phase. This may cause some inconvenience when the switch is used in its non-default role. For instance, a switch whose default role is leaf is cabled up as a spine switch. Once the switch is discovered, its role is automatically converted to spine. However, by default, it boots up as a leaf switch and most of its interfaces are not configured as fabric links that can be used for spine discovery. As a result, you may have limited cabling options to ensure that the switch is discovered.

Another example: when the switch whose default role is spine is cabled up as a leaf switch and is supposed to be directly connected to an APIC. All the interfaces of a spine switch, however, are fabric links that cannot be used to connect to an APIC. As a result, the switch must be discovered through another switch which is connected to another APIC, so that it can be converted to a leaf switch–which now can be connected to an APIC.

Changing the default role of a switch is similar to assigning a new job function before an employee joins a company. Assigning the correct role ensures the switch is properly integrated into the network infrastructure from the outset, avoiding connectivity limitations and extra work.

To address these issues, use this CLI command on the new, Undiscovered switch to change its role before being discovered.

(none)# acidiag setrole <leaf/spine>
This command will reboot the switch, Proceed? [y/N]

Note

The acidiag setrole <leaf/spine> command will NOT work if the switch has been discovered. You will see an error message.

Registering an Unregistered Switch Using the GUI

Before you begin

Ensure that all switches in the fabric are physically connected and booted.

Note

The infrastructure IP address range must not overlap with other IP addresses used in the ACI fabric for in-band and out-of-band networks.

Procedure

On the menu bar, choose Fabric > Inventory.

In the Navigation pane, choose Fabric Membership.

In the Work pane, click the Nodes Pending Registration tab.

Switches in the Nodes Pending Registration tab table may be in one of these conditions:

A newly discovered but unregistered node has a node ID of 0 and has no IP address.
A manually entered (in Cisco Application Policy Infrastructure Controller (APIC)) but unregistered switch has an original status of Undiscovered until it is physically connected to the network. Once connected, the status changes to Discovered.

In the Nodes Pending Registration table, locate a switch with an ID of 0 or a newly connected switch with the serial number you want to register.

(Optional) Double-click the row of that node if you want to see more information about the node.

A dialog appears that shows you various node properties, such as the ACI-mode switch release and information about LLDP neighbors.

Right-click the row of that switch, choose Register, and perform the following actions:

Verify the displayed Serial Number to determine which switch is being added.

Configure or edit the following settings:


Field	Setting
Pod ID	Identifier of the pod where the node is located.
Node ID	A number greater than 100. The first 100 IDs are reserved for Cisco APIC appliance nodes. Note We recommend that you number leaf nodes and spine nodes be differently. Assign numbers to spine nodes in the 100 range (such as 101 and 102) and assign numbers to leaf nodes in the 200 range (such as 201 and 202). After the node ID is assigned, it cannot be updated. After the node has been added to the Registered Nodes tab table, you can update the node name by right-clicking the table row and choosing Edit Node and Rack Name.
RL TEP Pool	Tunnel endpoint (TEP) pool identifier for the node.
Node Name	The node name, such as leaf1 or spine3.
Role	The assigned node role. The options are: spine leaf virtualleaf virtualspine remote leaf tier-2-leaf If you choose a role other than the default role for the node, the node automatically reboots during the registration to change the role.
Rack Name	Choose Default, or choose Create Rack to add a name and description.

Click Register.

Cisco APIC assigns an IP address to the node and the node is added to the Registered Nodes tab table. Next and if applicable, other nodes that are connected to this node are discovered and appear in the Nodes Pending Registration tab table.

Continue to monitor the Nodes Pending Registration tab table. As more nodes appear, repeat these steps to register each new node until all installed nodes are registered.

Add a switch before discovery using the GUI

You can add a switch description before the switch is physically connected to the network by following these steps:

Before you begin

Ensure you have the serial number of the switch.

Procedure

On the menu bar, choose Fabric > Inventory.

In the Navigation pane, choose Fabric Membership.

On the Registered Nodes or Nodes Pending Registration work pane, click the Actions icon, then click Create Fabric Node Member.

The Create Fabric Node Member dialog appears.

Configure the following settings:


Field	Setting
Pod ID	Identify the pod where the node is located.
Serial Number	Required: Enter the serial number of the switch.
Node ID	Required: Enter a number greater than one hundred. The first one hundred IDs are reserved for Cisco Application Policy Infrastructure Controller (APIC) appliance nodes. Note We recommend that you number leaf nodes and spine nodes differently. For example, number leaf nodes in the 100 range (such as 101, 102) and number spine nodes in the 200 range (such as 201, 202). After you assign the node ID, you cannot update it. Once the node has been added to the Registered Nodes tab table, you can update the node name. To do this, right-click the table row and choose Edit Node and Rack Name.
Switch Name	The node name, such as leaf1 or spine3.
Node Type	Choose the type (role) for the node. The options are: leaf Put a check in one of these boxes if applicable: Is Remote: Specifies that the node is a remote leaf switch. Is Virtual: Specifies that the node is virtual. Is Tier-2 Leaf: The fabric node member (leaf switch) being created will take on the characteristics of a tier-2 leaf switch in a multi-tier architecture. spine Put a check in this box if applicable: Is Virtual: Specifies that the node is virtual. unknown If you choose a role other than the default role for the node, the node automatically reboots during the registration to change the role.
VPC Pair	Optional. If the node is part of a vPC pair, choose the ID of the node with which to pair this node.
VPC Domain ID	Enter the vPC domain ID for the vPC pair. The range is from 1 to 1,000. This field only appears if you entered a value for VPC Pair, and is required in that case.

The Cisco APIC adds the new node to the Nodes Pending Registration tab table.

What to do next

Connect the physical switch to the network. After you connect the switch, the Cisco APIC matches the serial number of the physical switch to the new entry. Monitor the Nodes Pending Registration tab table until the Status for the new switch changes from Undiscovered to Discovered. Then, follow the steps in the Registering an Unregistered Switch Using the GUI section to complete the fabric initialization and discovery process for the new switch.

Switch discovery validation and switch management from the APIC

After you register the switches with the Cisco Application Policy Infrastructure Controller (APIC), the APIC automatically discovers the fabric topology so you can view and manage all the switches in the network.

You can configure, monitor, and upgrade each switch from the APIC without having to access the individual switches.

Validate the registered switches using the GUI

Confirm that switches are registered to the fabric and display their node IDs and assigned IP addresses.

Procedure

	1.	On the menu bar, navigate to Fabric > Inventory > Fabric Membership.
	2.	In the Fabric Membership work pane, click the Registered Nodes tab. The Registered Nodes tab displays the switches in the fabric, along with their node IDs and assigned IP addresses.

Validate the fabric topology

After you register all the switches with the APIC cluster, the APIC automatically discovers all links and connectivity in the fabric and discovers the entire topology.

Validate the fabric topology using the GUI

Procedure

	1.	On the menu bar, navigate to **Fabric > Inventory > Pod number**.
	2.	In the Work pane, click the Topology tab. The displayed diagram shows all attached switches, APIC instances, and links.
	3.	(Optional) Hover over any component to view its health, status, and inventory information.
	4.	(Optional) To view the port-level connectivity of a leaf switch or spine switch, double-click its icon within the topology diagram.
	5.	(Optional) To refresh the topology diagram, click the icon in the upper right corner of the Work pane.

Unmanaged switch connectivity in VM management

An unmanaged switch in VM management is a Layer 2 device that can be connected between VM hosts and leaf ports. It must be configured with a management address which is advertised using LLDP or CDP. APIC uses that management address to discover and identify the unmanaged switch.

You can connect hosts managed by the VM controller, such as a vCenter, to the leaf port through a Layer 2 switch. APIC automatically discovers Layer 2 switches and identifies them by management address.

Find your switch inventory using the GUI

Use this procedure when you need to locate hardware information for your switches managed by Cisco APIC.

Before you begin

You must have access to the Cisco APIC GUI

Follow this procedure to find your switch inventory using the GUI:

Procedure

	1.	On the menu bar, choose Fabric > Inventory.
	2.	In the navigation pane, click a Pod icon. Your switch icons appear in the navigation pane.
	3.	In the navigation pane, click on a switch icon. A list of tabs appears at the top of the work pane.
	4.	Click the General tab. Your switch information appears in the work pane.

What to do next

Troubleshoot Switch Discovery Issues

The ACI-mode switch software includes a comprehensive leaf and spine switch discovery validation program. If your switch is stuck in discovery mode, use the switch CLI command to launch the leaf and spine switch discovery validation program.

The validation program performs the following tests:

System state—Checks the state of the topSystem managed object (MO).
1. If the state is "out-of-service," checks for any scheduled upgrades.
2. If the state is "downloading bootscript," a failure has occurred in the downloading bootscript. The failure is reported. If the switch is an L3out spine, the program also checks the bootstrap download state and reports any failure.
DHCP status—Checks for DHCP status and information, such as the TEP IP, node Id, and name assigned from the dhcpResp MO.
AV details—Checks whether the APICs are registered and whether they have valid IP addresses.
IP reachability—Uses the iping command to verify IP reachability to the address assigner APIC. To retest this condition, use the show discoveryissues apic ipaddress command.
infra VLAN received—Checks for the presence of the infra VLAN details in the lldpInst MO. If this switch belongs to a pod that has no APIC, no infra VLAN details are present, and this section of the test result can be ignored.
LLDP adjacency—Checks for the presence of LLDP adjacencies and for any wiring mismatch issues. LLDP issues can generate fault reports such as infra VLAN mismatch, chassis ID mismatch, or no connection to the front end ports.
Switch version—Reports the running firmware version of the switch. Also reports the version of the APIC, if available.
FPGA/BIOS—Checks for any FPGA/BIOS version mismatch on the switch.
SSL validation—Checks for validity of the SSL certificate details using the acidiag verifyssl -s serialNumber command.
Policy downloads—Checks the pconsBootStrap MO to see whether registration to APIC (policy managment shards) is complete and whether all policies were downloaded successfully.
Time—Reports the current time on the switch.
Hardware status—Checks the module, power, and fan status from the eqptCh, eqptFan, eqptPsu, eqptFt, and eqptLC MOs.

Running the Test Manually

To run the switch discovery validation program, log in to the spine or leaf switch CLI console and run this command:

show discoveryissues [apic ipaddress]

Example of a Successful Test

This output is an example of a successful switch discovery validation test.


spine1# show discoveryissues

Checking the platform type................SPINE!
Check01 - System state - in-service                [ok]
Check02 - DHCP status                [ok]
    TEP IP: 10.0.40.65 Node Id: 106 Name: spine1
Check03 - AV details check                [ok]
Check04 - IP reachability to apic                [ok]
    Ping from switch to 10.0.0.1 passed
Check05 - infra VLAN received                [ok]
    infra vLAN:1093
Check06 - LLDP Adjacency                [ok]
    Found adjacency with LEAF
Check07 - Switch version                [ok]
    version: n9000-14.2(0.167)  and apic version: 5.0(0.25)
Check08 - FPGA/BIOS out of sync test                [ok]
Check09 - SSL check                [check]
    SSL certificate details are valid
Check10 - Downloading policies                [ok]
Check11 - Checking time                [ok]
    2019-08-21 17:15:45
Check12 - Checking modules, power and fans                [ok]

Example of a Failed Test

This ouptput shows an example of a switch discovery validation program with discovery issues.


spine1# show discoveryissues

Checking the platform type................SPINE!
Check01 - System state - out-of-service                [FAIL]
    Upgrade status is notscheduled
    Node upgrade is notscheduled state
Check02 - DHCP status                [FAIL]
    ERROR: discover not being sent by switch
    Ignore this, if the IP is already known by switch
    ERROR: node Id not configured
    ERROR: IP not assigned by dhcp server
    ERROR: Address assigner's IP not populated
    TEP IP: unknown Node Id: unknown Name: unknown
Check03 - AV details check                [ok]
Check04 - IP reachability to apic                [FAIL]
    please rerun the CLI with argument apic Ip
    (show discoveryissues apic <ip>) to check its reachability from switch
Check05 - infra VLAN received                [FAIL]
    Please ignore if this switch is part of a pod with no apic
Check06 - LLDP Adjacency                [FAIL]
    Error: spine not connected to any leaf
Check07 - Switch version                [ok]
    version: n9000-14.2(0.146)  and apic version: unknown
Check08 - FPGA/BIOS out of sync test                [ok]
Check09 - SSL check                [ok]
    SSL certificate details are valid
Check10 - Downloading policies                [FAIL]
    Registration to all PM shards is not complete
    Policy download is not complete
    Pcons booststrap is in triggered state
Check11 - Checking time                [ok]
    2019-07-17 19:26:29
Check12 - Checking modules, power and fans                [FAIL]
    Line card state is testing

Maintenance Mode

These terms are helpful to understand when using maintenance mode:

Maintenance mode: Used to isolate a switch from user traffic for debugging purposes. You can put a switch in maintenance mode by enabling the Maintenance (GIR) field in the Fabric Membership page in the APIC GUI, located at Fabric > Inventory > Fabric Membership (right-click on a switch and choose Maintenance (GIR)).

If you put a switch in maintenance mode, that switch is no longer considered part of the operational ACI fabric infra and it will not accept regular APIC communications.

You can use maintenance mode to gracefully remove a switch and isolate it from the network in order to perform debugging operations. When you enable maintenance mode, the switch is removed from the regular forwarding path with minimal traffic disruption.

In graceful removal, all external protocols except the fabric protocol (IS-IS) are gracefully brought down, and the switch is isolated from the network. During maintenance mode, the maximum metric is advertised in IS-IS within the Cisco Application Centric Infrastructure (Cisco ACI) fabric, so the leaf switch in maintenance mode does not attract traffic from the spine switches. In addition, all front-panel interfaces on the switch are shut down except for the fabric interfaces. To return the switch to its fully operational (normal) mode after the debugging operations, recommission the switch. This action triggers a stateless reload of the switch.

During graceful insertion, the system automatically decommissions, reboots, and recommissions the switch. After recommissioning is completed, the system restores all external protocols and resets maximum metric in IS-IS 10 minutes later.

These protocols are supported:

Border Gateway Protocol (BGP)
Enhanced Interior Gateway Routing Protocol (EIGRP)
Intermediate System-to-Intermediate System (IS-IS)
Open Shortest Path First (OSPF)
Link Aggregation Control Protocol (LACP)

Protocol Independent Multicast (PIM) is not supported.

Important Notes

If you place a border leaf switch with static route in maintenance mode, the route might remain in the routing tables of other switches in the ACI fabric. This can cause routing issues.

To work around this issue, perform one of these actions:
- Configure the same static route with the same admnistrative distance on the other border leaf switch, or
- Use IP SLA or BFD for track reachability to the next hop of the static route
While the switch is in maintenance mode, the Ethernet port module stops propagating the interface related notifications. As a result, if the remote switch is rebooted or the fabric link is flapped during this time, the fabric link will not come up afterward unless the switch is manually rebooted (using the acidiag touch clean command), decommissioned, and recommissioned.
While the switch is in maintenance mode, CLI 'show' commands on the switch show the front panel ports as being in the up state and the BGP protocol as up and running. The interfaces are actually shut and all other adjacencies for BGP are brought down, but the displayed active states allow you to to debug.
For multi-pod / multi-site, set the IS-IS metric for redistributed routes to less than 63 to minimize the traffic disruption when bringing the node back into the fabric. To set the IS-IS metric for redistributed routes, choose Fabric > Fabric Policies > Pod Policies > IS-IS Policy.
When you reboot a spine or leaf and after the IS-IS adjacency comes up the IS-IS metric for redistributed routes is advertised as high, which is, 34 and will not be available as an ECMP next hop.
Existing GIR supports all Layer 3 traffic diversion. With LACP, all the Layer 2 traffic is also diverted to the redundant node. After a node enters maintenance mode, LACP immediately informs neighboring switches that aggregation is no longer possible in the port-channel. Then, all traffic is diverted to the vPC peer node.
You cannot perform these operations in maintenance mode:
- Upgrade: Upgrading the network to a newer version to isolate a switch from user traffic
- Stateful Reload: Restarting the GIR node or its connected peers
- Stateless Reload: Restarting with a clean configuration or power cycle of the GIR node or its connected peers
- Link Operations: Shut / no-shut or optics OIR on the GIR node or its peer node
- Configuration Change: Any configuration change (such as clean configuration, import, or snapshot rollback)
- Hardware Change: Any hardware change (such as adding, swapping, removing FRU's or RMA)

Set a switch to maintenance mode using the GUI

Use this procedure to set a switch to maintenance mode using the GUI to safely perform upgrades or diagnostics. Out-of-band management interfaces will remain up and accessible during this process.

Procedure

	1.	On the menu bar, choose Fabric > Inventory.
	2.	In the navigation pane, click Fabric Membership.
	3.	In the work pane, click Actions > Maintenance (GIR).
	4.	Click OK. The gracefully removed switch displays Debug Mode in the Status column.

The switch enters the maintenance mode and displays Debug Mode in the Status column.

Set a switch to operational mode using the GUI

Use this procedure to insert a switch to operational mode using the GUI.

Procedure

	1.	On the menu bar, choose Fabric > Inventory.
	2.	In the navigation pane, click Fabric Membership.
	3.	In the Registered Nodes table in the work pane, right-click the row for the switch you want to set to operational mode and select Commission.
	4.	Click Yes.

The switch transitions into operational mode and is ready to use.

What to do next

Verify the switch connectivity and status in the operational dashboard.

Need help?

Open a support case

(Requires a Cisco Service Contract)

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.