Connecting the Switch to the ACI Fabric

ACI Fabric Topology

The ACI fabric topology includes the following major components:

As shown in the following figure, each APIC is connected to one or two leaf switches and each leaf switch should be connected to every spine switch in the same fabric.


Note

To prevent sub-optimal forwarding between endpoints, connect every leaf switch in the fabric to every spine switch in the same fabric.

Figure 1. Connections Between APIC Clusters, Leaf Nodes, and Spine Nodes

Preparing to Connect to Other Devices

When preparing to connect the fabric devices, consider the following for each type of interface, and gather all of the required equipment before making the connections:

  • Cabling type required for each interface type

  • Distance limitations for each signal type

  • Additional interface equipment required


Note

When running power and data cables in overhead or subfloor cable trays, we strongly recommend that you locate power cables and other potential noise sources as far away as practical from network cabling that terminates on Cisco equipment. In situations where long parallel cable runs cannot be separated by at least 3.3 feet (1 meter), we recommend that you shield any potential noise sources by housing them in a grounded metallic conduit.

The optical transceivers that are not already assembled to their cables come separate from their cables. To prevent these transceivers and their cables from being damaged, we recommend that you keep the transceivers disconnected from their cables when installing them in ports and then insert the optical cable into the transceiver. When removing transceivers from ports, remove their cables before removing the transceivers.

To maximize the effectiveness and life of your transceivers and optical cables, do the following:

  • Wear an ESD-preventative wrist strap that is connected to an earth ground whenever handling transceivers. The switch is typically grounded when you install transceivers and provides an ESD port to which you can connect your wrist strap. If you cannot find an ESD port, connect the wrist strap to an earth ground (such as the grounding connection for the chassis).

  • Do not remove or insert a transceiver more often than necessary. Repeated removals and insertions can shorten its useful life.

  • Keep the transceivers and fiber-optic cables clean and dust free to maintain high signal accuracy and to prevent damage to the connectors. Attenuation (loss of light) increases with contamination and should be kept below 0.35 dB.

    • Clean these parts before installing them to prevent dust from scratching the fiber-optic cable ends.

    • Clean the connectors regularly; the required frequency of cleaning depends upon the environment. In addition, clean connectors if they are exposed to dust or accidentally touched. Both wet and dry cleaning techniques can be effective; refer to your site's fiber-optic connection cleaning procedures.

    • Do not touch the ends of connectors. Touching the ends can leave fingerprints and cause other contamination.

  • Inspect routinely for dust and damage. If you suspect damage, clean and then inspect fiber ends under a microscope to determine if damage has occurred.

Connecting Leaf Switches to APICs

You must downlink one or two (recommended for redundancy) Cisco Nexus 9300 platform ACI-mode leaf switches to each Application Policy Infrastructure Controller (APIC) in your ACI fabric. The type of virtual interface card (VIC) installed on the APIC determines the types of interface cables that you can use to connect the leaf switches to the APICs.

  • The VIC 1225T module supports copper connectors, copper cables, and switches with copper downlink ports (such as: Cisco Nexus 93108TC-EX, 93108TC-FX, 93120TX, 93128TX, 9372TX, 9372TX-E, and 9396TX switches).

  • The VIC 1225 module supports optical transceivers, optical cables, and switches with optical downlink ports (such as: Cisco Nexus 93180LC-EX, 93180YC-EX, 93180YC-FX, 9332PQ, 9336C-FX2, 9348GC-FXP, 9372PX, 9372PX-E, 9396PX, and 93600CD-GC switches).

  • The VIC 1455 module supports optical transceivers, optical cables, and switches with optical downlink ports (such as: Cisco Nexus 9336C-FX2, 93180LC-EX, 93180YC-EX, 93180YC-FX, 93240YC=FX2, and 93600CD-GC switches).

Before you begin

The APIC and leaf switches in the fabric must be fully installed in their racks and grounded.

Procedure

Step 1

Connect an interface cable to one of the two to four ports on the virtual interface card (VIC) installed on the APIC. If the cable is not already assembled to its transceivers, insert the transceiver into the VIC port and then connect the optical interface cable to the transceiver.

  • For a VIC 1225T 10GBASE-T copper module, use 10GBASE-T cables with RJ-45 connectors.

  • For a VIC 1225 optical module, use one of the following sets of transceivers and cables:

  • For a VIC 1455 SFP28 module, 10-Gigabit only, use one of the following sets of transceivers and cables:

    • Cisco 10GBASE-LR transceivers (SFP-10G-LR) supporting a link length of up to 6.1 miles (10 km)

    • Cisco 10GBASE-SR transceivers (SFP-10G-SR) supporting the following link lengths:

      • Using 2000 MHz MMF (OM3) for up to 984 feet (300 m)

      • Using 4700 MHz MMF (OM4) for up to 1312 feet (400 m)

    • Cisco SFP+ Active Optical Cables (SFP-10G-AOCxM [where x=1, 2, 3, 5, 7, or 10 for lengths in meters])

    • Cisco SFP+ Twinax Cables (SFP-H10GB-CUxM [where x=1, 2, 3, 5, 7, or 10 for lengths in meters])

    Note 

    The VIC 1455 has 4 ports, port-1, port-2, port-3, and port-4 from left to right.

    • All ports must have the same speed, either 10-Gigabit or 25-Gigabit.

    • Port-1 and port-2 is one pair, corresponding to eth2-1 on APIC and port-3 and port-4 is another pair, corresponding to eth2-2 on APIC. Only one connection is allowed for each pair. For example, you can connect one cable to either port-1 or port-2, and connect another cable to either port-3 or port-4 (please do not connect two cables on any pair).

    To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

    To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.

Step 2

Connect the other end of the interface cable to a downlink port on a leaf switch.

  • For a Cisco 10GBASE-LR or -SR transceiver and cable, insert the transceiver into a downlink optical port on a leaf switch before connecting the cable to the transceiver.

  • For Cisco SFP+ Active Optical Cables, insert the transceiver on the cable into a downlink optical port on a leaf switch.

  • For a 10GBASE-T copper cable, insert the RJ-45 connector on the cable into a downlink BASE-T port on a leaf switch.

Note 

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

Connecting Leaf Switches to Spine Switches

For optimal forwarding between endpoints, you must connect each leaf switch to every spine switch in the same ACI fabric.

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.


Warning

Warning – Invisible Laser Radiation. Do not expose users of telescopic optics. Class 1/1M Laser Products.

Warning

Invisible laser radiation may be emitted from the end of the unterminated fiber cable or connector. Do not view directly with optical instruments. Viewing the laser output with certain optical instruments (for example, eye loupes, magnifiers, and microscopes) within a distance of 100 mm may pose an eye hazard.

Before you begin

  • The leaf and spine switches in the fabric (such as: N9k-C9364C, N9K-C9332C, and N9K-C9316D-GX) must be fully installed in their racks and grounded.

  • If there are modular switches in the fabric, their ACI-mode line cards must already be installed. The line cards can be of the following types:

    • 36-port 40-Gigabit (such as: N9K-X9736PQ)

    • 32-port 100-Gigabit (such as: N9K-X9732C-EX) (supported by Cisco Nexus 9504 and 9508 modular switches)

    • 36-port 100-Gigabit (such as: N9K-X9736C-FX)


Note

You cannot include NX-OS line cards in the same chassis when running in ACI mode.

Note

Multiple uplinks from a leaf switch to a spine switch is supported. A symmetrical topology is recommended so that all devices have equal access to resources.

Procedure

Step 1

For the transceivers with removable cables, make sure that the transceivers are separated from their interface cables.

Step 2

Insert the appropriate transceiver into an active uplink port on the leaf switch.

Step 3

Insert the same type of transceiver in the spine switch port on the line card.

Step 4

For transceivers with removable cables, insert the interface cable into the open end of each of those transceivers.

Step 5

Repeat Steps 1 through 4 for each spine switch in the ACI fabric.

The leaf switch is connected to each spine switch in the ACI fabric.
Step 6

Repeat Steps 1 through 5 for each leaf switch in the ACI fabric.

Each leaf switch in the ACI fabric is connected to each spine switch in the network,

The fabric automatically implements Equal Cost Multi-Pathing (ECMP) and enables all links. You do not need to configure the links.

Installing a Gigabit Ethernet module (GEM)

Procedure

Step 1

Clear the switch's current configuration by using the setup-clean-config command.

Step 2

Power off the switch by disconnecting the power.
Step 3

Replace the current GEM card with the new GEM card.
Step 4

Power on the switch.

Virtual Port Channel Migration - Migration of Nodes from a First-Generation Switch to a Second-Generation Switch

Initially the fabric is configured with vPCs between two first-generation switches. Traffic flows are designed so that only these vPCs are used for data traffic. Now that you want to migrate both of the first-generation switches to second-generation switches, the following steps are required.

For this procedure, vPC primary and vPC secondary are first generation switches in vPC pair, sending traffic as described above.

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.

Before you begin

You have two first-generation Cisco Nexus 9000 Series switches comprising a virtual port channel (vPC). You are migrating to two second-generation Cisco Nexus 9000 Series switches using the same cables.

First-generation Cisco Nexus 9000 Series switches include those switches that do not contain an EX or an FX in the PID (product identification).

Second-generation Cisco Nexus 9000 Series switches include those switches that have an EX or an FX in the PID.

Move any APIC controllers that are connected to the migrating vPC first-generation switches to any other switches in the fabric and wait for the APIC cluster to become "Fully Fit".

Procedure

Step 1

From the APIC GUI, perform the Remove From Controller operation for vPC secondary. The switch is clean rebooted by the APIC. Wait for about 10 minutes for this operation to finish. This action prompts all traffic to use the other first-generation switch for data traffic. Disconnect the cabling from vPC secondary.

Step 2

Uninstall the first-generation switch by reversing the order of the steps in the Installing the Switch Chassis section, of the switch specific Hardware Installation Guide.

Step 3

Install the second-generation switch by following the steps in the Installing the Switch Chassis section, of the switch specific Hardware Installation Guide.

Step 4

Connect the loose cabling that you removed from the first-generation switch, to the same ports on the second-generation switch.
Step 5

Now you register the new second-generation switch with the APIC. Register the new node with the same node name and node ID. This switch becomes part of the fabric. Policies are pushed to the new switch and the vPC legs are kept down since there is a mismatch of generation switches. At this point, vPC primary continues to send the data traffic.

Step 6

From the APIC GUI, perform the Remove From Controller operation for vPC primary. This switch is clean rebooted by the APIC.

Wait for about 10 minutes for this operation to finish. The vPC leg on the second-generation switch, which was kept down earlier comes up. This action prompts all traffic to move to the new second-generation switch. Please note the vPC ports on the new second-generation switch will come up in about 10 to 22 seconds when STP is disabled for the deployed VLANs on the remote devices, and there will be traffic drops in the range of 10 to 40 seconds, depending upon the flows in the fabric. When STP is enabled on the VLANs on the remote devices, the traffic loss will be in the range of 40 to 75 seconds, depending upon the flows in the fabric.

Step 7

Disconnect the cabling from the other first-generation switch.
Step 8

Uninstall the first-generation switch, like you did in step 2.
Step 9

Install the second-generation switch, like you did in step 3.
Step 10

Connect the loose cabling, like you did in step 4.
Step 11

Register the new second-generation switch with the APIC. Register the new node with the same node name and node ID. This switch becomes part of the fabric. Policies are pushed to the new switch and the vPC legs comes up and starts passing traffic.

Setting Up an Optional Console Interface

You can optionally set up a console interface for performing the initial configuration of the switch. To do this, use the interface cable provided in the accessory kit to connect the switch to your console device. You can connect the console port on the switch to a modem. If you do not connect it to a modem, make the connection either before powering up the switch or after completing the boot process for the switch.

Before you begin

The console device must support VT100 terminal emulations and asynchronous transmissions.

Procedure

Step 1

Configure the terminal emulator program to match each of the following default port characteristics:

  • 9600 baud

  • 8 data bits

  • 1 stop bit

  • No parity
Step 2

Insert the RJ-45 connector on the interface cable found in the accessory kit into the RS-232 port on the switch and insert the DB-9 connector on the other end of the cable to the serial port on the console device.

What to do next

You can now perform the initial configuration for the switch (see the Cisco ACI Getting Started Guide).

Setting Up an Optional Management Connection

You can optionally set up an out-of-band management connection for monitoring and troubleshooting purposes. To do this, depending on your switch, you connect either the RJ-45 management port or the SFP management port on the switch to an external hub, switch, or router.

Before you begin

To prevent an IP address conflict, you must complete the initial configuration for the switch and establish an IP address before you create the management connection.

Procedure

Step 1

Connect the interface cable to a management port on the switch.
Step 2

Connect the other end of the cable to an external hub, switch, or router.

Optic Transceiver Removal Using the Optics Extraction Tool

You use both ends of the optics extraction tool in this procedure. You use the wide end for releasing the bale latch, and the narrow end for removing the transceiver module.

Before you begin

Ensure that before you handle any switch components, you are wearing a grounded electrostatic discharge (ESD) strap. To ground the strap, attach it directly to earth ground or to a grounded rack or chassis. There must be a metal-to-metal connection to earth ground.

Procedure

Step 1

Remove the fiber-optic cables from the transceiver module before removing the transceiver module.
Step 2

Use the wide side of the optics extraction tool to release the bale latch (see the following image).

Step 3

Use the narrow end of the optics extraction tool to carefully remove the transceiver module (see the following image).
Step 4

Place the transceiver module in an antistatic bag or other protective environment.

Maintaining Transceivers and Optical Cables

Transceivers and fiber-optic cables must be kept clean and dust free to maintain high signal accuracy and prevent damage to the connectors. Contamination increases attenuation (loss of light) and should be below 0.35 dB.

Consider the following maintenance guidelines:

  • Transceivers are static sensitive. To prevent ESD damage, wear an ESD-preventative wrist strap that is connected to the grounded chassis.

  • Do not remove and insert a transceiver more often than is necessary. Repeated removals and insertions can shorten its useful life.

  • Keep all optical connections covered when not in use. Clean them before using to prevent dust from scratching the fiber-optic cable ends.

  • Do not touch the ends of connectors. Touching the ends can leave fingerprints and cause other contamination.

  • Clean the connectors regularly; the required frequency of cleaning depends upon the environment. In addition, clean connectors if they are exposed to dust or accidentally touched. Both wet and dry cleaning techniques can be effective; refer to the fiber-optic connection cleaning procedures for your site.

  • Inspect routinely for dust and damage. If you suspect damage, clean and then inspect fiber ends under a microscope to determine if damage has occurred.