The fabric interconnect is the core component of Cisco UCS. The Cisco UCS Fabric Interconnects provide uplink access to LAN, SAN, and out-of-band management segment. Cisco UCS infrastructure management is through the embedded management software, Cisco UCS Manager, for both hardware and software management. The Cisco UCS Fabric Interconnects are Top-of-Rack devices, and provide unified access to the Cisco UCS domain.
The Cisco UCS FIs provide network connectivity and management for the connected servers. The Cisco UCS Fabric Interconnects run the Cisco UCS Manager control software and consist of expansion modules for the Cisco UCS Manager software.
For more information about Cisco UCS Fabric Interconnects, see the Cisco UCS Manager Getting Started Guide.
Use fabric interconnect ports configured as uplink ports to connect to uplink upstream network switches. Connect these uplink ports to upstream switch ports as individual links, or as links configured as port channels. Port channel configurations provide bandwidth aggregation as well as link redundancy.
You can achieve northbound connectivity from the fabric interconnect through a standard uplink, a port channel, or a virtual port channel configuration. The port channel name and ID configured on fabric interconnect should match the name and ID configuration on the upstream Ethernet switch.
It is also possible to configure a port channel as a vPC, where port channel uplink ports from a fabric interconnect are connected to different upstream switches. After all uplink ports are configured, create a port channel for these ports.
Each fabric interconnect is connected to IOMs in the UCS chassis, which provides connectivity to each blade server. Internal connectivity from blade servers to IOMs is transparently provided by Cisco UCS Manager using 10BASE-KR Ethernet standard for backplane implementations, and no additional configuration is required. You must configure the connectivity between the fabric interconnect server ports and IOMs. Each IOM, when connected with the fabric interconnect server port, behaves as a line card to fabric interconnect, hence IOMs should never be cross-connected to the fabric interconnect. Each IOM is connected directly to a single fabric interconnect.
The Fabric Extender (also referred to as the IOM, or FEX) logically extends the fabric interconnects to the blade server. The best analogy is to think of it as a remote line card that’s embedded in the blade server chassis, allowing connectivity to the external world. IOM settings are pushed via Cisco UCS Manager and are not managed directly. The primary functions of this module are to facilitate blade server I/O connectivity (internal and external), multiplex all I/O traffic up to the fabric interconnects, and help monitor and manage the Cisco UCS infrastructure.
Configure Fabric interconnect ports that should be connected to downlink IOM cards as server ports. Make sure there is physical connectivity between the fabric interconnect and IOMs. You must also configure the IOM ports and the global chassis discovery policy.
![]() Note | For UCS 2200 I/O modules, you can also select the Port Channel option and all I/O module-connected server ports will be automatically added to a port channel. |
You must configure the information policy to display the uplink switches that are connected to Cisco UCS.
You must enable the information policy on the fabric interconnect to view the SAN, LAN, and LLDP neighbors of the fabric interconnect.
![]() Note | By default, the information policy is disabled on the fabric interconnect. |
The following example shows how to enable the information policy on the fabric interconnect:
UCS-A# scope system UCS-A/system # scope info-policy UCS-A/system/info-policy # show Info Policy: State: Disabled UCS-A/system/info-policy # enable UCS-A/system/info-policy* # commit-buffer UCS-A/system/info-policy #
The following example shows how to disable the information policy on the fabric interconnect:
UCS-A# scope system UCS-A/system # scope info-policy UCS-A/system/info-policy # show Info Policy: State: Enabled UCS-A/system/info-policy # disable UCS-A/system/info-policy* # commit-buffer UCS-A/system/info-policy #
Command or Action | Purpose |
---|
The following example shows how to display the LAN neighbors of the fabric interconnect:
UCS-A # scope fabric-interconnect a UCS-Afabric-interconnect # show lan-neighbors Info Policy:Enabled Lan Neighbors: Local Interface: Ethernet1/2 Device Id: bgl-samc02-B(SSI140305YK) IPv4 Address: 10.105.214.105 FI Port DN: sys/switch-A/slot-1/switch-ether/port-2
Command or Action | Purpose |
---|
The following example shows how to display the SAN neighbors of the fabric interconnect :
UCS-A # scope fabric-interconnect a UCS-A/fabric-interconnect # show san-neighbors Info Policy: Enabled San neighbors: Local Interface: fc2/1 Port VSAN: 100 Fabric Mgmt Addr: 10.65.124.252 Fabric pwwn: 20:02:00:05:9b:22:ad:C0 Fabric nwwn: 20:64:00:05:9b:22:ad:C1 My pwwn: 20:41:00:0d:ec:ee:dd:00 My nwwn: 20:64:00:0d:ec:ee:dd:01 FI Port DN: sys/switch-A/slot-2/switch-fc/port-1
Command or Action | Purpose |
---|
The following example shows how to display the LLDP neighbors of the fabric interconnect :
UCS-A # scope fabric-interconnect a UCS-A/fabric-interconnect # show lldp-neighbors Info Policy: Enabled Lldp Neighbors: Local Interface: Eth1/5 Chassis Id: 000d.ecff.5e90 Remote Interface: Eth1/9 Remote Port Description: Ethernet1/9 System Name: bgl-samc02-B System Description: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc System Capabilities: B Enabled Capabilities: B Native VLAN: 1 IPv4 Mgmt Address: 10.105.214.105 FI Port DN: sys/switch-A/slot-1/switch-ether/port-5
Cisco UCS Manager introduces fabric evacuation, which is the ability to evacuate all traffic that flows through a fabric interconnect from all servers attached to it through an IOM or FEX while upgrading a system. Upgrading the secondary fabric interconnect in a system disrupts active traffic on the fabric interconnect. This traffic fails over to the primary fabric interconnect. Using fabric evacuation, you can stop all traffic that is active through a fabric interconnect. After verifying that the traffic has failed over completely and upgrading the secondary fabric interconnect, you can reboot the secondary fabric interconnect and all the connected IOMs. You can then restart all the stopped flows. In a cluster configuration, you can change the cluster lead to this subordinate FI, stop all the flows and upgrade the other fabric interconnect in the same way.
Upgrading the secondary fabric interconnect in a system disrupts active traffic on the fabric interconnect. This traffic fails over to the primary fabric interconnect. You can use fabric evacuation during the upgrade process as follows:
Stop all the traffic that is active through a fabric interconnect.
For vNICs configured with failover, verify that the traffic has failed over by using Cisco UCS Manager, or tools such as vCenter.
Upgrade the secondary fabric interconnect.
Restart all the stopped traffic flows.
Change the cluster lead to the secondary fabric interconnect.
Repeat steps 1 to 4 and upgrade the primary fabric interconnect.
![]() Note | Fabric evacuation is supported only with the following: |
This example shows how to stop all traffic that is active through Fabric Interconnect B:
UCS-A# scope fabric-interconnect b UCS-A /fabric-interconnect # stop server traffic Warning: Enabling fabric evacuation will stop all traffic through this Fabric Interconnect from servers attached through IOM/FEX. The traffic will fail over to the Primary Fabric Interconnect for fail over vnics. UCS-A /fabric-interconnect # commit-buffer
Command or Action | Purpose |
---|
![]() Note | Admin Evacuationand Oper Evacuation and show the status of evacuation at the fabric interconnect. |
UCS-A /fabric-interconnect # show detail Fabric Interconnect: ID: B Product Name: Cisco UCS 6248UP PID: UCS-FI-6248UP VID: V01 Vendor: Cisco Systems, Inc. Serial (SN): SSI171400HG HW Revision: 0 Total Memory (MB): 16165 OOB IP Addr: 10.193.32.172 OOB Gateway: 10.193.32.1 OOB Netmask: 255.255.255.0 OOB IPv6 Address: :: OOB IPv6 Gateway: :: Prefix: 64 Operability: Operable Thermal Status: Ok Admin Evacuation: On Oper Evacuation: On Current Task 1: Current Task 2: Current Task 3:
Command or Action | Purpose |
---|
![]() Note | Oper Evacuation shows the operational status of evacuation for the IOM. |
UCS-A# scope chassis 1 UCS-A /chassis # scope iom 1 UCS-A /chassis/iom # show detail IOM: ID: 1 Side: Left Fabric ID: A User Label: Overall Status: Fabric Conn Problem Oper qualifier: Server Port Problem Operability: Operable Presence: Equipped Thermal Status: OK Discovery: Online Config State: Ok Peer Comm Status: Connected Product Name: Cisco UCS 2204XP PID: UCS-IOM-2204XP VID: V02 Part Number: 73-14488-02 Vendor: Cisco Systems Inc Serial (SN): FCH1718J9FT HW Revision: 0 Mfg Date: 2013-05-12T00:00:00.000 Controller Subject: Iocard Fabric Port Aggregation Capability: Port Channel Oper Evacuation: On Current Task 1: Current Task 2:
Command or Action | Purpose |
---|
The following example shows the VIF (Virtual NIC) paths before fabric evacuation.
![]() Note |
UCS-A# show service-profile circuit server 1/6 Service Profile: test1 Server: 1/6 Fabric ID: A Path ID: 1 VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- --------- 692 eth0 Up Active Active Primary 0/0 1/15 Ether Fabric ID: B Path ID: 1 VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- --------- 693 eth0 Up Active Passive Backup 0/0 1/15 Ether UCS-A#
The following example shows the VIF paths after Fabric Interconnect A is evacuated.
![]() Note |
UCS-A# show service-profile circuit server 1/6 Service Profile: test1 Server: 1/6 Fabric ID: A Path ID: 1 VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- --------- 692 eth0 Error Error Active Primary 0/0 0/0 Ether Fabric ID: B Path ID: 1 VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- --------- 693 eth0 Up Active Passive Backup 0/0 1/15 Ether UCS-A#
This example shows how to restart traffic through Fabric Interconnect B:
UCS-A# scope fabric-interconnect b UCS-A /fabric-interconnect # start server traffic Warning: Resetting fabric evacuation will cause server traffic that failed over to the Primary Fabric Interconnect to fail back to this Fabric Interconnect. UCS-A /fabric-interconnect # commit-buffer
Unconfigured—Port is not configured and cannot be used.
Server Port—Port is configured for downlink connection to an IOM Fabric Extender (FEX) module in a blade chassis.
Uplink Port—Port is configured for uplink connection to the upstream Ethernet switch. Uplink ports are always configured as trunk ports.
Disabled—Port is configured either as an uplink or server port and is currently disabled by the administrator.
For 6200 series fabric interconnects, all ports are unified ports; therefore you also configure all the ports as 1/10 Gigabit Ethernet, Fibre Channel (FC), FC uplink, appliance port, or FCoE port.
For 6300 series fabric interconnects, see the UCS Manager Getting Started Guide.
The Cisco UCS Fabric Interconnects are operated in two main switching modes: Ethernet or Fibre Channel. These modes are independent of each other. They determine how the fabric interconnect behaves as a device between the server and network/server and storage device.
The Ethernet switching mode determines how the fabric interconnect behaves as a switching device between the servers and the network. The fabric interconnect operates in either of the following Ethernet switching modes:
End-host mode allows the fabric interconnect to act as an end host to the network, representing all servers (hosts) connected to it through vNICs. This behavior is achieved by pinning (either dynamically pinning or hard pinning) vNICs to uplink ports, which provides redundancy to the network, and makes the uplink ports appear as server ports to the rest of the fabric.
In end-host mode, the fabric interconnect does not run the Spanning Tree Protocol (STP), but it avoids loops by denying uplink ports from forwarding traffic to each other and by denying egress server traffic on more than one uplink port at a time. End-host mode is the default Ethernet switching mode and should be used if either of the following is used upstream:
![]() Note | When you enable end-host mode, if a vNIC is hard pinned to an uplink port and this uplink port goes down, the system cannot repin the vNIC, and the vNIC remains down. |
Switch mode is the traditional Ethernet switching mode. The fabric interconnect runs STP to avoid loops, and broadcast and multicast packets are handled in the traditional way. Use the switch mode only if the fabric interconnect is directly connected to a router, or if either of the following is used upstream:
![]() Note | For both Ethernet switching modes, even when vNICs are hard-pinned to uplink ports, all server-to-server unicast traffic in the server array is sent only through the fabric interconnect and is never sent through uplink ports. Server-to-server multicast and broadcast traffic is sent through all uplink ports in the same VLAN. |
While creating a port channel between a Cisco MDS 9000 family FC switching module and a Cisco UCS Fabric Interconnect in switch mode, use the following order:
Create the port channel on the MDS side.
Add the port channel member ports.
Create the port channel on the Fabric Interconnect side.
Add the port channel member ports.
If you create the port channel on the Fabric Interconnect side first, the ports will go into a suspended state.
When the Cisco UCS Fabric Interconnect is in switch mode, the port channel mode can only be in ON mode and not Active. However, to get the peer wwn information for the Fabric Interconnect, the port channel must be in Active mode.
When you change the Ethernet switching mode, Cisco UCS Manager logs you out and restarts the fabric interconnect. For a cluster configuration, Cisco UCS Manager restarts both fabric interconnects. The subordinate fabric interconnect reboots first as a result of the change in switching mode. The primary fabric interconnect reboots only after you acknowledge it in Pending Activities. The primary fabric interconnect can take several minutes to complete the change in Ethernet switching mode and become system ready. The existing configuration is retained.
While the fabric interconnects are rebooting, all blade servers lose LAN and SAN connectivity, causing a complete outage of all services on the blades. This might cause the operating system to fail.
The following example sets the fabric interconnect to end-host mode and commits the transaction:
UCS-A# scope eth-uplink UCS-A /eth-uplink # set mode end-host Warning: When committed, this change will cause the switch to reboot UCS-A /eth-uplink* # commit-buffer UCS-A /eth-uplink #
The Fibre Channel switching mode determines how the fabric interconnect behaves as a switching device between the servers and storage devices. The fabric interconnect operates in either of the following Fibre Channel switching modes:
End-host mode is synonymous with N Port Virtualization (NPV) mode. This mode is the default Fibre Channel Switching mode. End-host mode allows the fabric interconnect to act as an end host to the connected fibre channel networks, representing all servers (hosts) connected to it through virtual host bus adapters (vHBAs). This behavior is achieved by pinning (either dynamically pinning or hard-pinning) vHBAs to Fibre Channel uplink ports, which makes the Fibre Channel ports appear as server ports (N-ports) to the rest of the fabric. When in end-host mode, the fabric interconnect avoids loops by preventing uplink ports from receiving traffic from one another.
![]() Note | When you enable end-host mode, if a vHBA is hard-pinned to an uplink Fibre Channel port and this uplink port goes down, the system cannot repin the vHBA, and the vHBA remains down. |
Switch mode is not the default Fibre Channel switching mode. Switch mode allows the fabric interconnect to connect directly to a storage device. Enabling Fibre Channel switch mode is useful in Pod models where there is no SAN (for example, a single Cisco UCS domain that is connected directly to storage), or where a SAN exists (with an upstream MDS).
![]() Note | In Fibre Channel switch mode, SAN pin groups are irrelevant. Any existing SAN pin groups are ignored. |
![]() Note | When the Fibre Channel switching mode is changed, both Cisco UCS fabric interconnects reload simultaneously. Reloading the fabric interconnects will cause a system-wide downtime for approximately 10 to 15 minutes. |
The following example shows how to set the fabric interconnect to end-host mode and commit the transaction:
UCS-A # scope fc-uplink UCS-A /fc-uplink # set mode end-host UCS-A /fc-uplink* # commit-buffer UCS-A /fc-uplink #