Overview of Cisco UCS Fabric Interconnects

Cisco UCS 6000 Series Fabric Interconnects

Overview of Cisco UCS 6664 Fabric Interconnect

The Cisco UCS 6664 Fabric Interconnect is a 2-rack unit (RU), fixed-port system designed for Top-of-Rack deployment in data centers. The fabric interconnect has both Ethernet and unified ports. Unified ports provide Fibre Channel over Ethernet (FCoE), Fibre Channel, NVMe over Fabric, and Ethernet. By supporting these different protocols, you can use a single multi-protocol Virtual Interface Card (VIC) in your servers.

The UCS 6664 Fabric Interconnect supports an array of Gigabit Ethernet (GbE), Fibre Channel (FC), and Fibre Channel over Ethernet (FCoE) ports to offer connectivity to peer data center devices. This device is also ideal for high-performance, scalable, and secure networking in modern data centers.

The Cisco UCS 6664 Fabric Interconnect includes:

  • 64 total ports

    • 48 ports of 40/100 Gbps

    • 16 unified ports

      • Supports 10/25 Gbps for Ethernet and Fibre Channel over Ethernet (FCoE)

      • Supports 10/25 Gbps or 16/32/64 Gbps Fibre Channel for maximum flexibility

  • 2RU fixed form factor for dense 100 Gbps connectivity

  • Fibre Channel end-host and switch-mode support

Cisco UCS 6664 Fabric Interconnect

The Cisco UCS 6664 Fabric Interconnect (UCSC-FI-6664) is a 2-rack unit (2RU), fixed-port system designed for Top-of-Rack (TOR) deployment in data centers. It serves as a central component in the Cisco UCS system, providing both network connectivity and management capabilities. It connects servers to the LAN or SAN, facilitating a unified computing environment. Management of the fabric interconnect can be performed through its console interface for initial setup, and subsequently via its management interface within Cisco UCS Manager.

Figure 1. Cisco UCS 6664 Fabric Interconnect Front View

Key features of Cisco UCS 6664 Fabric Interconnect:

  • Ports

    48 QSFP ports: These ports are divided into two groups:

    • QSFP Ports 1-24: Support Gigabit Ethernet links at speeds of 10 Gbps (through QSA transceivers), and 25 Gbps, 40 Gbps, and 100 Gbps (through QSA28 transceivers). FCoE links are also supported, but standard FC links are not.

    • QSFP Ports 41-64: Support only Gigabit Ethernet or Fibre Channel over Ethernet (FCoE) links at speeds of 10 Gbps (through QSA transceivers), and 25 Gbps, 40 Gbps, or 100 Gbps (through QSA28 transceivers). Standard Fibre Channel links are not supported on these ports.

  • 16 Unified Ports (SFP ports 25-40): These ports, indicated by yellow silk screening, are flexible and support:

    • Fibre Channel: 16 Gbps, 32 Gbps, or 64 Gbps. These are the only ports on the Fabric Interconnect that support standard FC traffic.

    • Gigabit Ethernet or Fibre Channel over Ethernet (FCoE): 10 Gbps or 25 Gbps.

  • MACsec-capable Ports: QSFP ports 49-64 are recommended as uplinks and support Media Access Control Security (MACsec) on Gigabit Ethernet links, but not FCoE links.

  • Breakout Support: Breakout port support for this fabric interconnect currently not available.

  • Secure Boot Support

  • Fan modules (4): These are port-side exhaust fan modules (NXASFAN-160CFM2-PE) with blue coloring, providing redundant and load-sharing cooling.

  • Power supply modules (2): Two 1400-W AC power supply modules (NXA-PAC-1400W-PE) are provided for 1+1 redundancy. These also feature port-side exhaust airflow with blue coloring.

Figure 2. Cisco UCS 6664 Fabric Interconnect Front Panel View

1

Beacon (BCN), Status (STS), and Environment (ENV) LEDs

2

QSFP Ports 1, 2, 5, 6, 9,10, 13, 14, 17, 18, 21, and 22. Ports are arranged vertically in pairs and support QSA or QSA28 transceivers.

3

SFP Ports 25, 26, 29, 30, 33, 34, 37, and 38 are Unified Ports

4

QSFP Ports 41, 42, 45, 46, 49, 50, 53, 54, 57,58, 61, and 62.

5

MACsec-capable QSFP ports 49, 50, 53, 54, 57, 58, 61, and 62.

6

MACsec-capable QSFP ports 51, 52, 55, 56, 59, 60, 63, and 64.

7

QSFP Ports 43, 44,47, 48, 51, 52, 55, 56, 59, 60, 63, and 64.

8

SFP Ports 27, 28, 31, 32, 35, 36, 39, and 40 are Unified Ports

9

QSFP Ports 3, 4, 7, 8, 11,12, 15, 16, 19, 20, 23, and 24. Ports are arranged vertically in pairs and support QSA or QSA28 transceivers.

Figure 3. Cisco UCS 6664 Fabric Interconnect Rear Panel View

1

Management Module

2

Fan Modules, numbered one through four with fan 1 on the left and fan four on the right.

3

Power Supply 1

4

Power Supply 2

Cisco UCS 6500 Series Fabric Interconnects

Cisco UCS 6536 Fabric Interconnect Overview

The Cisco UCS 6536 Fabric Interconnect is a core part of the Cisco Unified Computing System, providing both network connectivity and management capabilities for the system. The Cisco UCS 6536 Fabric Interconnect provides the communication backbone and management connectivity for UCS B-series blade servers and UCS C-series rack servers.

Cisco UCS 6500 Series Fabric Interconnects currently include Cisco UCS 6536 Fabric Interconnect. All servers attached to a Cisco UCS 6536 Fabric Interconnect become part of a single, highly available management domain. In addition, by supporting a unified fabric, Cisco UCS 6536 Fabric Interconnect provides both LAN and SAN connectivity for all servers within its domain.

The Cisco UCS 6536 Fabric Interconnect supports multiple traffic classes over a lossless Ethernet fabric from the server through the fabric interconnect.

Cisco UCS 6536 Fabric Interconnect

The Cisco UCS 6536 Fabric Interconnect (UCSC-FI-6536) is a One-rack unit (1RU), top of rack (TOR), fixed-port data center platform that provides both network connectivity and management capabilities to the Cisco UCS system.

The fabric interconnect can provide Ethernet and Fibre Channel connectivity to the servers in the system. The servers connect to the fabric interconnect, and then to the LAN or SAN.

Cisco UCS Manager on UCS 6536 Fabric Interconnect supports Cisco UCS X-Series Compute Nodes, B-Series Blade Servers and C-Series Rack Servers, Cisco UCS S-Series Storage Servers, as well as the associated storage resources and networks.

High availability and redundancy can be achieved by connecting a pair of fabric interconnects to each other through L1 or L2 ports in cluster mode configuration.

Each Cisco UCS 6536 Fabric Interconnect offers the following features:

  • Thirty-six QSFP28 ports capable of 100G including 4 unified ports (33-36). Ports also support:

    • Autonegotiating with peer devices to speeds of 100G, 40G, 25G, 10G, and 1G.

    • Port breakout is supported for Ethernet ports (1-32) and Unified ports (33-36).

      • Ethernet breakout is supported on switch ports 1 through 36 when each port is configured with a breakout cable.

        • The Dynamic Ethernet Breakout feature enables converting a standard Ethernet port to a breakout port on-the-fly so that you do not need to reboot the Fabric Interconnect. Dynamic Ethernet Breakout also supports converting breakout ports back to a standard Ethernet port without a reboot.

      • FC breakout is supported on ports 33 through 36 wherein each port is configured with a four-port breakout cable. For example 1/33/1, 1/33/2, 1/33/3, and 1/33/4 are the four FC breakout ports on the physical port 33.


        Note


        Fibre Channel support is only available through the configuration of Unified Ports (33-36) as FC breakout port.


        • FC breakout ports support peer communication at fixed speeds of 8Gbs, 16 Gbps, and 32 Gbps.

        • All four FC breakout ports must be configured with the same speed. Mixed speeds on a QSFP port's FC breakout ports are not supported.

        • Using breakout ports enables the fabric interconnect to support the maximum 16 FC ports supported by Fibre Channel.


      Note


      • Converting from Ethernet to FC breakout ports, or FC breakout ports back to Ethernet, requires a reboot/reload after changing the breakout type.

      • FCoE storage ports are not supported.


  • One management port (one 10/100/1000BASE-T port)

  • Two L1/L2 Ethernet RJ-45 ports for high availability or cluster configurations. Ethernet ports support 10/100/1000Mb speed.

  • One console port (RS-232)

  • One USB 3.0 port

  • CPU: 4 Core, 1.8GHz, Intel 5th-Generation core processor

  • Memory:

    • 32 GB DDR4 DIMMs

    • 128 GB M.2 SSD Flash Drive

    • 32 GB Boot Flash (16 MB primary, and 16 MB standby/golden)

This fabric interconnect includes the following user-replaceable components:

  • Fan modules (6), each is a port-side exhaust fan module with dark grey latch coloring (UCS-FAN-6536).

  • Power supply modules (2). One power supply module (PSU) is the active module for operations, and the second PSU is the standby for redundancy [1+1]) with the following choices:

    • 1100-W AC power supply with dark grey latch coloring (UCS-PSU-6536-AC)


    Note


    All fan modules and power supplies must use the same airflow direction.


The following figure shows the fabric interconnect features on the port side of the chassis.

1

LEDs

3

36 40/100-Gigabit QSFP28 ports

2

Lane Select button

To determine which transceivers, adapters, and cables are support the fabric interconnect, see the Cisco Transceiver Modules Compatibility Information document.

The following figure shows the fabric interconnect features on the power supply side of the chassis.

1

Power supply modules (1 or 2) (AC power supplies shown) with slots numbered 1 (left) and 2 (right).

2

Fan modules (6) with slots numbered from 1 (left) to 6 (right).

3

Layer 2 (L2) Ethernet port, 10/100/100Mb autonegotiating.

Supports high availability (HA) or clustering through an RJ-45 port.

4

Layer 1 (L1) Ethernet port, 10/100/100Mb autonegotiating.

Supports high availability (HA) or clustering through an RJ-45 port.

5

Ethernet network management port (RJ45), 10/100/1000Mb autonegotiating

6

Serial Console port (RJ45), 9600 baud.

7

USB 3.0/2.0 port

Supports booting the system and downloading scripts.

8

Beacon (BCN) LED

9

Status (STS) LED

-

The following figure shows the side of the chassis.

1

Screw holes for mounting brackets

2

Grounding pad

Plan to position the ports in a hot aisle so that fans and power supplies intake air from the cold aisle, blow the cool air through the fabric interconnect, and exhaust the heated air into the hot aisle.

The fan and power supply modules are field replaceable. You can replace one fan module or one power supply module during operations so long as the other modules are installed and operating. If you have only one power supply installed, you can install the replacement power supply in the open slot before removing the original power supply.


Note


All fan and power supply modules must have the same direction of airflow. Otherwise, the fabric interconnect can overheat and shut down.



Caution


Because fans and power supply modules have port-side exhaust airflow (blue coloring for fan modules), you must locate the ports in the hot aisle. If you locate the air intake in a hot aisle, the fabric interconnect can overheat and shut down.


Cisco UCS 6400 Series Fabric Interconnects

Cisco UCS 6400 Series Fabric Interconnect Overview

Cisco UCS 6400 Series Fabric Interconnect provides both network connectivity and management capabilities to the Cisco UCS system. The fabric interconnect provides Ethernet and Fibre Channel to the servers in the system, the servers connect to the fabric interconnect, and then to the LAN or SAN.

Each Cisco UCS 6400 Series Fabric Interconnect runs Cisco UCS Manager to fully manage all Cisco UCS elements. The fabric interconnect supports 10/25 Gigabit ports in the fabric with 40/100 Gigabit uplink ports. High availability can be achieved when a Cisco UCS 6400 Series Fabric Interconnect is connected to another Cisco UCS 6400 Series Fabric Interconnect through the L1 or L2 port on each device.

Cisco UCS 6400 Series Fabric Interconnect consists of:

  • Cisco UCS 6454 Fabric Interconnects

  • Cisco UCS 64108 Fabric Interconnects

The Cisco UCS 6400 Series fabric interconnect supports Cisco UCS B-Series Blade Servers, UCS 5108 B-Series Server Chassis, C-Series Rack Servers, and UCS S-Series Storage Servers.

Cisco UCS 64108 Fabric Interconnect

The Cisco UCS 64108 Fabric Interconnect is a 2 RU top-of-rack (TOR) switch that mounts in a standard 19-inch rack such as the Cisco R Series rack.

The high-density Cisco UCS 64108 Fabric Interconnect has 96 10/25 Gb SFP28 ports and 12 40/100 Gb QSFP28 ports. Each 40/100 Gb port can break out into 4 x 10/25 Gb uplink ports. Ports 1 - 16 are unified ports that support 10/25 GbE or 8/16/32G Fibre Channel speeds. Ports 89-96 support 1Gbps Ethernet speeds.

The Cisco UCS 64108 Fabric Interconnect supports either:

  • Eight FCoE port channels

  • Or Four SAN port channels

  • or Four SAN port channels and four FCoE port channels

The Cisco UCS 64108 Fabric Interconnect also has one network management port, one RS-232 serial console port for setting the initial configuration, and one USB port for saving or loading configurations. The FI also includes L1/L2 ports for connecting two fabric interconnects in a high-availability configuration.

The Cisco UCS 64108 Fabric Interconnect also contains a CPU board that consists of:

  • Intel Xeon Processor, 6 core

  • 64 GB of RAM

  • 8 MB of NVRAM (4 x NVRAM chips)

  • 128 GB SSD (bootflash)

Figure 4. Cisco UCS 64108 Fabric Interconnect Rear View

1

Ports 1-16

Unified ports:

  • 10/25 Gbps Ethernet or FCoE

  • 8/16/32 Gbps Fibre Channel

2

Ports 17-88

(10/25 Gbps Ethernet or FCoE)

3

Ports 89-96

  • 10/25 Gbps Ethernet or FCoE

  • 1 Gbps Ethernet

4

Uplink Ports 97-108 (40/100 Gbps Ethernet or FCoE)

Each of these ports can be 4 x 10/25 Gbps Ethernet or FCoE uplink ports when using a breakout cable.

5

System environment (fan fault) LED

6

System status LED

7

Beacon LED

The Cisco UCS 64108 Fabric Interconnect has two power supplies (redundant as 1+1) and three fans (redundant as 2+1).

Figure 5. Cisco UCS 64108 Fabric Interconnect Front View

1

Cooling fans

(hot swappable, 2+1 redundancy)

2

RS-232 serial console port (RJ-45 connector)

3

Network management port (RJ-45 connector)

4

USB port

5

Grounding pad for two-hole grounding lug (under protective label)

6

Power supplies

Two identical AC or DC PSUs, hot-swappable, 1+1 redundancy)

7

L1/L2 high-availability ports (RJ-45 connector)

8

Beacon LED

9

System status LED

Cisco UCS 6454 Fabric Interconnect

The Cisco UCS 6454 Fabric Interconnect (FI) is a 1-RU top-of-rack switch that mounts in a standard 19-inch rack such as the Cisco R Series rack.

The Cisco UCS 6454 Fabric Interconnect has 48 10/25 Gb SFP28 ports (16 unified ports) and 6 40/100 Gb QSFP28 ports. Each 40/100 Gb port can break out into 4 x 10/25 Gb uplink ports. The sixteen unified ports support 10/25 GbE or 8/16/32G Fibre Channel speeds.


Note


The Cisco UCS 6454 Fabric Interconnect supported 8 unified ports (ports 1 - 8) with Cisco UCS Manager 4.0(1) and 4.0(2), but with release 4.0(4) and later it supports 16 unified ports (ports 1 - 16).


The Cisco UCS 6454 Fabric Interconnect supports:

  • Maximum of 8 FCoE port channels

  • Or 4 SAN port channels

  • Or a maximum of 8 SAN port channels and FCoE port channels (4 each)

The Cisco UCS 6454 Fabric Interconnect also has one network management port, one console port for setting the initial configuration, and one USB port for saving or loading configurations. The FI also includes L1/L2 ports for connecting two fabric interconnects for high availability.

The Cisco UCS 6454 Fabric Interconnect also contains a CPU board that consists of:

  • Intel Xeon D-1528 v4 Processor, 1.6 GHz

  • 64 GB of RAM

  • 8 MB of NVRAM (4 x NVRAM chips)

  • 128 GB SSD (bootflash)

Figure 6. Cisco UCS 6454 Fabric Interconnect Rear View

1

Ports 1-16 (Unified Ports 10/25 Gbps Ethernet or FCoE or 8/16/32 Gbps Fibre Channel)

Note

 

When using Cisco UCS Manager releases earlier than 4.0(4), only ports 1-8 are Unified Ports.

2

Ports 17-44 (10/25 Gbps Ethernet or FCoE)

Note

 

When using Cisco UCS Manager releases earlier than 4.0(4), ports 9-44 are 10/25 Gbps Ethernet or FCoE.

3

Ports 45-48 (1/10/25 Gbps Ethernet or FCoE)

4

Uplink Ports 49-54 (40/100 Gbps Ethernet or FCoE)

Each of these ports can be 4 x 10/25 Gbps Ethernet or FCoE uplink ports when using an appropriate breakout cable.

The Cisco UCS 6454 Fabric Interconnect chassis has two power supplies and four fans. Two of the fans provide front to rear airflow.

Figure 7. Cisco UCS 6454 Fabric Interconnect Front View

1

Power supply and power cord connector

2

Fans 1 through 4, numbered left to right, when facing the front of the chassis.

3

L1 port, L2 port, RJ45, console, USB port, and LEDs

Ports on the Cisco UCS Fabric Interconnects

Cisco UCS Fabric Interconnect Port Overview

Cisco UCS Fabric Interconnects offer flexible and high-performance networking solutions, supporting various port configurations for Ethernet, Fibre Channel (FC), and Fibre Channel over Ethernet (FCoE) traffic. The specific capabilities vary by model and generation.

Port Details by Fabric Interconnect Model

This section details the unique port configurations and capabilities of each Cisco UCS Fabric Interconnect model.
  • Cisco UCS 6664 Fabric Interconnect

    Cisco UCS 6664 Fabric Interconnect features 64 ports, designed for flexible and high-performance networking:

    • Ports 1-24: supports 40/100 Gbps for Ethernet, FCoE, and storage connectivity.

    • Ports 25-40 (Unified Ports): supports configuration of 10/25Gbps Ethernet or 16/32/64Gbps Fibre Channel (FC).

    • Ports 41-64: supports configuration of 40/100 Gbps Ethernet with MACsec security (ports 49-64).

  • Cisco UCS Fabric Interconnets 9108 100G (Cisco UCS X-Series Direct)

    • Ports 1 & 2: Functions as unified ports to manage all SAN features and configurations.

    • 100G Ethernet ports [1-8]: Can be configured as 25Gx4 SFP28 compatible breakout ports or 4x10G ports, offering flexible networking solutions.

    • 32G Fibre Channel ports [1 & 2] : Can also be configured as 8Gx4 SFP28 compatible breakout ports for enhanced Fibre Channel connectivity.

  • Cisco UCS 6536 Fabric Interconnect

    Ports on the Cisco UCS 6536 Fabric Interconnects can be configured to carry either Ethernet or Fibre Channel traffic. You can configure only ports 33-36 to carry Fibre Channel traffic. The ports cannot be used by a Cisco UCS domain until you configure them.

  • Cisco UCS 6400 Series Fabric Interconnect

    Ports on the Cisco UCS 6400 Series Fabric Interconnects can be configured to carry either Ethernet or Fibre Channel traffic. The Cisco UCS 6400 Series Fabric Interconnect supported 8 unified ports (ports 1 - 8) with Cisco UCS Manager 4.0(1) and 4.0(2), but with Release 4.0(4) and later releases, it supports 16 unified ports (ports 1 - 16).

    Ports 1-16 can be configured to carry Fibre Channel traffic. The ports cannot be used by a Cisco UCS domain until you configure them.

Configuration Notes

When you configure a port on a Fabric Interconnect, the administrative state is automatically set to enabled. If the port is connected to another device, this may cause traffic disruption. The port can be disabled and enabled after it has been configured.

The following table summarizes the port support the Cisco UCS Fabric Interconnects:

Fourth Generation

Fifth Generation

Cisco UCS X-Series Direct

Sixth Generation

Item

Cisco UCS 6454

Cisco UCS 64108

Cisco UCS 6536

Cisco UCS Fabric Interconnect 9108 100G

Cisco UCS 6664 Fabric Interconnect

Description

54-Port Fabric Interconnect

108-Port Fabric Interconnect

36-Port Fabric Interconnect

8 Ports

64 Ports

Form factor

1 RU

2 RU

1 RU

1 RU

2 RU

Number of fixed 10 GB Interfaces

48 10G/25G interfaces

96 10G/25G interfaces

36 10G/25G/40G/100G interfaces

Note

 

144 breakout ports (36x4)

  • 16x 10G/25G ports

  • 48x 100G ports will support 10/25G using QSA/QSA28 modules

Number of Unified Ports

16

This FI supported 8 unified ports (ports 1 - 8) with Cisco UCS Manager 4.0(1) and 4.0(2), but with Release 4.0(4) and later it supports 16 unified ports (ports 1 - 16).

16

ports 1-16

4

Note

 

16 breakout ports (4x4)

Ports 1-2

16

Unified Port Speeds in Gbps

10G/25G or 8G/16G/32G-FC

10G/25G or 8G/16G/32G-FC

10G/25G/40G/100G or 8G/16G/32G-FC

10G/25G/40G/100G or 8G/16G/32G-FC

10G/25G or 16G/32G/64G-FC

Number of 40-Gbps ports

6 40G/100G

12 40G/100G

36

Unified Port Range

Ports 1-16

Ports 1-16

Ports 33-36

Ports 1-2

Ports 25-40

Compatibility with the IOM

UCS 2408

UCS 2408

UCS 2408

UCS 2408

Compatibility with the FEX

Cisco Nexus 93180YC-FX3

Cisco Nexus 93180YC-FX3

Cisco Nexus 93180YC-FX3

Cisco Nexus 93180YC-FX3

Compatibility with IFM

Cisco UCS 9108 25G

Cisco UCS 9108 25G

Cisco UCS 9108 25G

Cisco UCS 9108 100G

Cisco UCS 9108 25G

Cisco UCS 9108 100G

Cisco UCS 9108 25G

Cisco UCS 9108 100G

Expansion Slots

None

None

None

None

Fan Modules

4

3

6

3

4

Power Supplies

2 (AC/DC)

2 (AC/DC)

2 (AC)

Supplied by chassis

2 (AC)

Migration Limitations:

Migration from any generation Cisco UCS Fabric Interconnects (including fourth, fifth, or sixth generation) to Cisco UCS Fabric Interconnects 9108 100G (Cisco UCS X-Series Direct) is currently not supported.