Cisco Unified Computing System (UCS) is a unified computing solution
that fuses access layer networking and servers. This high-performance,
next-generation server system provides the data center with a high degree of
workload agility and scalability.
The hardware and software components in the UCS support the unified
fabric of Cisco, which allows multiple types of data center traffic over a
single physical Ethernet network. This Data Center Ethernet (DCE) technology
reduces the amount of cabling, management, and cost with the combination of the
host bus adapters (HBAs) and network interface cards (NICs) into a single
adapter called the Converged Network Adapter (CNA). This adapter can carry LAN
and SAN traffic on the same cable.
Cisco UCS uses Fibre Channel over Ethernet (FCoE) protocol to carry
Fibre Channel (FC) traffic inside Ethernet frame. Cisco UCS also adheres to
multiple 802.1 standards to provide the DCE underlying the FCoE needs to
effectively transport those frames. The fabric interconnect separates the LAN
and SAN traffic from the Ethernet frames and forwards them to the appropriate
network ports. This gives the flexibility to deploy this technology without the
need of implementing the unified fabric solution across the entire data center
Cisco UCS blade installed with Cisco UCS CNA M71KR – E Emulex Converged
Network Adapter or Cisco UCS CNA M71KR – Q QLogic Converged Network Adapter can
handle both FC and IP simultaneously. The converged network adapter presents an
Ethernet interface and an Fibre Channel interface to the operating system. The
OS is completely unaware of the encapsulation taking place in the Ethernet
segment. The only requirement is for the OS to have the appropriate drivers to
recognize the CNA hardware.
At the fabric interconnect, the server-facing Ethernet port receives
the Ethernet and Fibre Channel traffic. The fabric interconnect (which uses
Ethertype to differentiate the frames) separates the two traffic types.
Ethernet frames and Fibre Channel frames are switched to their respective
This document illustrates all the necessary steps to properly configure
FCoE connectivity for a Cisco UCS blade using either of the CNA's.
Cisco recommends that you:
Have a working knowledge of Cisco UCS Server Blade software and
Be familiar with the UCS Manager GUI.
Have a UCS system that runs version 1.0(1e) or
Understand the impact and implications of the different commands
described in this document
Be familiar with the UCS components and topology. Refer to the
diagram for a typical solution.
Ensure that you meet these requirements before you attempt this
The information in this document is based on these software and
The information in this document was created from the devices in a
specific lab environment. All of the devices used in this document started with
a default configuration. If your network is live, make sure that you understand
the potential impact of any command.
This is a typical topology used with the Cisco UCS:
This is the network diagram used in this test plan:
Refer to the
Technical Tips Conventions for more information on document
The Cisco UCS follows the FCoE protocol as defined by the ANSI T11
Standards Committee. The FC traffic encapsulated inside this Ethernet requires
the same lossless network characteristics that are found in a fabric network.
Instead of the buffer-to-buffer (B2B) credit system used in native fabric
topologies, the FCoE relies on a new set of Ethernet standards that were
develop to enhance the Ethernet protocol to ensure lossless transport of the
The Ethernet links on the system support these Ethernet enhancements to
ensure lossless transport for the FCoE traffic:
Priority Flow Control (PFC) IEEE 802.1Qbb is an extension of the
PAUSE (802.3x) mechanism. PFC creates eight virtual links in each physical link
and allows any of these links to be paused individually without affecting the
flow of traffic in the other links.
Enhanced Transmission Selection (ETS) IEEE 802.1Qaz is a scheduling
mechanism in hardware that allows a two-level Deficit Weighted Round Robin
(DWRR) with strict priority support. This allows control not only of bandwidth,
but also of latency.
Data Center Bridge eXchange (DCBX) is a discovery and capability
exchange protocol to verify that both ends are configured properly to support
the DCE traffic. It can provide basic configuration if one of the two sides is
not configured properly.
The fabric interconnect operates in N-Port Virtualization (NPV) mode
and not as a FC switch in the fabric. This means that it does not require a FC
domain ID to keep the number of domain IDs in the SAN fabric the same. The
fabric interconnect joins the fabric through a normal FLOGI. The FLOGIs that
comes from the server blade adapters is translated by the NPV process into
FDISC into the fabric.
Before you begin, make sure that your upstream MDS switches are NPIV
enabled, and assign the selected interface to the Cisco UCS with the
appropriate VSAN number.
The Cisco UCS must be up and operational by now and have basic
The Fabric Interconnect must be set in order to operate in FCoE mode.
Connect to the Fabric Interconnect and connect to nxos mode in order to verify
! Log into UCS Manager CLI mode
! Connect to "nxos" mode
cali-spring-A# connect nxos
Cisco UCS 6100 Series Fabric Interconnect
TAC support: http://www.cisco.com/tac
Copyright ©) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
! Make sure FCoE is supported on the switch,
by issuing "show fcoe"
cali-spring-A# show fcoe
FCoE/FC feature is desired.
Type “exit” to disconnect from nxos
Display of inventory will also show install of FC Modules into the switches
! exit from "nxos" mode to switch mode,
and making sure you are seeing the FC module
cae-sj-ca-B# show fabric-interconnect inventory expand
Slot Description Num Ports State PID Serial (SN)
----- -------------------- ---------- ---------- --------------- -----------
1 20x10GE/Supervisor 20 Online N10-S6100 JAF1242ASGK
2 8x1/2/4G FC Module 8 Online N10-E0080 JAB123800NM
Before you configure the fabric interconnect to support VSAN, you need
to make sure that your FC uplink expansion works and is visible from the fabric
interconnect. Complete these steps:
Log into UCS Manager GUI mode.
In the Navigation pane, choose
Choose Fabric Interconnect > Fabric Interconnect
Choose Uplink FC Ports.
The FC port defaults to VSAN 1 in Cisco UCS, but, if the upstream Fibre
Channel switch is connected to a different VSAN #, the proper VSAN
configuration is required on the FC ports of the Fabric Interconnect. Complete
In the Navigation pane, choose SAN.
Choose SAN Cloud.
Choose the FC Uplinks for the fabric in question (A or
Right-click VSANs in the FC Uplinks of the Fabric
# where you want to create the VSAN for Uplink.
In the Create VSAN wizard, enter a name for this new VSAN. In this
example, it is called vsan-100.
Now enter the VSAN ID. Remember that this has to
match to an existent VSAN ID in the fabric to which the UCS connects. In this
example, VSAN 100 is used for fabric A and VSAN 200 for fabric B.
Now enter the FCoE VLAN that maps to the VSAN
configured in this fabric. Remember that this VLAN has to match to an existent
VLAN in the LAN network. Click OK.
Click the VSAN name you created in order to display your
Now, repeat the same process for the uplink of Fabric B, except
with VSAN 200 and VLAN 200 for Fabric B, you get this result:
Now assign the VSAN you just created to the FC Uplink port in your
Fabric Interconnect that has a direct link to your SAN network. Complete these
In the Navigation pane, choose the Equipment
Choose Fabric Interconnect A > Expansion Module
Choose Uplink FC Ports and choose the FC Port 1
that has a connection to the upstream SAN switch.
In the Work pane to the right in VSAN, choose the VSAN you just
created from the drop-down menu.
Click Save Changes.
Repeat the same process to assign this VSAN to all the designated
FC Up link ports that remain on this same Fabric Interconnect.
In order to configure the second fabric, repeat steps 2 through 5.
Cisco recommends that you use a different VSAN and a different FCoE VLAN to
keep your fabrics separated. In this example, VSAN ID 200 and FCoE VLAN 200
Choose the FC port linked to your SAN network and assign this port to
the VSAN created in the previous step.
These FC Uplinks must already be provisioned with respective VSAN IDs
on the SAN network side (MDS switch).
In the Navigation pane, choose the SAN tab.
Expand SAN Cloud and then expand FC
Uplinks - Fabric A.
Expand Uplink FC Ports. In this test case, choose
FC Port 2/1 that has connection to the upstream SAN
Either right-click the desired FC port, or, in the Work pane to the
right, click the port as shown.
In the Properties pop-up window for FC Port 1, choose the VSAN to
which this FC port will pinned from the drop-down menu.
Click Save Changes.
Repeat the same process for all other FC ports in Fabric A and
Create two network adapters in a Service Profile to grant the Operating
System access to the two SAN fabrics provisioned in previous steps.
In the Navigation pane, choose the Server
Under Service Profiles, choose
In the work pane to the right, click Create Service Profile
In the Create Service Profile pop-up window, fill
in the Service Profile name. Click Next.
In Storage section, add the vHBAs that you have just configured on
Fabric Interconnect A and B. You can also give a name to these vHBA's. Make
sure that you have selected the correct WWNN pool for these
A vNIC is not assigned in this Service Profile for FCoE. Choose
No vNICs and click Next.
Choose the server boot order policy to use and click
Next. If you do not want to choose any policy, then click
In the Server Assignment window, choose the server pool to use and
click Next. If you do not want to choose a server pool now,
Choose the IPMI profile and the SoL Configuration profile that you
wnat to apply to this policy and click Finish. If you do not
want to apply and IPMI or SoL profile, then click
The results are successful.
The created vHBAs are visible in the vHBA tab of your created
Refer to the verification process in the previous section.
There is currently no specific troubleshooting information available
for this configuration.