Fibre Channel over Ethernet Operations

Preserving SAN Fabric Isolation

High availability (HA) is a requirement in any data center design—whether it is accomplished through HA at the port level, supervisor level, or even at the physical network level. Fibre Channel Storage Area Networks (FC SANs) achieve high availability by building out two identical but physically separate networks commonly referred to as SAN A and SAN B (also called Fabric A and Fabric B). These networks, unlike Data Center LAN networks, are completely physically isolated from one another and have no knowledge of each other. Depending on host operating systems and drivers, traffic is able to be load balanced or “multi-pathed” between the two isolated networks, from the application side, in order to provide better service to the storage traffic. This required isolation is an important element in building FCoE networks along side the data center Ethernet LANs.

This section includes the following methods on preserving fabric isolation:

• Maintaining Different FC-MAPs Per Fabric
• VLAN to VSAN Numbering

Maintaining Different FC-MAPs Per Fabric

Note We recommend using the “VLAN to VSAN Numbering” section method for preserving fabric isolation and leaving the FC-MAP default.

FC-MAP is a characteristic of a FCoE switch that identifies which fabric the switch belongs to. For instance, there can be an FC-MAP for Fabric A and a different FC-MAP for Fabric B. By configuring a specific FC-MAP value on a FCoE switch, it is possible to designate certain switches to belong to one fabric or another.

In order to maintain fabric isolation in an FCoE environment, it is recommended to use different FC-MAP values per SAN Fabric. Because the FC-MAP value of the Cisco Nexus 5000 Series switch is used in the addressing for FCoE-enabled devices, changing the FC-MAP value is a disruptive process to all hosts that are logged into the switch. Due to this disruption, it is recommended that the FC-MAP is configured as part of the initial switch set up.

By default, when the feature fcoe command is used to enable FCoE on a Cisco Nexus 5000 Series switch, a default FC-MAP is assigned to the switch. The simplest way to ensure SAN A and SAN B isolation between FCoE-enabled switches in the Ethernet fabric is to change the FC-MAP value to something other then the default for all switches belonging to Fabric B. This will prohibit FCoE switches from joining the wrong fabric and aide to providing the SAN isolation that is a requirement for FC and FCoE traffic.

To change the FC-MAP of a switch:

Note Changing the FC-MAP value of a switch is disruptive to all attached FCoE hosts and it requires the hosts to login to the fabric again. Therefore, it is recommended to change the FC-MAP when the switch is installed and initially configured or during a maintenance window.

Note The default value of the FC-MAP on a Cisco Nexus 5000 Series switch is 0E.FC.00. The configurable values for FC-MAP ranges from OE.FC.00 to 0E.FC.FF.

VLAN to VSAN Numbering

When configuring an FCoE fabric, the first step is to create a VLAN to VSAN mapping which allows the FC traffic in a single VSAN to traverse the Ethernet network. It is a best practice to have dedicated VLANs for FCoE traffic in order to separate the storage traffic from all other Ethernet VLANS. It is also recommended not to assign VLAN 1, VSAN 1 or the configured native VLAN to the FCoE network. Typically those VLAN/VSANs are utilized for management traffic or for devices that have no other VLAN or VSAN assigned to them. Using VLAN 1 as an FCoE VLAN will not be supported on the Cisco Nexus 5000 Series switch running Cisco NX-OS release 5.0(1)N1(2) or a later release.

VLAN to VSAN mapping is a one-to-one relationship. Mapping multiple VSANs to a single VLAN instance is not supported. Note that both the VLAN instance and VSAN instance in an FCoE VLAN/VSAN mapping take up a hardware VLAN resource. Currently, there can be up to 31 VLAN/VSAN mappings supported on the Cisco Nexus 5000 Series switch. VLAN and VSAN numbering can range from 1-4096.

FCoE VLANS are different from typical Ethernet VLANs in that it acts more of a container for the storage traffic than anything else. MAC learning, broadcasts, or flooding do not occur and it does not map to a subnet. FCoE VLANs are simply used to carry the traffic for a specified FC VSAN and keep it separate from any other Ethernet VLANs that may be traversing the network.

In order to avoid confusion and service disruption in the event of a misconfiguration, it is recommended that you configure different FCoE VLAN and VSAN numbers for both SAN A and SAN B. Using the same VLAN or VSAN numbering between the two fabrics could result in the merging of both SAN fabrics in the event of a miss-configuration or miss-cabling. It is also best practice to only define SAN A VLANs on SAN A switches and vice-versa.

Host-facing FCoE ports must be configured as trunk ports carrying the native VLAN, FCoE VLAN and any other Ethernet VLANs necessary for the host application. These host facing ports should also be configured as spanning tree edge ports using the spanning-tree port type edge [ trunk ] interface-level command.

Note ● FCoE Initialization Protocol (FIP) uses the native VLAN and therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the native VLAN.

• The FCoE VSAN must be configured and in the VSAN database of the Cisco Nexus 5000 Series switch prior to mapping it to a VLAN
• Enabling FCoE on VLAN 1 is NOT supported

FCoE and Spanning Tree Protocol Considerations

Native FC has no concept of a looped environment and therefore has no need for a protocol similar to the Spanning Tree Protocol (STP) in the Ethernet world. However, when placing FCoE onto an Ethernet fabric, STP is run on the FCoE VLANs connecting to a host (VF port) over a lossless Ethernet cloud. This lossless cloud could be made up of DCB bridges or FIP snooping devices. Because of this, there are certain recommendations for STP configurations that should be followed when deploying FCoE. The goal is to have isolated STP topologies between SAN A, SAN B, and the Ethernet fabric. This eliminates any Ethernet topology changes from affecting storage traffic.

Note STP is not run on FCoE VLANs on VE port connections between two FCFs.

Note Beginning with Cisco NXOS Release 5.0(1)N1(1) for the Cisco Nexus 5000 Series switch, STP is not run on FCoE VLANs on VF ports connecting directly to attached hosts (including host connections to a Cisco Nexus 2232 Fabric Extender). STP will continue to run on VF ports that connect to hosts through a DCB cloud or FIP snooping device.

Note In Cisco NXOS Release 4.2(1)N2(1a) and earlier releases, STP runs on FCoE VLANs for any VF port connection (either direct attached hosts or hosts connected over a DCB cloud). Because of this, it is required to configure the VF port as a spanning-tree port type edge trunk

This section includes the following topics:

• MST Instances For Dual Fabric FCoE Deployments
• PVST+ for Dual Fabric FCoE Deployments

MST Instances For Dual Fabric FCoE Deployments

When running multi-instance STP in an Ethernet environment, it is required that all switches in the same MST region have the identical mapping of VLANs to instances. This does not require that all VLANs be defined on all switches. When running FCoE over an environment using MST, it is recommended to have a dedicated MST instances for the FCoE VLANs belonging to SAN A and a dedicated MST instance for the FCoE VLANs belonging to SAN B. These instances should be separate from any instances that include regular Ethernet VLANs. This example shows the FCoE VLANS in Fabric A are VLANs 20-29 and the FCoE VLANS in Fabric B are VLANs 30-39:

Spanning-tree MST configuration:

• name FCoE-Fabric
• revision 5
• instance 5 vlan 1-19,40-3967,4048-4093
• instance 10 vlan 20-29
• instance 15 vlan 30-39

In the above configuration, instance 5 maps to native Ethernet VLANs, instance 10 maps to the VLANs for Fabric A (20-29) and instance 15 maps to the VLANs for Fabric B (30-39).

Due to the MST configuration requirement, it will be necessary to have the same MST configuration, containing both the SAN A and SAN B instance, on all switches within the same MST region. This means that switches participating in SAN A will also contain an MST configuration with a separate instance for SAN B VLANs even though those SAN B VLANs will not be defined on the SAN A switch.

PVST+ for Dual Fabric FCoE Deployments

When running PVST, each VLAN already has its own spanning tree topology. Because FCoE traffic in each SAN fabric is defined by different individual VLANs, PVST+ will automatically isolate the spanning tree domains for the VLANs in SAN A, SAN B, as well as the Ethernet fabric.

FCoE and Virtual Port Channel (vPC) Considerations

Virtual Port Channeling (vPC) is an Ethernet feature that allows a single device to connect to multiple upstream devices and forward out all available links without the implications of spanning tree blocking paths due to Ethernet loops. vPC is useful in three situations:

1. Connecting a server to two upstream switches

2. Connecting a FEX to two upstream Nexus 5X00s

3. Connecting a switch to two upstream switches

The upstream switches in all scenarios must support the virtual port channel feature. The downstream device has no knowledge of the vPC and simply views the connection as a standard Ethernet port channel.

Though it is not possible to run FCoE traffic on top of a vPC because of the SAN A and SAN B physical isolation requirement in native FC, it is possible to run FCoE and vPC side-by-side on the same physical infrastructure from the host to the first-hop FCoE device. To configure this topology, the following must be considered:

• A host must connect to the upstream Cisco Nexus 5000 Series vPC pair switches using only 2 10G links – one attaching to a Cisco Nexus 5000 Series switch in Fabric A and one attaching to a Cisco Nexus 5000 Series switch in fabric B. This is commonly referred to a single-port vPC because only one port goes to each switch.
• Generation 2 CNAs are required in the host in order to support vPC topologies.

Note ● FCoE and vPC can run side-by-side only on single-port host-connected vPCs. FCoE and vPC’s between a FEX and a Cisco Nexus 5000 Series switch or between two layers of switches is not supported.

• FCoE and vPCs containing more that one link to each access device is not supported. vPCs which coexist with FCoE must contain only a single link to each vPC peer device.
• vPC’s across switches (FCFs) within the same SAN fabric is not supported. Each vPC peer must be part of different fabrics—one peer in SAN A and one peer in SAN B.

This section includes the following topics:

• Required Teaming Drivers for vPC With CNAs
• Second Generation CNA Requirement
• View Of Ethernet Traffic And FC Traffic Through A CNA
• FCoE VLAN Configuration On A vPC

Required Teaming Drivers for vPC With CNAs

When connecting a host to an upstream vPC switch pair, the only requirement from the host side is to support link aggregation on the NIC interfaces. This can be accomplished using link aggregation control protocol (LACP) or standard 802.3ad port channel mode on behavior. It is important to check that either the host operating system or the native CNA hardware supports one of these options.

Second Generation CNA Requirement

When connecting a host containing a CNA to upstream Cisco Nexus 5000 Series switches configured in a vPC, 2nd generation CNAs are required from both Emulex and QLogic. This is regardless of the presence of FCoE traffic on the host connections. These 2nd generation CNAs are also required when connecting to a Cisco Nexus 2232 Fabric Extender with a vPC (Ethernet only), FCoE, or FCoE+vPC configuration from a host connection.

View Of Ethernet Traffic And FC Traffic Through A CNA

Currently CNAs present two different types of adapters to the host operating system: Ethernet NICs and Fibre Channel HBAs. Though these adapters physically correspond to the same 10GE port on a CNA, to the operating system, it will appear as two completely separate and physically isolated interfaces. Because of this adapter port virtualization, it is possible to build two separate topologies based on traffic type: one for the Ethernet fabric using the NICs and one for the FC fabric using the HBAs.

For FCoE and vPC to run side-by-side from the host, the port channel would be configured on the NICs interfaces presented and SAN multi-pathing or other SAN HA mechanisms would be configured on the FC HBAs presented to the OS by the CNA. Today, it is required that only 2X10GE links be used in a host side vPC port channel when running FCoE on the same wires. Each 10GE link will be used to provide a single connection to each upstream vPC switch.

Figure 5-1 Ethernet And FC Traffic Through A CNA

Figure 5-2 Adapter Control Panel Display

Note ● VPC + FCoE over a consolidated wire from the host requires the host supports port channels capabilities (LACP or “port channel mode ON”). Please check with specific CNA and OS vendors for a support matrix.

• VPC + FCoE over a consolidated wire are only supported between a host and either the first hop Nexus 5000 or Nexus 5000/2232 pair. VPC and FCoE on a consolidated wire is NOT supported beyond the access layer or when connecting a host to the Nexus 7000 platform.
• vPC and FCoE can not coexist on the same wire beyond any first hop access device.

FCoE VLAN Configuration On A vPC

Typically, interfaces belonging to the same port channel are required to have the same port configuration. This includes VLAN configuration. However, in order to maintain fabric separation alongside vPC connections, it is necessary to declare the FCoE VLAN for SAN A on one uplink and the FCoE VLAN for SAN B on the other uplink. This is a recommended best practice configuration. Figure 5-3 shows the hosts connected to a Cisco Nexus 5000 Series switch running vPC and FCoE simultaneously.

Figure 5-3 FCoE VLAN Configuration In A vPC Topology

Changing Buffer Allocations for Longer Distance FCoE

Beginning with the Cisco NXOS Release 5.0(1)N1(1) for the Cisco Nexus 5000 Series switch, it is possible to tune the port buffer allocation and xon and xoff thresholds in order to support increased distance between VE ports. The default distance configured for each port when configured to carry FCoE traffic (or any “no-drop” traffic) is 300 meters. This supported distance is based on the amount of available buffer space allocated to catch frames in flight between the time a PAUSE is initiated towards a downstream device and the time that downstream devices processes that PAUSE frame and stops sending frames. This per port buffer allocation and configuration must match between the two ports on either end of the link (including host CNA ports as well). This is similar to the way buffer-to-buffer credits is initialized between two devices in a native FC environment.

The current xon threshold and buffer size allocated for FCoE is such that buffer-size - xon = ~300 meters worth of FCoE frames. The default configuration parameters for the class-fcoe (or any no-drop class) on the Nexus 5000 series switch is shown below:

• qos-group 1
• q-size: 76800, HW MTU: 2400 (2240 configured)
• drop-type: no-drop, xon: 128, xoff: 240

In order to support a distance of 3000m for FCoE traffic between two FCoE capable switches (connecting two FCFs with VE ports), the buffer allocation as well as the xon and xoff values need to be altered for the FCoE class of services: class-fcoe. This can be accomplished by editing the quality of service configuration. For more information, see the Cisco NX-OS 5000 Series Quality of Service Configuration Guide.

The necessary thresholds for support no-drop service up to 3000m is outlined in the table below:

Consolidated Links And Dedicated Links for FCoE

Because FCoE uses the Ethernet fabric for transport, there is the possibility of consolidating both Ethernet LAN traffic and Storage SAN traffic onto the same infrastructure. There are multiple levels of consolidation; wire consolidation and device consolidation are two of the most common and are discussed below.

Link Consolidation refers to when Ethernet LAN traffic and Storage SAN traffic are sharing the same physical wire between host and switch or between two switches.

Device consolidation refers to when Ethernet LAN traffic and Storage SAN traffic are passing through the same switching device but maintain isolation through the use of dedicated wires or switch ports.

The topologies discussed throughout this guide will mention two terms to describe the scope of FCoE traffic: consolidated link – where FCoE and native Ethernet traffic simultaneously use the same link -- and dedicated link – where FCoE and native Ethernet traffic use two separate DCB Ethernet links. The following sections will discuss the different places in the Data Center Network where consolidated and dedicated links make sense.

Figure 5-4 shows an example of consolidated vs dedicated links. The wires running from the host to the access devices are consolidated links carrying both Ethernet and FCoE traffic. Moving from the access to aggregation, there are dedicated links: blue wires dedicated to the Ethernet traffic and orange wires dedicated to FCoE traffic only.

Figure 5-4 Consolidated And Dedicated Links

This section includes the following topics:

• Where Consolidated Links Makes Sense
• Where Dedicated Wires Makes Sense

Where Consolidated Links Makes Sense

One of the benefits of FCoE at the access layer is the ability to consolidate the FC SAN and Ethernet LAN onto the same physical wires and same physical devices. This consolidation lends to a large CapEx savings by reducing the number of access switches, host adapters, cables and optics required to run both LAN and SAN networks within the data center. This consolidation is made possible due to the excess bandwidth that 10GE to the server is able to provide. Because very few servers in the Data Center today are pushing 10-Gigabit Ethernet of Ethernet-only traffic, there is room for the added storage traffic to share these common wires without impacting the performance of the host application.

Also, due to the CNA behavior and ability to present to the host application different physical devices corresponding to both LAN and SAN networks, it is possible to separate Ethernet HA from FC HA at the host level. This is accomplished by being able to use separate Ethernet teaming options on the NICs while using separate FC multi-pathing options on the HBAs. Depending on the operating system and CNA being used, these teaming options will vary.

Moving beyond the access layer, oversubscription ratios and Ethernet bandwidth provisioning will determine the amount of excess bandwidth available and the benefit of running consolidated links vs. dedicated links within the Data Center.

Where Dedicated Wires Makes Sense

High Availability requirements in LAN and SAN networks differ considerably. Where in Ethernet, HA is achieved by multi-homing devices to one another (partial/full Mesh), in Fibre Channel (and FCoE), HA is achieved by building two physically isolated networks. Both of these requirements must be me in a network that combines FCoE and Ethernet.

There have been multiple enhancements to the Ethernet HA model that improves on Ethernet Data Center design by overcoming some of the challenges of the Spanning Tree protocol. One example of this is the virtual Port Channeling feature found in the Nexus product suite. The nature of vPC is to be able to forward out multiple paths to multiple upstream devices without spanning tree blocking any of the uplinks. While this is great for Ethernet traffic, it breaks the SAN A/SAN B isolation required for FC/FCoE.

Therefore, it is often beneficial to use dedicated wires for Ethernet traffic and Storage traffic independently. With dedicated wires, the Ethernet links can be configured to take advantage of advance Ethernet features such as vPC and the storage links can be configured based on the fabric isolation requirement. This is especially common when connected access switches to upstream LAN aggregation/SAN core devices.

Cisco Nexus 5000 Series Switch FCoE Considerations

The Cisco Nexus 5000 Series switches include a Unified Port Controller (UPC) ASIC responsible for the handling the forwarding decisions and buffering for multiple 10-Gigabit Ethernet ports:

• The Cisco Nexus 5000 Platform switches (the Cisco Nexus 5010 switch and the Cisco Nexus 5020 switch) include the first generation UPC ASIC.
• The Cisco Nexus 5500 Platform switches (the Cisco Nexus 5548P switch, Nexus 5548UP switch, and Nexus 5596UP switch) include the second generation UPC ASIC.

The following sections discuss the differences between the first and second generation architectures that relate to FCoE configuration and supported topologies.

This section includes the following topics:

• VLAN Scalability
• FCoE QoS Configuration
• Unified Port Options

VLAN Scalability

One of the differences between the first and second generation ASICs is the number of available VLAN resources available. The first generation ASICs support up to 512 VLANs (507 of which are user configurable). With the second generation ASIC, the available VLAN number has increased from 512 to 4096. Currently, 31 VLANs and 31 VSANs are supported for FCoE VLAN/VSAN mappings on both generations.

Note The VLAN and the VSAN in an FCoE VLAN/VSAN mapping consume a hardware VLAN resources.

FCoE QoS Configuration

The Nexus 5000 Series switches always reserve some buffer space for FCoE traffic. When you enable the FCoE feature on Nexus 5000 Series switch, Nexus automatically configures the necessary QoS policy and buffer allocations using the reserved buffers.

The Nexus 5500 Series switches allow all available port buffers to be configured based on traffic needs. This allows you to create a custom FCoE policy that can use any available buffers.

When you enable FCoE on a Nexus 5500 Series switch, the system looks for a custom QoS policy. If it does not find one, it automatically uses the default QoS configuration shown below:

For more information, see the Cisco Nexus 5000 Series NX-OS Quality of Service Configuration Guide, which is available from:

http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html

Unified Port Options

Unified ports are capable of operating a 1- and 10-Gigabit Ethernet or 1-, 2-, and 4-Gigabit or 2-, 4-, and 8-Gigabit FC (depending on the transceiver used) which provide more configuration flexibility. Unified ports no longer require you to purchase a set number of FC ports through an expansion module. Unified Ports are available in the expansion module on the Cisco Nexus 5548P switch and the Nexus 5548UP platform as well as on all base ports of the Cisco Nexus 5596UP switch. There are configuration requirements that must be carefully followed when utilizing unified ports.

Ports of a similar type, either Ethernet or FC, must be configured in a contiguous sequence. Changes to the port-type require a switch reboot or expansion module reboot depending on where the unified ports are configured. For this reason, careful planning should be done when first configuring the switch. Cisco recommends as a best practice to start Ethernet port configurations at one end of the platform (from Eth1/1 counting up) and the necessary Fibre Channel ports configured from the opposite end of the platform (Eth 1/48 counting down).

For additional information on configuring unified ports, see the Unified Port Configurations on Cisco Nexus 5500 Platform Switches documentation.

Priority Flow Control and Enhanced Transmission Selection Considerations

Both Priority Flow Control (PFC) and Enhanced Transmission Selection (ETS) are part of the IEEE 802.1Q Enhance Ethernet Standards that are currently in the final stages of standardization. Both PFC and ETS are support on all Cisco Nexus 5000 Series switches. PFC is class of service (COS) based PAUSE allowing for FCoE traffic assigned to a specific COS value to retain the lossless qualities which are required for the FC protocol. ETS is a mechanism for dividing a 10-Gigabit Ethernet link into multiple lanes based on the COS value and allocating the necessary bandwidth requirements which are honored in the presence of congestion. ETS prevents situations where default traffic would interfere with higher priority traffic.

PFC and ETS are often used in today’s FCoE networks to provide lossless transport and dedicated bandwidth for FCoE traffic. However, they are not specific to FCoE and have many uses outside of an FCoE environment for providing specific levels of service to specific traffic classes.

This section includes the following topics:

• Default PFC and ETS Settings
• Changing PFC and ETS Settings
• Host-Side Considerations For Altering PFC And ETS Settings

Default PFC and ETS Settings

PFC and ETS both use the Class of Service (COS) bits in order to classify between traffic types. There are 8 COS values in the IEEE 802.1Q standard trunking header for Ethernet frames. The Cisco Nexus 5000 Series switch allows you to manually configure 6 classes. Up to 4 of the 6 user configurable classes can be designated as no-drop classes of service, meaning that in the event of port congestions, traffic belonging to the no-drop classes will pause to prohibit packet drop.

By default, the Nexus 5000 Platform as well as other vendor’s FCoE products have decided on COS value of 3 for FCoE traffic. When FCoE is enabled on the Cisco Nexus 5000 Series switch, COS 3 is automatically configured for no-drop service (PFC setting) as well as a guarantee of 50% of the bandwidth in the case of congestion (ETS setting). It is best practice to leave the default COS value of 3 for FCoE traffic due to the agreement between vendors to support this as a “no-drop” class.The minimum guaranteed bandwidth for FCoE traffic is 50%. The FCoE can use all/free bandwidth if it is available.

In the event that other traffic already exists within the network that is using the COS value of 3 or there is another reason to move FCoE traffic from COS 3, this can be changed through a Quality of Service configuration.

Changing PFC and ETS Settings

PFC and ETS settings are configured and changed in the Quality of Service configuration on the Nexus 5000 Series switch. This example shows a QoS configuration that changes the FCoE no-drop class of service to COS 4 as the reserved bandwidth for FCoE to 20% of the 10-Gigabit Ethernet link:

Step 1 Create classification rules first by defining and applying policy-map type qos:

Step 2 Define and apply policy-map type network:

Step 3 Create classification rules first by defining and applying policy-map type qos:

Step 4 Create classification rules for the individual classes:

Host-Side Considerations For Altering PFC And ETS Settings

Data Center Bridging eXchange (DCBX) protocol is another portion of the IEEE 802.1Q Data Center Bridging (DCB) standard currently in review by the Ethernet standards body. DCBX is a protocol that runs between DCB-capable devices to ensure that PFC and ETS settings are configured consistently between DCB peers. DCB can also be used as a way to configure DCB peer devices from a central switching location. CNAs that support DCB- willing are configured to accept the DCB configurations (including PFC and ETS settings) of the upstream DCB switching device. This greatly simplifies management and configuration of DCB and FCoE devices.

If changing the default configuration for FCoE traffic on the Cisco Nexus 5000 Series switch, it is possible for the switch to relay these configuration changes to any connected CNAs using the DCBX protocol. It is necessary that the CNA vendor and platform support DCBX in a willing mode in order for this to take place. Please check with the individual CNA vendors on whether they support receiving DCBX configurations for a network device.

If the CNA does not support a method of DCB-willing, in order to change from a default PFC and ETS configuration, it is required to manually alter the configuration of the Nexus 5000 Series as well as the downstream CNA device so that they are the same. Depending on the CNA, different tools or commands will be used to change these settings.

Note If the DCBX negotiation fails between a host and switch or between a switch and switch, the PFC setting will not be set on the Nexus 5000 Series switch and the vFC interfaces will remain down until the DCB configuration matches.

Note Though the DCBX standard states that there are 8 possible no-drop lanes, CNA vendors differ on the number of COS values that are supported for FCoE and no-drop service today. Check with the CNA vendor for the correct number of supported FCoE and no-drop classes.

Cisco Nexus Interoperability

For information on interoperability, see the Cisco Data Center Interoperability Support Matrix.

Single-Hop FCoE Deployment Topologies

There are two possible single-hop solutions when deploying FCoE with a Cisco Nexus 5000 Series switch and Cisco Nexus 2000 Series Fabric Extender. The first solution is referred to as “direct connect” where a host is directly connected to the first hop converged access switch. The second single hop solution deploys a FEX between the server and the first hop switch. Because the FEX acts as a remote line card to the parent switch and has no local switching capabilities, it is not considered a hop in the Ethernet or Storage topologies. The following section outlines in detail the current single hop deployment options and configurations which are supported with the switch and FEX today.

This section includes the following topics:

• Switch Mode and NPV Mode
• vPC and Active/Standby
• Direct Attached CNAs With Active/Standby Ethernet Topologies
• Direct Attached CNAs With vPC Ethernet Topologies
• Cisco Nexus 5000 Series Switch and Cisco Nexus 2000 Fabric Extender Topologies
• FIP Snooping Bridges
• Cisco Nexus 4000 Series Switch To Cisco Nexus 5000 Series Switch FCoE With Consolidated Links
• Cisco Nexus 4000 Series Switch Connected To A Cisco Nexus 5000 Series Switch FCoE With Dedicated Wires

Switch Mode and NPV Mode

The Cisco Nexus 5000 Series switch has two modes of operation relating to storage traffic forwarding: switch mode and N-Port Virtualizer (NPV) mode. This is the same as the modes of operation available on the Cisco Multiprotocol Director Series (MDS) Fibre Channel switches. The default mode on both platforms is “switch” mode. In the following topologies, the Cisco Nexus 5000 Series switch can either be in switch or NPV mode. The only requirement for a Cisco Nexus 5000 Series switch in NPV mode is that the upstream device supports the standard N-Port ID Virtualization (NPIV) functionality.

When the Cisco Nexus 5000 Series switch is operating in switch mode, all fabric services, for example, FSPF, zoning or DNS, are native on the access device. This means that all forwarding decisions are made by FSPF running on the switch. This mode also means that the switch consumes a Domain ID within the Fibre Channel Fabric. Limitations exist as to the number of Domain IDs that are supported within a single fabric. Specific domain ID limitations are defined by the storage vendors and OSM partners.

NPV defines the ability for a Fibre Channel switch to act as a proxy for both FLOGIs and forwarding decision and pass those duties to an upstream device. This upstream device must be capable of running NPIV which is an FC standard allowing multiple FCiDs to be handed out a single FC port. The benefit of an NPV device in a FC network is the elimination of the domain ID and therefore the ability to add more FC switches to a fabric without exceeding the supported Domain ID limitation.

The Cisco Nexus 5000 Series switch can also operate in NPV mode. When NPV is enabled on the switch, no FC fabric services are run locally on the platform and instead, forwarding and zoning services are handled by the upstream NPIV device. To avoid interoperability challenges when connecting a switch to a non-Cisco SAN core switch, Cisco recommends that the switch be configured in NPV mode.

Enabling NPV on the switch is a disruptive process and should be done at the time of initial set up to avoid any disruption to the fabric. Because enabling NPV requires a switch reboot and erases the current running configuration, be sure to save the current running configuration to an external text file so that it can be reapplied after the reboot occurs if enabling NPV after the initial set up of the switch.

Changing between switch mode and NPV mode can be done using the following commands:

To enable NPV mode:

To disable NPV mode (return to switch mode):

Note Running NPV on the switch requires that the upstream connected device has NPIV functionality enabled

Note FC or FCoE hosts conversing with an FC or FCoE storage devices connected to the same switch in NPV is NOT supported.

vPC and Active/Standby

Host facing interfaces on the Nexus 5000 Series switch can provide connections to servers in a couple of different ways: single attached NICs for single attached hosts, active-standby NIC teaming for dual-homed servers and vPC for dual-homed servers. This guide focuses on the dual-homed server options as FC requires two independent paths to storage: Fabric A and Fabric B.

Active/Standby connections refer to servers that are dual-homed to an Ethernet LAN but only actively forwarding out one link. The second link is used as back-up in case of a failure but does not actively forward traffic unless a failure occurs. vPC is a technology introduced by Cisco Nexus products that allows a dual homed server to actively forward out both Ethernet links simultaneously. The benefits of vPC is that it gives servers access to twice as much bandwidth as in an active/standby configuration and also has the ability to converge faster than Spanning-tree in the event of a failure.

Based on the Ethernet high availability requirement, LAN admins may choose to attached servers using active/standby connections or vPC connections. Regardless of the method use to dual home a server, FCoE can co-exist with both of these topologies.

Direct Attached CNAs With Active/Standby Ethernet Topologies

Figure 5-5 shows a topology where a dual-port CNA is connecting to two switches in an active/standby configuration. Although Ethernet traffic will only traverse one link in this configuration, the FCoE traffic will be forwarded out both paths to the fabric. This is because of the way the CNA is able to differentiate between the NIC adapters for Ethernet and FC adapters for FC/FCoE. For more information on the CNA view of the Ethernet NICs and storage HBAs, see the “View Of Ethernet Traffic And FC Traffic Through A CNA” section.

Figure 5-5 Dual-Port CNA Connecting To Two Cisco Nexus 5000 Series Switches In An Active/Standby Topology

Direct Attached CNAs With vPC Ethernet Topologies

Figure 5-6 shows a topology where a dual-port CNA is connecting to two switches in a vPC configuration where only a single port connects the CNA to each switch. The operating system is able to see the Ethernet aspects of these two physical ports and port channel the Ethernet traffic coming out of the server. The FC traffic is still mapped to each link separately – one 10-Gigabit link transporting Fabric A traffic and the other 10-Gigabit link transporting Fabric B traffic. For more information on the CNA view of the Ethernet NICs and Storage HBAs, see the “View Of Ethernet Traffic And FC Traffic Through A CNA” section.

Figure 5-6 Dual-Port CNA Connecting To Two Cisco Nexus 5000 Series Switches In A vPC Topology

Note Direct-connect FCoE (a CNA that is directly connected to a Cisco Nexus 5000 Series switch switchport) is not supported on a port channel interface configured to have more then one member port. Directly connected FCoE devices are supported over virtual port channels where a single link from each CNA port connects through to each upstream switch or fabric extender.

Cisco Nexus 5000 Series Switch and Cisco Nexus 2000 Fabric Extender Topologies

The Nexus 2232 Fabric Extender acts as a remote line card to the parent Cisco Nexus 5000 Series switch. The Nexus 2232 Fabric Extender has 32 10-Gigabit Ethernet host facing interfaces, all of which support lossless Ethernet and FCoE. Supporting FCoE over a Cisco Nexus 5000 Series switch and FEX topology has the following requirements:

• Each Nexus 2232 Fabric Extender running FCoE must be single-homed to the upstream parent switch.
• Generation 2 (FIP Enabled) CNAs are required for host connections to the Cisco Nexus 2232 Fabric Extender host interfaces.

Adding the Cisco Nexus 2232 Fabric Extender into the FCoE topology does not change the supported configurations. Hosts can be connected to the Cisco Nexus 2232 Fabric Extender using active/standby Ethernet connections or over vPC connections. Figure 5-7 shows the supported topology.

Figure 5-7 Hosts Connected To The Cisco Nexus 2232 Fabric Extender Using Active/Standby Ethernet Connections or vPC Connections

Note FCoE is not supported on a FEX interface or port channel interfaces when the FEX is connected to two switches in a FEX active-active topology.

FIP Snooping Bridges

FIP Snooping Bridges (FSBs) are lossless Ethernet bridges that are capable of watching a FIP conversation between a CNA and FCF. They have no FC/FCoE forwarding logic capabilities but instead “snoop” FIP packets and watch the FIP conversation, including FLOGI/LOGIN, between the CNA and FCF. Once a FIP snooping bridge sees a CNA login to the FC/FCoE fabric through a specific FCF, it dynamically creates an access list to guarantee that the communication between that CNA and FCF will remain point-to-point. FIP snooping is a security precaution used when transversing lossless Ethernet bridges to ensure that rogue devices can not enter the data center network and pretend to be an FCF.

It is important to note that FSBs are Layer 2 Lossless Ethernet bridges that have been enhanced to dynamically create ACLs based on the FIP communication that is seen within the fabric. FSBs have no knowledge of FC/FCoE protocols or services and do not forward FCoE traffic based on FSPF. Instead, all traffic runs over the Layer 2 protocol (STP) and is switched based on MAC address.

The Cisco Nexus 4000 Series switch is a FIP Snooping device for IBM blade chassis and must be connected to a Cisco Nexus 5000 Series FCF switch in order to support passing FCoE frames. The Cisco Nexus 4000 Series switch has 14 down-facing 10-Gigabit ports connecting to each of the 14 blade servers and 6 10-Gigabit Ethernet uplink ports used to connect to a Cisco Nexus 5000 Series switch. Figure 5-8 and Figure 5-9 shows the two supported configurations when connecting a Cisco Nexus 4000 Series switch FIP Snooping bridge to a Cisco Nexus 5000 Series FCF switch:

Cisco Nexus 4000 Series Switch To Cisco Nexus 5000 Series Switch FCoE With Consolidated Links

Figure 5-8 shows a Cisco Nexus 4000 Series switch connected to a Cisco Nexus 5000 Series switch using consolidated links where both FCoE and Ethernet traffic are utilizing the same link simultaneously. Because FCoE requires fabric separation, the Ethernet traffic must also only follow one path and can not take advantage of other Ethernet HA technologies such as vPC.

Figure 5-8 Cisco Nexus 4000 Series Switch Connected To A Cisco Nexus 5000 Series Switch FCoE With Consolidated Links

Cisco Nexus 4000 Series Switch Connected To A Cisco Nexus 5000 Series Switch FCoE With Dedicated Wires

Figure 5-9 shows the Cisco Nexus 4000 Series switches connecting to Cisco Nexus 5000 Series switches using dedicated links; blue links are Ethernet ONLY links and pink and blue links are FCoE-only links. There are no consolidated links shown in Figure 5-9. The benefit of running dedicated links between the Cisco Nexus 4000 Series switches and Cisco Nexus 5000 Series switches in this topology is the fact that both storage and Ethernet traffic are able to take advantage of their respective HA models. Ethernet traffic is multi-homed to the upstream switches and using vPC to forward out all available paths while FCoE is maintaining fabric isolation through the Ethernet network.

Figure 5-9 Cisco Nexus 4000 Series Switch Connected To A Cisco Nexus 5000 Series Switch FCoE With Dedicated Wires

Multi-Hop FCoE Solutions

Multi-Hop FCoE is achieved with the support of Virtual E-ports (VE ports) connection two FCFs. Like E_Ports in native FC, VE ports are use to expand the FCoE fabric. VE ports are supported on the Nexus 5000 Series switch as of the NXOS Release 5.0(1)N2(2). There are two options for connecting Nexus 5000 Series switches with the use of VE ports: using single-links or over a port channel. For configuration examples of VE ports, see Appendix B, “Port Configuration Examples” .

In order to maintain fabric isolation, the Cisco Nexus 5000 FCF switches in each fabric should be configured to have the same FC-MAP value. The FC-MAP values should be different between Fabric A and Fabric B. For additional information on FC-MAP configurations, see Appendix B, “Port Configuration Examples” . VE ports brought up between two Cisco Nexus 5000 Series switches with differing FC-MAPs are not supported which ensures that fabrics are not merged by connecting FCFs in Fabric A to FCFs in Fabric B. Figure 5-10 shows FCF connections using VE ports.

Figure 5-10 VE Ports And FCF Mapping

Note VE ports are not supported over vPCs.

Tracking FCoE Statistics

FCoE statistics for FCoE traffic transversing an interface on a Cisco Nexus 5000 Series switch can be seen by monitoring the statistics on the vFC interface which is bound to the physical Ethernet interface or port channel interface.

This section includes the following topics:

• Tracking VE Port Statistics
• Tracking VF Port Statistics

Tracking VE Port Statistics

The following example shows how to monitor VE port statistics:

Tracking VF Port Statistics

The following example shows how to monitor VF port statistics:

SPAN for FC and FCoE Traffic

This section includes the following topics:

• Possible SPAN Sources
• Possible SPAN Destinations
• SPAN Configuration Examples

Possible SPAN Sources

Following are possible SPAN sources:

• FC interface (only rx-source on 5500 platform)
• VFC interface
• VSAN (not supported on 5500 platform)
• VLAN
• Ethernet interface
• Port channel interface
• SAN port channel interface

Possible SPAN Destinations

Following are possible SPAN destinations:

• FC interface
• Ethernet interface

SPAN Configuration Examples

This example shows how to display configuration information on Ethernet 1/1:

This example shows how to display the health monitoring of all interfaces for failover purposes:

This example shows the health monitoring of session 1:

This example shows the health monitoring configuration:

This example shows the health monitoring of all sessions:

Roles Based Access Control

With the Cisco Nexus Family of switches deploying unified I/O capabilities, the roles of LAN and SAN administrators are converging. To help manage these two different roles on the Cisco Nexus Series Family of switches, the Roles Based Access Control (RBAC) feature facilitates various administrative operations.

When deploying unified I/O within a data center, Cisco recommends defining the following three roles:

• Unified Administrator—This role includes all actions that impact both LAN and SAN operations. This role is sometimes referred to as a global administrator.
• LAN Administrator—This role includes a set of actions that impact LAN operation while denying any actions that could impact SAN operations.
• SAN Administrator—This role includes a set of actions that impact SAN operation while denying any actions that could impact LAN operations.

These are general roles that are used to enforce the operational model where separate LAN and SAN administrative teams retain management control of their perspective networks without interference. More specific roles may be added if operations need to be more tightly defined.

This section includes the following topics:

• Unified Administrator Role
• LAN Administrator Role
• SAN Administrator Role

Unified Administrator Role

The Unified Administrator role may perform all actions. In addition, the Unified Administrator plays a large role in the initial set up of the unified network.

Before implementing a unified network design, the physical interfaces and VLANs used for unified traffic should be identified and defined. Standard implementation of FCoE requires binding a virtual Fibre Channel interface (vFC) to either a physical Ethernet interface or MAC-Address. It is also required to map the VSAN used to carry the FC traffic to a corresponding Ethernet VLAN. While Ethernet interfaces and VLANs normally fall under the scope of a LAN administration, the unified interfaces and FCoE VLANs must be identified so that they can be separated from the LAN administration domain.

Cisco recommends that you identify the interfaces used for Unified I/O, and that you designate a range of VLANs for FCoE use before implementation begins. The Unified Administrator role will configure these unified interfaces and FCoE VLANs.

LAN Administrator Role

This role is assigned all the permissions that impact LAN traffic. This role also denies any actions that would possibly impact SAN traffic (FCoE and FC). One of the main difference between the LAN administrator role and a LAN administrator in a legacy data center without unified I/O is the inability to shut down a physical Ethernet port carrying FCoE traffic. Potentially, both FC and Ethernet traffic could be traveling over the link simultaneously and, therefore, shutting the port could have an impact on SAN operations.

A list of commands which can impact SAN operations, and therefore should be limited from the role of the LAN Administrator, can be found in Appendix A, “RBAC Configuration” . Individual network designs may require additional limited commands.

SAN Administrator Role

This role is assigned all the permissions that impact SAN traffic. The role also denies actions that would impact LAN traffic.

SAN administration in a unified environment and a legacy SAN environment are similar. Today, unified I/O runs only between the servers and the top-of-rack Cisco Nexus 5000 switch, where FC links are run back into the core of the existing SAN infrastructure. The FC module inside the Cisco Nexus 5000 switch can operate in either NPV or switch mode. The switch most commonly operates in NPV mode and, from a management perspective, looks identical to a FC blade or fabric switch operating in NPV mode.

A list of commands which can impact LAN operations and therefore should be limited from the role of the SAN Administrator can be found in Appendix A, “RBAC Configuration” . Individual network designs may require additional limited commands.

FCoE Limitations

If you have end devices like Blade Center Chassis, which is not a FIP Snooping Bridge and you have multiple servers connected behind it with FCoE, and it is connected to FEX ports. Then, you cannot bind using MAC-address into the VFC nor can you bind multiple port channel into the VFC. The only configuration supported is bind a single physical port or single port channel with one member.

This section includes the following topics:

• Generation 1 And Generation 2 CNA Limitations
• LACP and FCoE To The Host
• Deploying a Cisco Nexus 5000 Series Switch as an NPIV Core
• VE Ports on a Cisco Nexus 5010 Switch or Cisco Nexus 5020 Switch

Generation 1 And Generation 2 CNA Limitations

When FCoE was introduced on the Cisco Nexus 5000 Series switch, Cisco worked with QLogic and Emulex to create the first generation of CNA adapters. These CNAs used a pre-standard implementation of the DCBX protocol nicknamed CIN-DCBX. These adapters also did not support the standard FIP implementation as defined in the FCoE Standard (FC-BB-5) and they are often referred to as Pre-FIP adapters.

Starting in 2009, after the ratification of the FCoE standard, second generation CNAs were put out by both QLogic and Emulex that supported standard FIP and FCoE. These CNAs also used a pre-standard version of the DCBX protocol nicknamed CEE-DCBX which has been decided on by multiple venders to be the de-facto standard until IEEE DCBX is ratified.

Topologies and Platforms Which Require Generation 2 CNAs

While the Cisco Nexus 5010 switch and Nexus 5020 switch are backwards compatible with both Generation 1 and Generation 2 CNAs and support, the Nexus 2000 Fabric Extenders and the Nexus 5500 Platform switches only support Generation 2 CNA connections. Also, Generation 2 CNAs are required when connecting a host using vPC into a fabric, whether the host is running FCoE or just native Ethernet.

LACP and FCoE To The Host

Today, when deploying FCoE over a host-facing vPC, the vFC interface is bound to the port channel interfaces associated with the vPC. This requires that the port channel interface be up and forwarding before FCoE traffic can be switched. Cisco recommends when running vPC in an Ethernet environment is to use LACP in order to negotiate the parameters on both sides of the port channel to ensure that configurations between both sides is consistent.

However, if there are inconsistencies in any of the Ethernet configuration parameters LACP uses to bring up the port channel interface, both sides of the virtual port channel will remain down. This means that FCoE traffic from the host is now dependent on the correct configuration on the LAN/Ethernet side. When this dependency occurs, Cisco recommends that you use the static port channel configuration (channel-group # mode on) when deploying vPC and FCoE to the same host.

Deploying a Cisco Nexus 5000 Series Switch as an NPIV Core

The Nexus 5000 Series switch supports both NPV and NPIV functionality. If acting as an NPIV core switch with downstream NPV switches attached to it, it is important to note that hosts and targets which are communicating to one another can not be attached to the same downstream NPV device.

VE Ports on a Cisco Nexus 5010 Switch or Cisco Nexus 5020 Switch

Cisco Nexus 5000 Series and Cisco Nexus 5500 Platform switches support VE port connections. On Cisco Nexus 5010 and Nexus 5020 switches, VE ports can be configured between two switches using a single port channel or multiple individual links. VE ports configured between two switches using multiple port channels is not supported. This has to do with the number of MAC addresses available for the VE port on the Cisco Nexus 5010 switch and the Cisco Nexus 5020 switch. This limitation does not apply to the Cisco Nexus 5500 Platform.