Preparing the Switch for Configuring FCoE

Information About FCoE

FCoE provides a method of transporting Fibre Channel traffic over a physical Ethernet connection. FCoE requires the underlying Ethernet to be full duplex and to provide lossless behavior for Fibre Channel traffic.


Note

Lossless behavior on Ethernet is provided by using a priority flow control (PFC) mechanism that prevents packet loss during congestion conditions.

The Cisco NX-OS software supports T11-compliant FCoE on all 10-Gigabit and 40-Gigabit Ethernet interfaces.

FCoE and FIP

FCoE Initiation Protocol

The FCoE Initialization Protocol (FIP) allows the switch to discover and initialize FCoE-capable entities that are connected to an Ethernet LAN. Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) protocol for T11-compliant Generation 2 CNAs.

The following switches do not support Pre-FIP on Generation 1 CNAs:

FIP Virtual Link Instantiation

FIP is used to perform device discovery, initialization, and link maintenance. FIP performs the following tasks:

  • FIP VLAN discovery— Discovers the FCoE VLAN that will be used by all other FIP protocols as well as by the FCoE encapsulation for Fibre Channel payloads on the established virtual link. FIP VLAN discovery occurs in the native VLAN used by the initiator or target to exchange Ethernet traffic. The FIP VLAN discovery protocol is the only FIP protocol running on the native VLAN; all other FIP protocols run on the discovered FCoE VLANs.

  • FIP FCF discovery—When a FCoE device is connected to the fabric, it sends a Discovery Solicitation message. A Fibre Channel Forwarder (FCF) or a switch responds to the message with a Solicited Advertisement that provides an FCF MAC address to use for subsequent logins.

  • FCoE virtual link instantiation— FIP defines the encapsulation of fabric login (FLOGI), fabric discovery (FDISC), logout (LOGO), and exchange link parameter (ELP) frames along with the corresponding reply frames. The FCoE devices use these messages to perform a fabric login.

  • FCoE virtual link maintenance— FIP periodically sends maintenance messages between the switch and the CNA to ensure that the connection is still valid.

FCoE Frame Format

FCoE is implemented when the switch encapsulates a Fibre Channel frame in an Ethernet packet with a dedicated Ethernet type, 0x8906. The packet has a 4-bit version field. The other header fields in the frame (the source and destination MAC addresses, VLAN tags, and frame markers) are all standard Ethernet fields. Reserved bits pad the FCoE frame to the IEEE 802.3 minimum packet length of 64 bytes.

A Fibre Channel frame consists of 36 bytes of headers and up to 2112 bytes of data for a total maximum size of 2148 bytes. The encapsulated Fibre Channel frame has all the standard headers, which allow it to be passed to the storage network without further modification. To accommodate the maximum Fibre Channel frame in an FCoE frame, the class-fcoe is defined with a default Maximum Transmission Unit (MTU) of 2240 bytes.

VLAN Tagging for FCoE Frames

The Ethernet frames that are sent by the switch to the adapter include the IEEE 802.1Q tag. This tag includes a field for the class of service (CoS) value used by the priority flow control (PFC). The IEEE 802.1Q tag also includes a VLAN field.

The switch expects frames from a FIP T11-compliant CNA to be tagged with the VLAN tag for the FCoE VLAN. Frames that are not correctly tagged are discarded.


Note

You cannot map VLAN 1 or the native VLAN to an FCoE VSAN.

FIP Ethernet Frame Format

FIP is encapsulated in an Ethernet packet with a dedicated EtherType, 0x8914. The packet has a 4-bit version field. Along with the source and destination MAC addresses, the FIP packet also contains a FIP operation code and a FIP operation subcode. The following table describes the FIP operation codes and subcodes.

Table 1. FIP Operation Codes

FIP Operation Code

FIP Subcode

FIP Operation

0x0001

0x01

Discovery Solicitation

0x02

Discovery Advertisement

0x0002

0x01

Virtual Link Instantiation Request

0x02

Virtual Link Instantiation Reply

0x0003

0x01

FIP Keepalive

0x02

FIP Clear Virtual Links

0x0004

0x01

FIP VLAN Request

0x02

FIP VLAN Notification

DCBX

Data Center Bridging Exchange Protocol

The Data Center Bridging Exchange (DCBX) protocol is an extension of the Link Layer Discovery Protocol (LLDP). DCBX end points exchange request and acknowledgment messages. For flexibility, parameters are coded in a type-length-value (TLV) format. Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) is supported on all T11-compliant Generation 2 CNAs.

DCBX runs on the physical Ethernet link between the switch and the CNA. By default, DCBX is enabled on Ethernet interfaces. When an Ethernet interface is brought up, the switch automatically starts to communicate with the CNA.

During the normal operation of FCoE between the switch and the CNA, DCBX provides link-error detection.

DCBX is also used to negotiate capabilities between the switch and the CNA and to send configuration values to the CNA.

The CNAs that are connected to a switch are programmed to accept the configuration values that are sent by the switch, allowing the switch to distribute configuration values to all attached CNAs, which reduces the possibility of configuration errors and simplifies CNA administration.

Lossless Ethernet

Standard Ethernet is a best-effort medium which means that it lacks any form of flow control. In the event of congestion or collisions, Ethernet drops packets. The higher level protocols detect the missing data and retransmit the dropped packets.

To properly support Fibre Channel, Ethernet has been enhanced with a priority flow control (PFC) mechanism.

Logical Link Up/Down

On a native Fibre Channel link, some configuration actions (such as changing the VSAN) require that you reset the interface status. When you reset the interface status, the switch disables the interface and then immediately reenables the interface.

Caution:

If an Ethernet link provides FCoE service, do not reset the physical link because this action is disruptive to all traffic on the link.

The logical link up/down feature allows the switch to reset an individual virtual link. The logical link down is signaled with a FIP Clear Virtual Link message.

Caution:

If the CNA does not support the logical link level up/down feature, the CNA resets the physical link, which means that all traffic on the Ethernet interface is disrupted.

Converged Network Adapters

Cisco NX-OS switches support the following CNA types:

  • Hardware adapter

    • Works with the existing Fibre Channel host bus adapter (HBA) driver and Ethernet Network Interface Card (NIC) driver in the server.

    • Server operating system view of the network is unchanged; the CNA presents a SAN interface and a LAN interface to the operating system.

  • FCoE software stack

    • Runs on existing 10-Gigabit Ethernet adapters.

The following Cisco NX-OS series and platforms support Generation 2 CNAs that use the FIP to exchange information about its available capabilities and to negotiate the configurable values with the switch:

  • Cisco Nexus 7000

  • Cisco Nexus 7700

To reduce configuration errors and simplify administration, the switch distributes the configuration data to all the connected adapters.

STP Lite

FCoE does not require full Spanning Tree Protocol (STP) because FCoE has no bridging functionality, which means that no STP loops are created in the network. STP Lite on FCoE interfaces ensures rapid convergence across the network by sending an agreement Bridge Protocol Data Unit (BPDU) whenever it receives a proposal BPDU. The FCoE link sends the identical agreement BPDU in response to either an Multiple Spanning Tree (MST) or a Per VLAN Rapid Spanning Tree Plus (PVRST+) proposal BPDU. Additionally, STP Lite suppresses the MAC address flushing function for FCoE VLANs.

STP Lite is enabled automatically by default across the entire device for FCoE VLANs as soon as the first FCoE VLAN comes up. At the same time, the system automatically converts all FCoE links as the STP-type normal ports. This feature runs only in FCoE VLANs.

Default Settings for FCoE

This table lists the default settings for FCoE parameters.

Table 2. Default FCoE Parameter Settings

Parameters

Default

FCoE feature

Not installed, disabled

FC-Map

0E.FC.00

Fabric priority

128

Advertisement interval

8 seconds

Guidelines and Limitations

FCoE

  • VDCs apply only to Cisco Nexus 7000 Series Switches.

  • You cannot enable FCoE on default VLAN.

  • The QoS policy must be the same on all Cisco FCoE switches in the network.

  • Beginning with Cisco NX-OS Release 6.1, FCoE is supported on F2 and F2e Series modules. F3 Series modules are supported from Cisco NX-OS Release 6.2(6) onwards.

    • FCoE supports only F2e (SFP+) modules.

    • FCoE does not support F2e (Copper) modules.

FCoE VDC

FCoE in a dedicated storage VDC has the following guidelines:

  • Enable the FCoE feature set in only one VDC.

  • Create VLANs in the FCoE allocated VLAN range.

  • Do not enable any other features other than storage-related features in the dedicated FCoE VDC.

  • Allocate resources for the dedicated FCoE VDC from an F Series module, such as the 32-port 10-Gigabit Ethernet I/O module (PID N7K-F132XP-15) .

  • Rollback is not supported in a storage VDC.

  • For Cisco NX-OS Release 7.2(0)D1(1), ports from only 24 FEXes can be shared to storage VDC. System will not restrict the user to go beyond 24 but, more than 24 is not tested and not supported.

  • FCoE on F2, F2e, and F3 Series modules is supported with the Supervisor 2 module (N7K-SUP2 for Cisco Nexus 7000 Series devices) and the Supervisor 2E module (N77-SUP2E for Cisco Nexus 7700 Series devices and N7K-SUP2E for Cisco Nexus 7000 Series devices).

  • In order to enable FCoE over FEX on the storage VDC, you must execute the allow feature-set FEX command from the Admin or default VDC beforehand for storage VDC. FCoE over FEX is available from 7.2(0)D1(1) and onwards.

  • IVR (Inter VSAN route) zone configuration is not supported for FCoE over FEX.

  • F3 Fiber Channel over Ethernet (FCoE) feature licensing is supported from 7.2.0 release onwards. To downgrade to the older version of the image 6.2.x, first uninstall the F3 FCoE license and then proceed. For more information about licensing, refer Cisco NX-OS Licensing Guide.

  • F2, F2e, and F3 Series modules can co-exist in the same VDC. This applies to both LAN and storage VDCs.

  • F1 and F3 Series modules cannot co-exist in the same VDC. This applies to both LAN and storage VDCs.

  • F1 and F2 series modules cannot exist in the same VDC. This applies to both LAN and storage VDCs.

  • Use the limit-resource module-type command in the admin or default VDC to assign module resources such as F1, F2, F2e and F3 to a storage VDC. The supported line card modules are F1, F2, F2e and F3.

  • When you configure a multi-hop FCoE, ensure that you use the same no-drop classes on both sides. Priority flow control does not work when you use different no-drop classes. Use the show interface priority-flow-control command to verify the priority flow control operation.

Shared Interfaces


Note

  • Any change in protocol state that flaps the parent port of a shared interface because of any port feature also affects the FCoE traffic on the storage vdc.

  • 1500 MTU do not carry FCoE traffic in all FCoE supported platforms.

The following interface config modes are not allowed while sharing an interface from Ethernet vdc to a storage vdc:

  • SPAN destination

  • Private VLAN mode

  • Port-channel interfaces

  • Access mode

  • mac-packet-classify

  • Interfaces that are part of a VLAN that has an associated QoS policy

Shared Ethernet interfaces must be in trunk mode and only shared with one other VDC.

Storage VDC

Configuring a VDC for the Out-Of-Band (OOB) management interface mgmt0 is accomplished with the vrf context management command. However, a storage VDC does not support VRF, so configuring mgmt0 requires a different approach.

The following table shows how to configure mgmt 0 for a VDC and for a storage VDC:

Configuring mgmt 0 for VDC Configuring mgmt 0 for storage VDC 
vrf context management
ip route 0.0.0.0/0 default_gateway
 
interface mgmt 0
ip address mgmt0_ip_address mgmt0_subnet_mask
no shut
ip route 0.0.0.0/0 default_gateway

Note

 

The ip route command specifies the default route that points to the default gateway.

where

  • mgmt0_ip_address is the mgmt0 IPv4 address.

  • mgmt0_subnet_mask is the mgmt0 IPv4 netmask.

  • default_gateway is the IPv4 address of the default-gateway.

For more information about VDC, see the Cisco Nexus 7000 Series NX-OS Virtual Device Context Configuration Guide.

Multi-Hop FCoE Distance Configuration

In Multi-Hop FCoE, when a device sends a pause, the interface that generates the pause frame must have an ingress queue with a buffer space large enough to buffer twice the link distance. This is because, when the pause is generated the wire might get congested. By the time the adjacent device receives or processes the generated pause frame, the wire might get congested again. Therefore, the device that generates the pause must have the ability to buffer twice the link distance.

As per calculations, there can be more than 100 packets traveling on the 10 kilometer link. Due to an ASIC limitation, the F1 series line card does not support lossless FCoE on a link greater than or equal to 10 kilometers.

For more information about Multi-Hop FCoE distance limitations, see http://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/117785-probsol-nexus7000-00.html

The F2 and F2E line cards support a long haul lossless distance of up to 80 kilometers. In Cisco NX-OS Release 6.2(8) and later, you can change the ingress queuing buffer configuration.

The F3 line cards support long haul lossless distance of up to 40 kilometers. In Cisco NX-OS Release 7.2(0) and later, you can change the ingress queuing buffer configuration.

Table 3. Buffer Tuning Table for FCoE Long Distance on F2, F2E, and F3 Line Cards

Distance

Line Card

SFP

Ingress Buffer Queue-Limit

< 5 km F2/F2e LR 60% no-drop and 40% drop queue
> 5 km - 10 km F2/F2e LR 70% no-drop and 30% drop queue
> 10 km - 40 km F2/F2e ER 80% no-drop and 20% drop queue
< 10 km F3 LR 90% no-drop and 10% drop queue
< 40 km F3 ER 90% no-drop and 10% drop queue

Enabling and Disabling FCoE

Enabling FCoE

You must install the FCoE feature set in the default VDC and enable dependent features in order to enable FCoE in a storage VDC.

Before you begin

  • Ensure you are in the default VDC.

  • Do not enable FCoE on VLAN 1 or the native VLAN.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

install feature-set fcoe

Example:

switch(config)# install feature-set fcoe

Installs the FCoE feature set in the default VDC.

Step 3

feature lldp

Example:

switch(config)# feature lldp

Enables the Link Layer Discovery Protocol (LLDP) feature in the default VDC. This feature is required for FCoE operation.

Step 4

(Optional) feature lacp

Example:

switch(config)# feature lacp

Enables the Link Aggregation Control Protocol (LACP) feature in the default VDC. This feature is considered a best practice for FCoE operation.

Step 5

system qos

Example:

switch(config)# system qos

Enters Quality of service (QoS) configuration mode.

Step 6

service-policy type network-qos policy-name

Example:

switch(config-sys-qos)# service-policy type
 network-qos default-nq-7e-policy

Enables the QoS policy that supports FCoE traffic. The policy-name default is default-nq-8e-policy.

Step 7

(Optional) show feature

Example:

switch(config-sys-qos)# show feature

Displays information about the enabled features.

Step 8

(Optional) copy running-config startup-config

Example:

switch(config-sys-qos)# copy running-config startup-config

Copies the running configuration to the startup configuration.

What to do next

You must create a storage VDC and allocate resources to finish enabling FCoE.

Disabling FCoE

You can disable or uninstall the FCoE feature set. You can also disallow the FCoE feature set in a VDC.

Before you begin

Ensure you are in the correct VDC.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

vdc vdc_id type storage

Example:

switch(config)# vdc fcoe type storage
switch(config-vdc)#

Enters VDC configuration mode. The vdc_id can be any case-sensitive, alphanumeric string up to 32 characters.

Step 3

(Optional) no allow feature-set fcoe

Example:

switch(config-vdc)# no allow feature-set fcoe
switch(config-vdc)#

Disallows any FCoE in this VDC. The default is allow.

Step 4

no feature-set fcoe

Example:

switch(config-vdc)# no feature-set fcoe

Disables the FCoE feature set. The no feature-set fcoe command may take some time to complete if the size of the configuration is very large. The command must clean up all of the configuration associated with the FCoE feature set.

Step 5

exit

Example:

switch(config-vdc)# exit
switch(config)#

Exits VDC configuration mode.

Step 6

no install feature-set fcoe

Example:

switch(config)# no install feature-set fcoe

Uninstalls the FCoE feature set. Use this command in the default VDC after you disable the FCoE feature set.

Step 7

(Optional) show feature-set

Example:

switch(config)# show feature-set

Displays information about the feature sets.

Step 8

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config

Copies the running configuration to the startup configuration.

Configuring the FC-Map

You can prevent data corruption due to cross-fabric talk by configuring an FC-Map that identifies the Fibre Channel fabric for this switch. When the FC-Map is configured, the switch discards the MAC addresses that are not part of the current fabric. An FCF can assign Fabric Provide MAC Addresses (FPMA) to the CNAs consisting of the FC-Map Value for the Fabric and the Fibre Channel ID (FCID) assigned during Fabric Login


Note

In most deployments, changing the fc-map is not needed. We recommend that you use VSAN to VLAN mapping. For more information, see the Configuring FCoE VLANs and Virtual Fibre Channel Interfaces chapter.

Before you begin

For Cisco Nexus 7000 Series, you must be in the storage VDC to configure this feature.

Procedure

Step 1

switchto vdc vdc-id type storage

Example:

switch# switchto vdc fcoe type storage
fcoe#

Step 2

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 3

fcoe fcmap fabric-map

Example:

fcoe(config)# fcoe fcmap 0x0efc2a

Configures the global FC-Map. The default value is 0x0EFC00. The range is from 0x0EFC00 to 0x0EFCFF. Use the no fcoe map command to reset to the default value.

Example

This example shows how to configure the global FC-Map on a Cisco Nexus 7000 Series switch: 

switch# switchto vdc fcoe type storage
fcoe# configure terminal
fcoe(config)# fcoe fcmap 0x0efc2a

Configuring the Fabric Priority

The FCoE switch advertises its priority. The priority is used by the CNAs in the fabric to determine the best switch to connect to.

Before you begin

For Cisco Nexus 7000 Series, you must be in the storage VDC to configure this feature.

Procedure

Step 1

switchto vdc vdc-id type storage

Example:

switch# switchto vdc fcoe type storage
fcoe#

Step 2

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 3

fcoe fcf-priority fabric-priority

Example:

fcoe(config)# fcoe fcf-priority 42

Configures the global fabric priority. The default value is 128. The range is from 0 (higher) to 255 (lower). Use the no fcoe fcf-priority command to reset the global fabric priority to the default value.

Example

This example shows how to configure the global fabric priority for a Cisco Nexus 7000 Series switch: 

switch# switchto vdc fcoe type storage
fcoe# configure terminal
fcoe(config)# fcoe fcf-priority 42

Setting the Advertisment Interval

You can configure the interval for Fibre Channel fabric advertisement on the switch.

Before you begin

For Cisco Nexus 7000 Series, you must be in the storage VDC to configure this feature.

Procedure

Step 1

switchto vdc vdc-id type storage

Example:

switch# switchto vdc fcoe type storage
fcoe#

Step 2

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 3

fcoe fka-adv-period interval

Example:

fcoe(config)# fcoe fka-adv-period 8
fcoe#

Configures the advertisement interval for the fabric. The default value is 8 seconds. The range is from 4 to 60 seconds.

Example

This example shows how to configure the advertisement interval for the fabric on a Cisco Nexus 7000 Series switch: 

switch# switchto vdc fcoe type storage
fcoe# configure terminal
fcoe(config)# fcoe fka-adv-period 42

Disabling LAN Traffic on an FCoE Link

You can disable LAN traffic on an FCoE link.

DCBX allows the switch to send a LAN Logical Link Status (LLS) message to a directly-connected CNA. Enter the shutdown lan command to send an LLS-Down message to the CNA. This command causes all VLANs on the interface that are not enabled for FCoE to be brought down. If a VLAN on the interface is enabled for FCoE, it continues to carry SAN traffic without any interruption.


Note

The shutdown lan command is supported in Cisco Nexus 7000 Series switches running Cisco NX-OS Release 6.2(6) and later. See the Cisco Nexus 7000 Series NX-OS Interfaces Command Reference for more information about this command.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

interface ethernet slot/port

Example:

switch(config)# interface e 2/1
switch(config-if)#

Specifies an interface to configure, and enters interface configuration mode. Use ? to view a list of supported interfaces.

Step 3

shutdown lan

Example:

switch(config-if)# shutdown lan

Shuts down Ethernet traffic on the interface. If the interface is part of an FCoE VLAN, the shutdown has no impact on the FCoE traffic. Use no shutdown lan to reenable Ethernet traffic on this interface.

Step 4

(Optional) show interface

Example:

switch(config-if)# show interface

Displays information about the interface.

Step 5

(Optional) copy running-config startup-config

Example:

switch(config-if)# copy running-config startup-config

Copies the running configuration to the startup configuration.

Additional References for FCoE

Related Documents

Related Topic Document Title

Command reference

Cisco NX-OS FCoE Command Reference Guide, Nexus 7000 and MDS 9500

Configuration guide

Cisco Nexus 7000 Series NX-OS Virtual Device Context Configuration Guides

Cisco Nexus 7000 Series NX-OS Quality of Service Configuration Guide

Cisco NX-OS licensing

Cisco NX-OS Licensing Guide

Standards and RFCs

Standard/RFC Title
T11 FC BB-5 Fibre Channel Backbone 5

MIBs

MIB MIBs Link

To locate and download MIBs for selected platforms, Cisco IOS releases, and feature sets, use Cisco MIB Locator found at the following URL:

http://www.cisco.com/go/mibs

Technical Assistance

Description Link

The Cisco Support website provides extensive online resources, including documentation and tools for troubleshooting and resolving technical issues with Cisco products and technologies.

To receive security and technical information about your products, you can subscribe to various services, such as the Product Alert Tool (accessed from Field Notices), the Cisco Technical Services Newsletter, and Really Simple Syndication (RSS) Feeds.

Access to most tools on the Cisco Support website requires a Cisco.com user ID and password.

http://www.cisco.com/support