FCoE provides a
method of transporting Fibre Channel traffic over a physical Ethernet
connection. FCoE requires the underlying Ethernet to be full duplex and to
provide lossless behavior for Fibre Channel traffic.
Note
Lossless behavior on Ethernet is provided by using a priority flow control (PFC) mechanism that prevents packet loss during congestion conditions.
The Cisco NX-OS software supports T11-compliant FCoE on all 10-Gigabit Ethernet interfaces.
The FCoE Initialization Protocol (FIP) allows the switch to discover
and initialize FCoE-capable entities that are connected to an Ethernet LAN. Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange
(CEE-DCBX) protocol for T11-compliant Generation 2 CNAs.
The following switches do not support Pre-FIP on Generation 1 CNAs:
Cisco Nexus 7000
Cisco MDS 9500
FIP Virtual Link Instantiation
FIP is used to perform device discovery, initialization, and link
maintenance. FIP performs the following tasks:
FIP VLAN discovery— Discovers the FCoE VLAN that will be used by all other FIP protocols as well as by the FCoE encapsulation for Fibre Channel payloads on the established virtual link. FIP VLAN discovery occurs in the native VLAN used by the initiator or target to exchange Ethernet traffic. The FIP VLAN discovery protocol is the only FIP protocol running on the native VLAN; all other FIP protocols run on the discovered FCoE VLANs.
FIP FCF discovery—When a FCoE device is connected to the fabric, it sends a
Discovery Solicitation message. A Fibre Channel Forwarder (FCF) or a switch
responds to the message with a Solicited Advertisement that provides an FCF MAC
address to use for subsequent logins.
FCoE virtual link instantiation—
FIP defines the encapsulation of fabric login (FLOGI), fabric discovery (FDISC), logout (LOGO), and exchange link parameter (ELP) frames
along with the corresponding reply frames. The FCoE devices use these messages
to perform a fabric login.
FCoE virtual link maintenance—
FIP periodically sends maintenance messages between the switch and the CNA to ensure that the connection
is still valid.
FCoE Frame Format
FCoE is implemented when the switch encapsulates a Fibre Channel frame in an Ethernet packet with a dedicated Ethernet type, 0x8906. The packet has a 4-bit version field. The other
header fields in the frame (the source and destination MAC addresses, VLAN
tags, and frame markers) are all standard Ethernet fields. Reserved bits pad the FCoE frame to the IEEE 802.3 minimum packet length of 64 bytes.
A Fibre Channel frame consists of 36 bytes of headers and up to 2112
bytes of data for a total maximum size of 2148 bytes. The encapsulated Fibre
Channel frame has all the standard headers, which allow it to be passed to the
storage network without further modification. To accommodate the maximum Fibre
Channel frame in an FCoE frame, the
class-fcoe is defined with a default Maximum Transmission Unit (MTU) of 2240
bytes.
VLAN Tagging for FCoE Frames
The Ethernet frames that are sent by the switch to the adapter include the IEEE 802.1Q tag. This tag includes a field for the class of service (CoS) value
used by the priority flow control (PFC). The IEEE 802.1Q tag also includes a VLAN field.
The
switch expects frames from a FIP T11-compliant CNA to be tagged
with the VLAN tag for the FCoE VLAN. Frames that are not correctly tagged are discarded.
Note
You cannot map VLAN 1 or the native VLAN to an FCoE VSAN.
The following switches do not support Pre-FIP on Generation 1 CNAs:
Cisco Nexus 7000
Cisco MDS 9500
FIP Ethernet Frame Format
FIP is encapsulated in an Ethernet
packet with a dedicated EtherType, 0x8914. The packet has a 4-bit version field.
Along with the source and destination MAC addresses, the FIP packet also
contains a FIP operation code and a FIP operation subcode.
The following table describes the FIP operation codes and subcodes.
Table 1 FIP Operation Codes
FIP Operation Code
FIP Subcode
FIP Operation
0x0001
0x01
Discovery Solicitation
0x02
Discovery Advertisement
0x0002
0x01
Virtual Link Instantiation Request
0x02
Virtual Link Instantiation Reply
0x0003
0x01
FIP Keepalive
0x02
FIP Clear Virtual Links
0x0004
0x01
FIP VLAN Request
0x02
FIP VLAN Notification
DCBX
Data Center Bridging Exchange Protocol
The Data Center Bridging Exchange (DCBX) protocol
is an extension of the Link Layer Discovery Protocol (LLDP). DCBX end points
exchange request and acknowledgment messages. For flexibility, parameters are
coded in a type-length-value (TLV) format.
Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) is supported on all
T11-compliant Generation 2 CNAs.
DCBX runs on the physical Ethernet link between the
switch and the CNA. By default, DCBX is enabled on Ethernet interfaces. When an
Ethernet interface is brought up, the switch automatically starts to
communicate with the CNA.
During the normal operation of FCoE between the switch
and the CNA, DCBX provides link-error detection.
DCBX is also used to negotiate capabilities between
the switch and the CNA and to send configuration values to the CNA.
The CNAs that are connected to a
switch are programmed to accept the configuration values that are sent by the switch,
allowing the switch to distribute configuration values to all attached CNAs, which reduces the possibility of configuration errors and simplifies
CNA administration.
Lossless Ethernet
Standard Ethernet is a best-effort medium which means that it lacks any form of
flow control. In the event of congestion or collisions, Ethernet drops
packets. The higher level protocols detect the missing
data and retransmit the dropped packets.
To properly support Fibre Channel, Ethernet has been enhanced with a
priority flow control (PFC) mechanism.
On a native Fibre Channel link, some configuration
actions (such as changing the VSAN) require that you reset the interface status.
When you reset the interface status, the switch disables the interface and then immediately
reenables the interface.
Caution:
If an Ethernet link provides FCoE service, do not reset the
physical link because this action is disruptive to all
traffic on the link.
The logical link up/down feature allows the switch to reset an individual virtual link. The logical link down is signaled
with a FIP
Clear Virtual Link message.
Caution:
If the CNA does not support the logical link
level up/down feature, the CNA resets the physical link, which means that all
traffic on the Ethernet interface is disrupted.
Converged Network Adapters
Cisco NX-OS switches support the following CNA types:
Hardware adapter
Works with the existing Fibre Channel host bus adapter (HBA) driver and
Ethernet Network Interface Card (NIC) driver in the server.
Server operating system view of the network
is unchanged; the CNA presents a SAN interface and a LAN interface to the
operating system.
FCoE software stack
Runs on existing 10-Gigabit Ethernet
adapters.
The following Cisco NX-OS series and platforms support Generation 2 CNAs that use the FIP to
exchange information about its available capabilities and to negotiate the
configurable values with the switch:
Cisco Nexus 2232
Cisco Nexus 5000
Cisco Nexus 5500
Cisco Nexus 7000
Cisco MDS 9500
To reduce configuration errors and simplify
administration, the switch distributes the configuration data to all the
connected adapters.
STP Lite
FCoE does not require full Spanning Tree Protocol (STP) because FCoE has no bridging functionality, which means that no STP loops are created in the network. STP Lite on FCoE interfaces ensures rapid convergence across the network by sending an agreement Bridge Protocol Data Unit (BPDU) whenever it receives a proposal BPDU. The FCoE link sends the identical agreement BPDU in response to either an Multiple Spanning Tree (MST) or a Per VLAN Rapid Spanning Tree Plus (PVRST+) proposal BPDU. Additionally, STP Lite suppresses the MAC address flushing function for FCoE VLANs.
STP Lite is enabled automatically by default across the entire device for FCoE VLANs as soon as the first FCoE VLAN comes up. At the same time, the system automatically converts all FCoE links as the STP-type normal ports. This feature runs only in FCoE VLANs.
Default Settings for FCoE
This table lists the default settings for FCoE parameters.
Table 2
Default FCoE Parameter Settings
Parameters
Default
FCoE feature
Not installed, disabled
FC-Map
0E.FC.00
Fabric priority
128
Advertisement interval
8 seconds
Licensing Requirements for FCoE
Product
License
Cisco Nexus 7000 Series
Each F Series module that runs FCoE requires an FCoE license. FCoE enabled in a nondefault VDC does not require the Advanced Services License. For a complete explanation of the Cisco NX-OS licensing scheme and how to obtain and apply licenses, see the Cisco NX-OS Licensing Guide.
Cisco MDS 9500 Series
FCoE does not require a license. For a complete explanation of the Cisco NX-OS licensing scheme and how to obtain and apply licenses, see the Cisco NX-OS Licensing Guide.
The QoS policy must be the same on all Cisco FCoE switches in the network.
Beginning with Cisco NX-OS Release 6.1, FCoE is supported on F2 and F2e Series modules.
FCoE supports only F2e (SFP+) modules.
FCoE does not support F2e (Copper) modules.
FCoE VDC
FCoE in a dedicated storage VDC has the following guidelines:
Enable the FCoE feature set in only one VDC.
Create VLANs in the FCoE allocated VLAN range.
Do not enable any other features other than storage-related features in the dedicated FCoE VDC.
Allocate resources for the dedicated FCoE VDC from an F Series module, such as the 32-port 1- and 10-Gigabit Ethernet I/O module (PID N7K-F132XP-15) .
Rollback is not supported in a storage VDC.
FCoE support on F2 and F2e Series modules, SF248XP-25 requires Supervisor 2 module.
F2 and F2e series modules cannot exist in the same VDC with any other module type. This applies to both LAN and storage VDCs.
Shared Interfaces
Note
Any change in protocol state that flaps the parent port of a shared interface because of any port feature also affects the FCOE traffic on the storage vdc.
The following interface config modes are not allowed while sharing an interface from Ethernet vdc to a storage vdc:
SPAN destination
Private VLAN mode
Port-channel interfaces
Access mode
mac-packet-classify
Interfaces that are part of a VLAN that has an associated QoS policy
Shared Ethernet interfaces must be in trunk mode and only shared with one other VDC.
Storage VDC
Configuring a VDC for the OOB management interface mgmt0 is accomplished with the vrf context management command. However, a storage VDC does not support VRF, so configuring mgmt0 requires a different approach.
The following table shows how to configure mgmt 0 for a VDC and for a storage VDC:
Configuring mgmt 0 for VDC
Configuring mgmt 0 for storage VDC
vrf context management
ip route 0.0.0.0/0 default_gateway
interface mgmt 0
ip address mgmt0_ip_addressmgmt0_subnet_mask
no shut
ip route 0.0.0.0/0 default_gateway
Note
The ip route command specifies the default route that points to the default gateway.
where
mgmt0_ip_address is the mgmt0 IPv4 address.
mgmt0_subnet_mask is the mgmt0 IPv4 netmask.
default_gateway is the IPv4 address of the default-gateway.
For more information about VDC, see the Cisco Nexus 7000 Series NX-OS Virtual Device Context Configuration Guide.
Cisco MDS 9500: Guidelines and Limitations
You cannot enable FCoE on the default VLAN.
You can only enable FCoE on an FCoE module.
The QoS policy must be the same on all Cisco FCoE switches in the network.
Cisco MDS 9500 does not support LAN traffic and only accepts and processes FCoE frames.
You must install a Supervisor-2A module to configure FCoE.
You can install an FCoE module in a fabric that uses FC-Redirect services but do not install the FCoE module in the same switch where FC-Redirect services application modules are enabled.
SME does not support any FCoE connected devices including devices connected through the MDS FCoE linecard (DX-X9708-K9)
In Cisco MDS NX-OS Release 5.2(x), you cannot install a FCoE module in a switch that is running DMM, SME, or IOA.
Configuring FCoE
Enabling FCoE
Cisco Nexus 7000: Enabling FCoE
You must install the FCoE feature set in the default VDC and enable dependent features in order to enable FCoE in a storage VDC.
Before You Begin
Ensure you are in the default VDC.
Ensure you have installed the correct license for your switch.
Do not enable FCoE on VLAN 1 or the native VLAN.
SUMMARY STEPS
1.configure terminal
2.install feature-set fcoe
3.feature lldp
4.
(Optional) feature lacp
5.system qos
6.service-policy type network-qos policy-name
7.
(Optional) show feature
8.
(Optional) copy running-config startup-config
DETAILED STEPS
Command or Action
Purpose
Step 1
configure terminal
Example:
switch# configure terminal
switch(config)#
Enters configuration mode.
Step 2
install feature-set fcoe
Example:
switch(config)# install feature-set fcoe
Installs the FCoE feature set in the default VDC.
Step 3
feature lldp
Example:
switch(config)# feature lldp
Enables the Link Layer Discovery Protocol (LLDP) feature in the default VDC. This feature is required for FCoE operation.
Step 4
feature lacp
Example:
switch(config)# feature lacp
(Optional)
Enables the Link Aggregation Control Protocol (LACP) feature in the default VDC. This feature is considered a best practice for FCoE operation.
Step 5
system qos
Example:
switch(config)# system qos
Enters Quality of service (QoS) configuration mode.
Step 6
service-policy type network-qos policy-name
Example:
switch(config-sys-qos)# service-policy type
network-qos default-nq-7e-policy
Enables the QoS policy that supports FCoE traffic. The policy-name default is default-nq-8e-policy.
Copies the running configuration to the startup configuration.
What to Do Next
You must create a storage VDC and allocate resources to finish enabling FCoE.
Cisco MDS 9500: Enabling FCoE
The Cisco MDS 9500 switch automatically enables the FCoE feature when an FCoE-capable module is present in the chassis.
You must enable supporting features and the appropriate QoS policy for FCoE.
Before You Begin
Do not enable FCoE on VLAN 1 or the native VLAN.
SUMMARY STEPS
1.configure terminal
2.
(Optional) feature lacp
3.system qos
4.service-policy type network-qospolicy-name
5.
(Optional) show feature
6.
(Optional) copy running-config startup-config
DETAILED STEPS
Command or Action
Purpose
Step 1
configure terminal
Example:
switch# configure terminal
switch(config)#
Enters configuration mode.
Step 2
feature lacp
Example:
switch(config)# feature lacp
(Optional)
Enables the Link Aggregation Control Protocol (LACP) feature. This feature is considered a best practice for FCoE operation.
Step 3
system qos
Example:
switch(config)# system qos
Enters Quality of Service (QoS) configuration mode.
Step 4
service-policy type network-qospolicy-name
Example:
switch(config-sys-qos)# service-policy type
network-qos default-nq-7e-policy
Enables the QoS policy that supports FCoE traffic. The policy-name default is default-nq-6e-policy.
Copies the running configuration to the startup configuration.
Disabling FCoE
Cisco Nexus 7000: Disabling FCoE
You can disable or uninstall the FCoE feature set. You can also disallow the FCoE feature set in a VDC.
Before You Begin
Ensure you are in the correct VDC.
SUMMARY STEPS
1.configure terminal
2.vdcvdc_idtype storage
3.
(Optional) no allow feature-set fcoe
4.no feature-set fcoe
5.exit
6.no install feature-set fcoe
7.
(Optional) show feature-set
8.
(Optional) copy running-config startup-config
DETAILED STEPS
Command or Action
Purpose
Step 1
configure terminal
Example:
switch# configure terminal
switch(config)#
Enters configuration mode.
Step 2
vdcvdc_idtype storage
Example:
switch(config)# vdc fcoe type storage
switch(config-vdc)#
Enters VDC configuration mode. The vdc_id can be any case-sensitive, alphanumeric string up to 32 characters.
Step 3
no allow feature-set fcoe
Example:
switch(config-vdc)# no allow feature-set fcoe
switch(config-vdc)#
(Optional)
Disallows any FCoE in this VDC. The default is allow.
Step 4
no feature-set fcoe
Example:
switch(config-vdc)# no feature-set fcoe
Disables the FCoE feature set. The no feature-set fcoe command may take some time to complete if the size of the configuration is very large. The command must clean up all of the configuration associated with the FCoE feature set.
Step 5
exit
Example:
switch(config-vdc)# exit
switch(config)#
Exits VDC configuration mode.
Step 6
no install feature-set fcoe
Example:
switch(config)# no install feature-set fcoe
Uninstalls the FCoE feature set. Use this command in the default VDC after you disable the FCoE feature set.
Copies the running configuration to the startup configuration.
Cisco MDS 9500: Disabling FCoE
After you disable the FCoE feature, all FCoE
commands are removed from the CLI and the FCoE configuration is deleted.
SUMMARY STEPS
1.configure terminal
2.no feature-set fcoe
DETAILED STEPS
Command or Action
Purpose
Step 1
configure terminal
Example:
switch# configure terminal
switch(config)#
Enters configuration mode.
Step 2
no feature-set fcoe
Example:
switch(config)# no feature-set fcoe
Disables the FCoE feature set. The no feature-set fcoe command may take some time to complete if the size of the configuration is very large. The command must clean up all of the configuration associated with the FCoE feature set.
This example shows how to disable FCoE on
the switch:
switch# configure terminal
switch(config)# no feature-set fcoe
Configuring the FC-Map
You can prevent data corruption due to cross-fabric talk by configuring an FC-Map that identifies the Fibre Channel fabric for this switch. When the FC-Map
is configured, the switch discards the MAC addresses that are not part of the
current fabric.
An FCF can assign Fabric Provide MAC Addresses (FPMA) to the CNAs consisting of the FC-Map Value for the Fabric and the Fibre Channel ID (FCID) assigned during Fabric Login
Note
For Cisco MDS 9500 switches, start at configure terminal below .
Before You Begin
For Cisco Nexus 7000 Series, you must be in the storage VDC to configure this feature.
Switches to the storage VDC. The step is required for Cisco Nexus 7000 Series only.
Step 2
configure terminal
Example:
fcoe# configure terminal
fcoe(config)#
Enters configuration mode.
Step 3
fcoe fcmapfabric-map
Example:
fcoe(config)# fcoe fcmap 0x0efc2a
Configures the global FC-Map. The default value is 0x0EFC00. The
range is from 0x0EFC00 to 0x0EFCFF.
Use the no fcoe map command to reset to the default value.
This example shows how to configure the global FC-Map on a Cisco Nexus 7000 Series switch:
Switches to the storage VDC. The step is required for Cisco Nexus 7000 Series only.
Step 2
configure terminal
Example:
fcoe# configure terminal
fcoe(config)#
Enters configuration mode.
Step 3
fcoe fcf-priorityfabric-priority
Example:
fcoe(config)# fcoe fcf-priority 42
Configures the global fabric priority. The
default value is 128. The range is from 0 (higher) to 255 (lower).
Use the no fcoe fcf-priority command to reset the global fabric priority to the default value.
This example shows how to configure the
global fabric priority for a Cisco Nexus 7000 Series switch:
DCBX allows the switch to send a LAN Logical Link
Status (LLS) message to a directly-connected CNA. Enter the
shutdown lan command to send an LLS-Down
message to the CNA. This command causes all VLANs on the interface that are not enabled for FCoE
to be brought down. If a VLAN on the interface is enabled for
FCoE, it continues to carry SAN traffic without any interruption.
SUMMARY STEPS
1.configure terminal
2.interfaceethernetslot/port
3.shutdown lan
4.
(Optional) show interface
5.
(Optional) copy running-config startup-config
DETAILED STEPS
Command or Action
Purpose
Step 1
configure terminal
Example:
switch# configure terminal
switch(config)#
Enters configuration mode.
Step 2
interfaceethernetslot/port
Example:
switch(config)# interface e 2/1
switch(config-if)#
Specifies an interface to configure, and enters interface
configuration mode.
Use ? to view a list of supported interfaces.
Step 3
shutdown lan
Example:
switch(config-if)# shutdown lan
Shuts down Ethernet traffic on the interface. If the interface is
part of an FCoE VLAN, the shutdown has no impact on the FCoE traffic.
Use no shutdown lan to reenable Ethernet traffic on this interface.
Copies the running configuration to the startup configuration.
Verifying the FCoE Configuration
To display FCoE configuration information, perform
one of these tasks:
Command
Purpose
show fcoe
Displays whether FCoE is enabled on the
switch.
show fcoe
database
Displays the contents of the FCoE
database.
show interface vfc
[number]
Displays information about the vFC interfaces.
show interface
[interfacenumber]
fcoe
Displays the FCoE settings for an
interface or all interfaces.
This example shows how to verify that the
FCoE capability is enabled:
switch# show fcoe
Global FCF details
FCF-MAC is 00:0d:ec:6d:95:00
FC-MAP is 0e:fc:00
FCF Priority is 128
FKA Advertisement period for FCF is 8 seconds
This example shows how to display the FCoE
database:
switch# show fcoe database
------------------------------------------------------------------------------
INTERFACE FCID PORT NAME MAC ADDRESS
------------------------------------------------------------------------------
vfc3 0x490100 21:00:00:1b:32:0a:e7:b8 00:c0:dd:0e:5f:76
This example shows how to display the FCoE
settings for an interface.
switch# show interface ethernet 1/37 fcoe
Ethernet1/37 is FCoE UP
vfc3 is Up
FCID is 0x490100
PWWN is 21:00:00:1b:32:0a:e7:b8
MAC addr is 00:c0:dd:0e:5f:76
Additional References for FCoE
Related Documents
Related Topic
Document Title
Command reference
Cisco NX-OS FCoE Command Reference for Cisco Nexus 7000 and Cisco MDS 9500
Cisco NX-OS licensing
Cisco NX-OS Licensing Guide
Standards and RFCs
Standard/RFC
Title
T11 FC BB-5
Fibre Channel Backbone 5
MIBs
MIB
MIBs Link
To locate and download MIBs for selected platforms, Cisco IOS
releases, and feature sets, use Cisco MIB Locator found at the
following URL:
The Cisco Support website provides extensive online resources,
including documentation and tools for troubleshooting and
resolving technical issues with Cisco products and technologies.
To receive security and technical information about your
products, you can subscribe to various services, such as the
Product Alert Tool (accessed from Field Notices), the Cisco
Technical Services Newsletter, and Really Simple Syndication
(RSS) Feeds.
Access to most tools on the Cisco Support website requires a
Cisco.com user ID and password.