FCoE Over FEX

Information About FCoE Over FEX

The Fibre Channel over Ethernet (FCoE) over Fabric Extenders (FEX) feature allows Fibre Channel traffic to be carried on a FEX port. To enable this feature, the FEX port is shared with the storage Virtual Device Context (VDC). The FEX is connected to the Cisco Nexus 7000/7700 device through a Fabric Port Channel (FPC). FCoE over FEX enables provision of FCoE on host connections.

FCoE Over FEX With Physical Port vPC

The FCoE over FEX with Virtual PortChannel (vPC) feature allows Fibre Channel traffic to be carried over a FEX using a physical port virtual PortChannel (vPC). To enable this feature, the FEX port is shared with the storage VDC.

LAN Shutdown

The LAN Shutdown feature detects the capability of the FCoE host to support Data Center Bridging Exchange (DCBX). DCBX allows the switch to send LAN Logical Link Status (LLS) messages in a type-length-value (TLV) format. The LAN shutdown feature enables bring up and bring down of LAN links on a unified link carrying both FCoE and LAN traffic. When you enable the shutdown lan command, only the LAN traffic stops while the FCoE traffic continues.

Default Settings for FCoE

This table lists the default settings for FCoE parameters.

Table 1. Default FCoE Parameter Settings

Parameters

Default

FCoE feature

Not installed, disabled

FC-Map

0E.FC.00

Fabric priority

128

Advertisement interval

8 seconds

Guidelines and Limitations

FCoE Over FEX

FCoE over FEX has the following guidelines and limitations:

  • FCoE over FEX is supported with 2232PP, B22-HP, N2348UPQ FEX.

  • To enable FCoE over FEX on the storage VDC, use the feature-set fex command in the storage VDC.

  • FCoE license should be enabled on the FEX uplink port on the parent switch.

  • To enable support of FCoE over FEX on the parent switch explicitly, use the hardware qos fcoe-fex command.

  • Inter VSAN route (IVR) zone configuration is not supported for FCoE over FEX.

  • Custom QoS policy cannot be applied on the uplink interfaces of the FEX Port channel.

  • FCoE over host interface port channel (HIF-PC) is not supported.

  • If you connect a FEX with FPC ports that is sharing its ports with a storage VDC to an F2e module that is also sharing its ports with the same storage VDC, the port manager times out when you reload the F2e module. The ports on the F2e module that were shared with the storage VDC will be in the removed state in the storage VDC. You will have to reload the switch to recover from this state.

  • FCoE over FEX is not supported on Active/Active FEX.


Note
For more information about FEX, refer http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_4_0_1a/NX2000CLIConfig/FEX-features.html

FCoE over FEX with Physical Port vPC

FCoE over FEX with Physical Port vPC has the following guidelines and limitations:

  • FCoE over physical port vPC+ is supported.

  • FCoE over physical port vPC is supported on shared interfaces.

  • FCoE over physical port vPC is supported only on host facing links.

  • Port channel vPC does not support FCoE.

  • Both vPC legs should be shared between the ethernet VDC and the storage VDC.

LAN Shutdown

The LAN Shutdown feature has the following guidelines and limitations:

  • LAN shutdown is supported only on shared interfaces in the parent virtual device context (VDC).

  • LAN shutdown requires Link Layer Discovery Protocol (LLDP) to run in the parent VDC.

FCoE Over FEX

Configuring FCoE over FEX

SUMMARY STEPS

  1. configure terminal
  2. hardware qos fcoe-fex
  3. license fcoe module module-number
  4. exit
  5. switchto vdc ethernet-vdc-name
  6. configure terminal
  7. interface port-channel channel-number
  8. switchport
  9. switchport mode fex-fabric
  10. priority-flow-control mode { auto | off | on}
  11. fex associate chassis-id
  12. no shutdown
  13. interface ethernet slot/port-list
  14. switchport
  15. switchport mode fex-fabric
  16. priority-flow-control mode { auto | off | on}
  17. fex associate chassis-id
  18. channel-group number
  19. no shutdown
  20. interface if-range
  21. switchport
  22. switchport mode trunk
  23. spanning tree port type edge trunk
  24. no shutdown
  25. exit
  26. feature lldp
  27. switchback
  28. switchto vdc storage-vdc-name
  29. configure terminal
  30. feature-set fex
  31. feature lldp
  32. switchback
  33. configure terminal
  34. vdc storage-vdc-name
  35. allocate fcoe-vlan-range vlan-range from vdc ethernet-vdc-name
  36. [no] allocate shared interface if-range

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode in the default VDC.

Step 2

hardware qos fcoe-fex

Example:

switch(config)# hardware qos fcoe-fex

Enables FCoE over FEX support and Priority Flow Control (PFC) behavior on the FEX port. Use the no form of the command to disable FCoE over FEX support.

Step 3

license fcoe module module-number

Example:

switch(config)# license fcoe module 3

Associates an FCoE license to a module in which the fabric port channel (FPC) ports are present. .

Step 4

exit

Example:

switch(config)# exit

Exits the default VDC configuration mode.

Step 5

switchto vdc ethernet-vdc-name

Example:

switch# switchto vdc ethernet-vdc

Switches to the ethernet VDC from the default VDC.

Step 6

configure terminal

Example:

switch-ethernet-vdc# configure terminal
switch-ethernet-vdc(config)#

Enters configuration mode.

Step 7

interface port-channel channel-number

Example:

switch-ethernet-vdc(config)# interface port-channel 4
switch-ethernet-vdc(config-if)#

Creates a port-channel interface and enter interface configuration mode.

Step 8

switchport

Example:

switch-ethernet-vdc(config-if)# switchport

Sets the interface as a Layer 2 switching port.

Step 9

switchport mode fex-fabric

Example:

switch-ethernet-vdc(config-if)# switchport mode fex-fabric

Sets the interface type to be an uplink port for a Fabric Extender (FEX).

Step 10

priority-flow-control mode { auto | off | on}

Example:

switch-ethernet-vdc(config-if)# priority-flow-control mode on
Configure Priority Flow Control on a per-port basis and on the port channel.

Note

 

This command is enabled by default when you enable the hardware qos fcoe-fex command in the default VDC.

Step 11

fex associate chassis-id

Example:

switch-ethernet-vdc(config-if)# fex associate 100

Associates a Fabric Extender (FEX) to a fabric interface.

Step 12

no shutdown

Example:

switch-ethernet-vdc(config-if)# no shutdown

Administratively enables the Ethernet shared interface.

Step 13

interface ethernet slot/port-list

Example:

switch-ethernet-vdc(config)# interface ethernet 1/37
switch-ethernet-vdc(config-if)#

Configures an Ethernet interface and enter interface configuration mode.

Step 14

switchport

Example:

switch-ethernet-vdc(config-if)# switchport

Sets the interface as a Layer 2 switching port.

Step 15

switchport mode fex-fabric

Example:

switch-ethernet-vdc(config-if)# switchport mode fex-fabric

Sets the interface type to be an uplink port for a Fabric Extender (FEX).

Step 16

priority-flow-control mode { auto | off | on}

Example:

switch-ethernet-vdc(config-if)# priority-flow-control mode on

Configure Priority Flow Control on a per-port basis.

Note

 

This command is enabled by default when you enable the hardware qos fcoe-fex command in the default VDC.

Step 17

fex associate chassis-id

Example:

switch-ethernet-vdc(config-if)# fex associate 100

Associates a Fabric Extender (FEX) to a fabric interface.

Step 18

channel-group number

Example:

switch-ethernet-vdc(config-if)# channel-group 4

Assigns and configures a physical interface to a port-channel group.

Step 19

no shutdown

Example:

switch-ethernet-vdc(config-if)# no shutdown

Administratively enables the Ethernet shared interface.

Step 20

interface if-range

Example:

switch-ethernet-vdc(config-if)# interface ethernet 100/1/1
switch-ethernet-vdc(config-if)#

Configures the interface and enters interface configuration mode in the ethernet VDC.

Step 21

switchport

Example:

switch-ethernet-vdc(config-if)# switchport

Sets the interface as a Layer 2 switching port.

Step 22

switchport mode trunk

Example:

switch-ethernet-vdc(config-if)# switchport mode trunk

Puts the Ethernet interface into trunk mode.

Step 23

spanning tree port type edge trunk

Example:

switch-ethernet-vdc(config-if)# spanning tree port type edge trunk

Configures an interface connected to a host as an edge port

Step 24

no shutdown

Example:

switch-ethernet-vdc(config-if)# no shutdown

Administratively enables the Ethernet shared interface.

Step 25

exit

Example:

switch-ethernet-vdc(config-if)# exit

Exits the interface configuration mode.

Step 26

feature lldp

Example:

switch-ethernet-vdc(config)# feature lldp

Enables the LLDP feature in the ethernet VDC.

Step 27

switchback

Example:

switch-ethernet-vdc(config)# switchback
switch#

Go to default VDC.

Step 28

switchto vdc storage-vdc-name

Example:

switch# switchto vdc storage-vdc
switch-storage-vdc#

Switches to the storage VDC.

Step 29

configure terminal

Example:

switch-storage-vdc# configure terminal
switch-storage-vdc(config)#

Enters configuration mode.

Step 30

feature-set fex

Example:

switch-storage-vdc(config)# feature-set fex

Enables the Fabric Extender (FEX) feature set in the storage VDC.

Step 31

feature lldp

Example:

switch-storage-vdc(config)# feature lldp

Enables the LLDP feature in the storage VDC.

Step 32

switchback

Example:

switch-storage-vdc(config)# switchback
switch#

Go to default VDC.

Step 33

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode in the default VDC.

Step 34

vdc storage-vdc-name

Example:

switch(config)# vdc storage-vdc

Specifies the storage VDC in the default VDC.

Step 35

allocate fcoe-vlan-range vlan-range from vdc ethernet-vdc-name

Example:

switch(config-vdc)# allocate fcoe-vlan-range 2-8 from vdc ethernet-vdc

Allocates Fibre Channel over Ethernet (FCoE) VLANs to a virtual device context (VDC),

Step 36

[no] allocate shared interface if-range

Example:

switch(config-vdc)# allocate shared interface ethernet 100/1/1
Ports that share the port group of the interfaces you have specified will be affected as well. Continue (y/n)? y

Shares FEX interface from the Ethernet VDC to the storage VDC.

Note

 

It is expected that, if the feature-set FEX is not enabled, then the allocate shared interface fex_interfaces command fails on VDC write-erase/reload. To resolve, un-configure the shared interfaces and configure it back.

Example

This example shows how to configure FCoE over FEX: 

!Default VDC
switch# configure terminal
switch(config)# hardware qos fcoe-fex
switch(config)# license fcoe module 3
switch(config)# exit
switch# switchto vdc ethernet-vdc
!Ethernet VDC
switch-ethernet-vdc# configure terminal
switch-ethernet-vdc(config)# interface port-channel 4
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode fex-fabric
switch-ethernet-vdc(config-if)# priority-flow-control mode on
switch-ethernet-vdc(config-if)# fex associate 100
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config)# interface ethernet 1/37
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode fex-fabric
switch-ethernet-vdc(config-if)# priority-flow-control mode on
switch-ethernet-vdc(config-if)# fex associate 100
switch-ethernet-vdc(config-if)# channel-group 4
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config-if)# interface ethernet 100/1/1
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode trunk
switch-ethernet-vdc(config-if)# spanning tree port type edge trunk
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config-if)# exit
switch-ethernet-vdc(config)# feature lldp
switch-ethernet-vdc(config)# switchback
!Default VDC
switch# switchto vdc storage-vdc
!Storage VDC
switch-storage-vdc# configure terminal
switch-storage-vdc(config)# feature-set fex
switch-storage-vdc(config)# feature lldp
switch-storage-vdc(config)# switchback
!Default VDC
switch# configure terminal
switch(config)# vdc storage-vdc
switch(config-vdc)# allocate fcoe-vlan-range 2-8 from vdc ethernet-vdc
switch(config-vdc)# allocate shared interface ethernet 100/1/1

Binding VFC to Shared FEX Interface Explicitly

SUMMARY STEPS

  1. configure terminal
  2. interface vfc vfc-id
  3. bind {interface {ethernet [chassid-id]/ slot/ port}}
  4. no shutdown
  5. (Optional) show interface vfc
  6. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch-storage-vdc# configure terminal
switch-storage-vdc(config)#

Enters configuration mode.

Step 2

interface vfc vfc-id

Example:

switch-storage-vdc(config)# interface vfc 100
switch-storage-vdc(config-if)#

Creates a virtual Fibre Channel interface (if it does not already exist) and enters interface configuration mode. The chassis-id range is from 100 to 199.

Step 3

bind {interface {ethernet [chassid-id]/ slot/ port}}

Example:

switch-storage-vdc(config-if)# bind interface ethernet 100/1/1

Explicitly binds the virtual Fibre Channel interface to the specified interface. Use the no form of this command to unbind the virtual Fibre Channel interface from the specified interface. The chassis-id range is from 100 to 199.

Step 4

no shutdown

Example:

switch-storage-vdc(config-if)# no shutdown

Administratively enables the Ethernet shared interface.

Step 5

(Optional) show interface vfc

Example:

switch-storage-vdc(config-if)# show interface vfc
 (Optional)

Displays information about the virtual Fibre Channel interfaces.

Step 6

(Optional) copy running-config startup-config

Example:

switch-storage-vdc(config)# copy running-config startup-config
 (Optional)

Copies the running configuration to the startup configuration.

Example

This example shows how to bind a virtual Fibre Channel interface to a FEX interface: 

switch-storage-vdc# configure terminal
switch-storage-vdc(config)# interface vfc 100
switch-storage-vdc(config-if)# bind interface ethernet 100/1/1
switch-storage-vdc(config-if)# no shutdown

Binding VFC to Shared FEX Interface Implicitly

SUMMARY STEPS

  1. configure terminal
  2. interface vfc [chassid-id]/ slot/ port
  3. no shutdown
  4. (Optional) show interface vfc
  5. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch-storage-vdc# configure terminal
switch-storage-vdc(config)#

Enters configuration mode.

Step 2

interface vfc [chassid-id]/ slot/ port

Example:

switch-storage-vdc(config)# interface vfc 100/1/1
switch-storage-vdc(config-if)#

Creates a virtual Fibre Channel interface (if it does not already exist) and enters interface configuration mode. This command binds the VFC to the shared FEX interface implicitly. The chassis-id range is from 100 to 199. This command binds the Ethernet interface 100/1/1 to the shared FEX interface.

Step 3

no shutdown

Example:

switch-storage-vdc(config-if)# no shutdown

Administratively enables the Ethernet shared interface.

Step 4

(Optional) show interface vfc

Example:

switch-storage-vdc(config-if)# show interface vfc
 (Optional)

Displays information about the virtual Fibre Channel interfaces.

Step 5

(Optional) copy running-config startup-config

Example:

switch-storage-vdc(config)# copy running-config startup-config
 (Optional)

Copies the running configuration to the startup configuration.

Example

This example shows how to bind a virtual Fibre Channel interface to a FEX interface implicitly: 

switch-storage-vdc# configure terminal
switch-storage-vdc(config)# interface vfc 100/1/1
switch-storage-vdc(config)# no shutdown

Verifying the FCoE over FEX Configuration

To display FCoE over FEX configuration information, perform one of these tasks:

Command

Purpose

show fex

Displays information about all Fabric Extenders (FEX) available in a specific VDC. Use this command in storage VDC and ethernet VDC.

show fex detail

Displays detailed information about all Fabric Extenders (FEX) available in a specific VDC. Use this command in storage VDC and ethernet VDC.

show fex chassis-id

Displays information about a specific FEX. Use this command in storage VDC and ethernet VDC.

show vdc shared membership

Displays the FEX interfaces shared with the storage virtual device context (VDC). Use this command in the default VDC.

show vdc shared membership both

Displays the FPC ports that are implicitly shared as a result of the shared FEX interface. Use this command in the default VDC.

show flogi database

Lists all the FLOGI sessions. Use this command in the storage VDC.

This example shows how to display information about all attached Fabric Extender (FEX) chassis: 

switch-storage-vdc# show fex
FEX             FEX                 FEX                            FEX

Number        Description            State                 Model               Serial
------------------------------------------------------------------------------------------
100           FEX0100                Online                N2K-C2232PP-10GE    SSI181202V9

This example shows how to display detailed information about all attached FEX chassis: 

switch-ethernet-vdc# show fex detail
 FEX: 101 Description: FEX0100   state: Online
  FEX version: 7.2(0)D1(1) [Switch version: 7.2(0)D1(1)]
  FEX Interim version: (1)FIP_0_28
  Switch Interim version: 7.2(0)D1(1)
  Extender Serial: SSI181202V9
  Extender Model: N2K-C2232PP-10GE,  Part No: 73-12533-06
  Card Id: 82, Mac Addr: 58:0a:20:37:e4:02, Num Macs: 64
  Keepalive pending for 1 intervals
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
  Pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/19
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Po100 - Interface Up. State: Active
    Eth1/37 - Interface Up. State: Active
  Fex Port        State  Fabric Port
       Eth100/1/1  Up         Po4
       Eth100/1/2 Down        Po4
       Eth100/1/3  Down       Po4
       Eth100/1/4  Down       Po4
       Eth100/1/5  Down       Po4
       Eth100/1/6  Down       Po4
       Eth100/1/7  Down       Po4
       Eth100/1/8  Down       Po4
       Eth100/1/9  Down       Po4
      Eth100/1/10  Down       Po4
      Eth100/1/11  Down       Po4
      Eth100/1/12  Down       Po4
      Eth100/1/13  Down       Po4
      Eth100/1/14  Down       Po4
      Eth100/1/15  Down       Po4
      Eth100/1/16  Down       Po4
      Eth100/1/17  Down       Po4
      Eth100/1/18  Down       Po4
      Eth100/1/19  Down       Po4
      Eth100/1/20  Down       Po4
      Eth100/1/21  Down       Po4
      Eth100/1/22  Down       Po4
      Eth100/1/23  Down       Po4
      Eth100/1/24  Down       Po4
      Eth100/1/25  Down       Po4
      Eth100/1/26  Down       Po4
      Eth100/1/27  Down       Po4
      Eth100/1/28  Down       Po4
      Eth100/1/29  Down       Po4
      Eth100/1/30  Down       Po4
      Eth100/1/31  Down       Po4
      Eth100/1/32  Down       Po4
<---output truncated--->

This example shows how to display information about a specific FEX: 

switch-storage-vdc# show fex 100
FEX: 101 Description: FEX0100   state: Online
  FEX version: 7.2(0)D1(1) [Switch version: 7.2(0)D1(1)]
  Extender Serial: SSI181202V9
  Extender Model: N2K-C2232PP-10GE,  Part No: 73-12533-06
  Keepalive pending for 1 intervals
  Pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/37
  FCoE Admin: false
  FCoE Oper: false
  FCoE FEX AA Configured: false
  Fabric interface state:
    Po101 - Interface Up. State: Active
    Eth3/1 - Interface Up. State: Active
    Eth3/2 - Interface Up. State: Active

This example shows how to display the FEX interfaces shared with the storage virtual device context (VDC): 

switch# show vdc shared membersip
vdc_id: 3 vdc_name: fcoe-vdc interfaces:
        Ethernet100/1/1

This example shows how to display the FPC ports that are implicitly shared as a result of the shared FEX interface: 

switch# show vdc shared membership both
vdc_id: 3 vdc_name: fcoe-vdc interfaces:
        Ethernet100/1/1    
Implicitly shared interfaces:
        Ethernet100/1/2

This example shows how to list the FLOGI sessions: 

switch-storage-vdc# show flogi database
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
--------------------------------------------------------------------------------
vfc100/1/1       100   0x800020  10:00:00:00:06:67:e9:00 20:00:00:00:06:67:e9:00

Total number of flogi = 1.

Example: FCoE Over FEX Configuration 

!Configuring FCoE over FEX:
!Default VDC
switch# configure terminal
switch(config)# hardware qos fcoe-fex
switch(config)# license fcoe module 3
switch(config)# exit
switch# switchto vdc ethernet-vdc
!Ethernet VDC
switch-ethernet-vdc# configure terminal
switch-ethernet-vdc(config)# interface port-channel 4
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode fex-fabric
switch-ethernet-vdc(config-if)# priority-flow-control mode on
switch-ethernet-vdc(config-if)# fex associate 100
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config)# interface ethernet 1/37
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode fex-fabric
switch-ethernet-vdc(config-if)# priority-flow-control mode on
switch-ethernet-vdc(config-if)# fex associate 100
switch-ethernet-vdc(config-if)# channel-group 4
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config-if)# interface ethernet 100/1/1
switch-ethernet-vdc(config-if)# switchport
switch-ethernet-vdc(config-if)# switchport mode trunk
switch-ethernet-vdc(config-if)# spanning tree port type edge trunk
switch-ethernet-vdc(config-if)# no shutdown
switch-ethernet-vdc(config-if)# exit
switch-ethernet-vdc(config)# feature lldp
switch-ethernet-vdc(config)# switchback
!Default VDC
switch# switchto vdc storage-vdc
!Storage VDC
switch-storage-vdc# configure terminal
switch-storage-vdc(config)# feature-set fex
switch-storage-vdc(config)# feature lldp
switch-storage-vdc(config)# switchback
!Default VDC
switch# configure terminal
switch(config)# vdc storage-vdc
switch(config-vdc)# allocate fcoe-vlan-range 2-8 from vdc ethernet-vdc
switch(config-vdc)# allocate shared interface ethernet 100/1/1

!Bind VFC to Shared FEX Interface Explicitly:
switch-storage-vdc# configure terminal
switch-storage-vdc(config)# interface vfc 100
switch-storage-vdc(config-if)# bind interface ethernet 100/1/1
switch-storage-vdc(config-if)# no shutdown

!Bind VFC to Shared FEX Interface Implicitly:
switch-storage-vdc# configure terminal
switch-storage-vdc(config)# interface vfc 100/1/1
switch-storage-vdc(config)# no shutdown

Additional References for FCoE over FEX

Related Documents

Related Topic Document Title

Command reference

Cisco NX-OS FCoE Command Reference for Cisco Nexus 7000

Cisco NX-OS licensing

Cisco NX-OS Licensing Guide

Configuring vPCs

Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide

MIBs

MIB MIBs Link

To locate and download MIBs for selected platforms, Cisco IOS releases, and feature sets, use Cisco MIB Locator found at the following URL:

http://www.cisco.com/go/mibs

Technical Assistance

Description Link

The Cisco Support website provides extensive online resources, including documentation and tools for troubleshooting and resolving technical issues with Cisco products and technologies.

To receive security and technical information about your products, you can subscribe to various services, such as the Product Alert Tool (accessed from Field Notices), the Cisco Technical Services Newsletter, and Really Simple Syndication (RSS) Feeds.

Access to most tools on the Cisco Support website requires a Cisco.com user ID and password.

http://www.cisco.com/support