UCS FCoE Uplink Configuration Example

Introduction

This document describes how to configure Fibre Channel over Ethernet (FCoE) uplinks. In Cisco Unified Computing System (UCS) Version 2.1 and later Versions, multihop Fibre Channel over Ethernet (FCoE) support was introduced. This support allows uplink consolidation from two separate links to a single uplink that carries both Fibre Channel (FC) and Ethernet.

Note: Unified uplinks are different from unified ports. Any fabric interconnect (FI), which includes generation 1, can have unified uplinks. However, only generation 2 FIs have unified ports.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on UCS Manager (UCSM) Version 2.1. You must run at least Version 2.1(1a) in order to use this document.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Configure

Supported Configurations

In all of these configurations, you can replace any link with a port-channel of the same type. For example, if the configuration shows one link, it can be configured with one port-channel instead. However, you cannot have an Ethernet and a Unified port in the same port-channel.

FCoE Uplink with virtual Port Channel (vPC)

This configuration contains no unified links, but it is the easiest way to integrate FCoE uplinks into your current environment. This configuration eliminates the need to cross the FC fabrics.

Cross-connected with Pinning

In this configuration, the Nexus 5000 Series switches (N5ks) and FIs are cross-connected, but not port-channeled together. This allows you to have some unified uplinks, but you still maintain a well-known network design.

All Unified Uplinks

This configuration provides the most consolidation. Remember that these links can also be port-channels, but all links must be unified uplinks. This is likely to be the most common implementation.

Configuration on the UCS

There are three steps to configure the UCS:

1. Configure Virtual Storage Area Network (VSAN).
2. Configure Unified Uplinks.
3. Configure Port-channels (optional).

Configure VSAN

In this configuration, VSAN 500 is out of Fabric A, and VSAN 600 is out of Fabric B. This is the same configuration you would use in order to create a VSAN for a normal FC.

Configure Unified Uplinks

The Unified uplink is hidden in the GUI. In order to make a unified port, you must first configure it as an uplink or a FCoE uplink. Then, you can configure it as a unified port. This configuration does not remove the first role. Instead, it creates an interface that has both.

Make sure to set your VSAN at this point.

At this point, your UCS is configured for unified traffic. This example does not use port-channels. However, you can configure port-channels normally at this point. Make sure to set the VSAN under the VSAN tab for the port-channel.

Configuration on the N5k

Configuration on the N5k consists of 3 steps:

1. Enable FCoE and default Quality of Service (QoS) settings. Create the FCoE VLAN and bind the VSAN to it.
2. Configure the Ethernet interface (or port-channel).
3. Create a virtual Fibre Channel (vFC) and bind it to the Ethernet interface.

Note: The UCS is a N Port Virtualization (NPV) switch by default so the upstream switch needs to be in N Port Identifer Virtulization (NPIV) mode (enter the feature npiv command in order to enable).  See Configuring N Port Virtualization for more information on this feature.

Enable FCoE and Default QoS Settings

You can skip this step if the N5k is already configured for FCoE. If the N5k is not configured for FCoE, refer to Appendix A at the end of this document for instruction.

Configure the Ethernet Interfaces (or Port-channels)

This is the configuration for the interface that normally connects to the UCS.  Make sure to allow the FCoE VLAN.

f340-31-15-5596-1(config)# interface ethernet 1/32
f340-31-15-5596-1(config-if)# switchport mode trunk
f340-31-15-5596-1(config-if)# switchport trunk allowed vlan all

This is an example of a very basic trunking configuration. Notice that there is no FCoE configuration on this interface.

Configure the vFC

This is the configuration for the vFC, which handles all the FCoE traffic from the UCS:

f340-31-15-5596-1(config-if)# int vfc 25
f340-31-15-5596-1(config-if)# switchport trunk allowed vsan 500
f340-31-15-5596-1(config-if)# no shut
f340-31-15-5596-1(config-if)# bind interface ethernet 1/32
f340-31-15-5596-1(config-if)#

There are a few points to notice in this example:

• The vFC number does not matter. It can be anything, as long as it is not already in use.
• This is a Trunking Fabric (TF) port, so make sure to allow your VSAN.
• Bind this to the interface that faces the UCS. This is the physical interface on which FCoE traffic is sent/received. It is sent on the VLAN associated with the VSAN.

At this point, you have everything fully configured. Enter the show interface vfc xx command in order to ensure that the VSAN comes up.

f340-31-15-5596-1(config-if)# show interface vfc 25
vfc25 is trunking
    Bound interface is Ethernet1/32
    Hardware is Ethernet
    Port WWN is 20:18:54:7f:ee:d3:f8:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 500
    Trunk vsans (admin allowed and active) (500)
    Trunk vsans (up)                       (500)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 4448 bits/sec, 556 bytes/sec, 1 frames/sec
    1 minute output rate 610448 bits/sec, 76306 bytes/sec, 27 frames/sec
      27347 frames input, 45743308 bytes
        0 discards, 0 errors
      114710 frames output, 218235936 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters Thu May  9 13:37:42 2013

     Interface last changed at Thu May  9 14:03:44 2013

If the VSAN is not up, verify your configuration and your connections. If it is still down, flap the Ethernet port on the N5k.

At this point, you apply your normal FC configuration (boot policy, zoning).

f340-31-15-5596-1(config-if)# show flogi database
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
--------------------------------------------------------------------------------
fc2/3            500   0xd80000  20:42:00:0d:ec:b1:34:c0 21:f4:00:0d:ec:b1:34:c1
fc2/3            500   0xd80001  20:00:00:25:b5:12:dc:5f 20:00:00:25:b5:13:dc:2f
                           [svinfra-esxi-1]
fc2/3            500   0xd80003  20:00:00:25:b5:12:dc:3f 20:00:00:25:b5:13:dc:0f
                           [svinfra-esxi-2]
vfc7             500   0xd80060  25:f6:00:0d:ec:cc:d3:ff 21:f4:00:0d:ec:cc:d3:c1
vfc7             500   0xd80061  20:00:00:25:b5:19:70:28 20:00:00:25:b5:19:70:18
vfc25            500   0xd80042  20:00:00:25:b5:ab:1a:01 20:00:00:25:b5:00:00:8f
vfc25            500   0xd80043  20:00:00:25:b5:ab:1a:02 20:00:00:25:b5:00:00:9f
vfc25            500   0xd80044  20:00:00:25:b5:00:3a:08 20:00:00:25:b5:00:00:5f
vfc25            500   0xd80045  20:00:00:25:b5:00:3a:09 20:00:00:25:b5:00:00:4f
vfc25            500   0xd80048  2d:9a:00:05:73:c2:c6:3f 21:f4:00:05:73:c2:c6:01

Verify

There is currently no verification procedure available for this configuration.

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Appendix A - N5k basic FCoE Configuration

In order to configure the N5k for FCoE, enter:

feature fcoe
system qos 
  service-policy type network-qos fcoe-default-nq-policy
  service-policy type queuing input fcoe-default-in-policy
  service-policy type queuing output fcoe-default-out-policy
  service-policy type qos input fcoe-default-in-policy

These commands first enable FCoE, and then enable the default QoS policies that are needed for FCoE. If you have custom QoS settings, use those as a base. Here are the default configurations:

  policy-map type qos fcoe-default-in-policy
    class type qos class-fcoe
      set qos-group 1
    class type qos class-default
      set qos-group 0

   policy-map type queuing fcoe-default-in-policy
    class type queuing class-fcoe
      bandwidth percent 50
    class type queuing class-default
      bandwidth percent 50
  policy-map type queuing fcoe-default-out-policy
    class type queuing class-fcoe
      bandwidth percent 50
    class type queuing class-default
      bandwidth percent 50

   policy-map type network-qos fcoe-default-nq-policy
    class type network-qos class-fcoe

       pause no-drop
      mtu 2158
    class type network-qos class-default

       mtu 1500
      multicast-optimize

As you can see, these policies create the FCoE traffic class, mark it for no-drop, and give it dedicated bandwidth. You can modify these as you need (for example, in order to add Jumbo Frames).

Next, you need to bind the VSAN to the VLAN:

f340-31-15-5596-1(config)# vlan 500
f340-31-15-5596-1(config-vlan)# name VNX_500
f340-31-15-5596-1(config-vlan)# fcoe vsan 500
f340-31-15-5596-1(config-vlan)#

This tells the N5k that VLAN 500 is an FCoE VLAN and that any traffic that comes in should be switched to VSAN 500.

At this point, your N5k is configured for FCoE and ready for the UCS connection.