Contents
Introduction
This document describes a Nexus 5000 in a Virtual Port Channel (VPC) configuration to a NetApp storage to achieve Fibre Channel over Ethernet (FCoE).
Prerequisites
Requirements
Cisco recommends that you have knowledge of Unified Computing System (UCS) and Nexus 5000.
Components Used
The information in this document is based on these software and hardware versions:
- 2x Nexus 5548 - runs NXOS 5.2. Called bdsol-n5548-05 and -06-.
- 2x UCS 6248 fabric interconnects (FIs) - runs 2.2 software. Called bdsol-6248-03-, A and B.
- 2x NetApp 3220 - runs version 8.1. Called bdsol-3220-01-, A and B.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
This document describes the configuration of the Nexus 5000 switch in a Virtual Port Channel (VPC) configuration to a NetApp storage in order to achieve Fibre Channel over Ethernet (FCoE).
Configuration
Network Diagram
All of the diagrams in this document were created with App Diagrams.
Physical Network
This configuration is meant to be a simple reference. It utilizes single links even though the port channels mentioned can be easily extended with more links to provide more bandwidth.
Between the UCS FIs and the Nexus 5000s there is only a single link on each side. Between each Nexus 5000 and each NetApp there is also one link bound to associated VPCs.
Logical VSAN View
In this case, VSAN 70 is used for the left side of the fabric and VSAN 80 for the right side. Respectively, VLANs 970 and 980 are the FCoE VLANs.
Platform Configurations
The majority of the complexity of this setup is concentrated on the Nexus 5000 platform. The UCS and NetApp configuration is pretty straightforward.
Nexus 5000
This configuration has a lot of similarities to the Nexus 5000 NPIV FCoE with FCoE NPV Attached UCS Configuration Example.
Enable the Features
On both Nexus 5000s:
feature npiv
feature fcoe
feature lacp
Create the Virtual Storage Area Network (VSAN)
bdsol-n5548-05
vsan database
vsan 70
bdsol-n5548-06
vsan database
vsan 80
Add New VLANs to Carry FCoE Traffic
bdsol-n5548-05
vlan 970
fcoe vsan 70
bdsol-n5548-06
vlan 980
fcoe vsan 80
Physical Connectivity Configuration
This type of configuration allows you to have FCoE and another VLAN carried in a single trunk. In this case VPCs carry VLAN 3380, configured as Network File System (NFS) VLAN on the NetApp side as seen later.
The configuration is taken from bdsol-n5548-05.
Configure VPC to NetApp-B:
interface Ethernet1/5
switchport mode trunk
switchport trunk allowed vlan 970,3380
spanning-tree port type edge trunk
channel-group 105 mode active
interface port-channel105
switchport mode trunk
switchport trunk allowed vlan 970,3380
vpc 105
Configure VPC to NetApp-A:
interface Ethernet1/6
switchport mode trunk
switchport trunk allowed vlan 970,3380
spanning-tree port type edge trunk
channel-group 106 mode active
interface port-channel106
switchport mode trunk
switchport trunk allowed vlan 970,3380
vpc 106
Configure Port Channel that Leads to UCS:
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan 970
spanning-tree port type edge trunk
spanning-tree bpdufilter enable
channel-group 103 mode active
interface port-channel103
switchport mode trunk
switchport trunk allowed vlan 970
This configuration is reflected on the bdsol-n5548-06 switch. The only difference is that VLAN 980 is carried in trunks and not 970.
Create and Bind the VSAN Configuration
Now that physical connectivity is configured, the newly configured Virtual Fibre Channel (VFC) interfaces must be bound to the port channels created previously.
The bdsol-n5548-05 configuration to NetApp:
interface vfc105
bind interface port-channel105
switchport trunk allowed vsan 70
no shutdown
interface vfc106
bind interface port-channel106
switchport trunk allowed vsan 70
no shutdown
The bdsol-n5548-05 FCoE configuration to UCS:
interface vfc103
bind interface port-channel103
switchport trunk allowed vsan 70
no shutdown
As mentioned in the introduction to this document, this configuration depicts a single link in port channel between UCS and Nexus switches. It is used for reference only and can be extended to additional links.
The previous configuration is reflected on bdsol-n5548-06. The only change is to use VSAN 80 instead of VSAN 70.
Peer Link Port Channel
In this configuration each side of the fabric has only one FCoE VLAN. The VLANs cannot trunked between Nexus 5000s.
Dependent upon the configuration standards and best practices, this can be completed by:
- No additional changes - since VLANs are not created on both Nexus 5000s they cannot be trunked. Those VLANs are seen under "Vlans Err-disabled on Trunk".
- Remove VLANs from the trunk via the switchport trunk allowed vlan except command.
- Explicitly mention which VLANs are allowed -do not mention the FCoE VLANs.
Add the VFCs to the VSAN Database
The newly configured VFCs are added to VSAN database.
bdsol-n5548-05:
vsan database
vsan 70 interface vfc103
vsan 70 interface vfc105
vsan 70 interface vfc106
bdsol-n5548-06:
vsan database
vsan 80 interface vfc103
vsan 80 interface vfc105
vsan 80 interface vfc106
Quality of Service (QoS) Configuration
This configuration has a simple QoS profile, that is meant as a reference.
In order for Priority Flow Control (PFC) to be enabled, which allows loss-less transmission that is necessary for FCoE, the QoS configuration needs to match.
In this case only two classes are used. One for FC/FCoE and one for NFS traffic that allows jumbo frames.
bdsol-n5548-05# show running-config ipqos
class-map type qos class-fcoe
class-map type queuing class-fcoe
match qos-group 1
class-map type network-qos class-fcoe
match qos-group 1
policy-map type network-qos jumbo
class type network-qos class-fcoe
pause no-drop
mtu 2158
class type network-qos class-default
mtu 9216
multicast-optimize
system qos
service-policy type network-qos jumbo
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type qos input fcoe-default-in-policy
UCS Manager
A step-by-step guide that shows you how to set up FCoE Uplinks can be found in UCS FCoE Uplink Configuration Example.
As mentioned previously, in this example a single link is used only, but this configuration can be extended to multiple links in a port channel.
Port Channel Configuration
The previous example shows the UCS point of view. Each side of the fabric has a port channel, number 2, based on port ethernet1/21.
In this example the only VSANs which exist are 70 and 80, so it is safe to assume they are the ones trunked.
bdsol-6248-03-A(nxos)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
970 70 Operational
4048 1 Operational
interface Ethernet1/21
description UF: UnifiedUplink
pinning border
switchport mode trunk
switchport trunk native vlan 4049
switchport trunk allowed vlan 1,970,4048-4049
udld disable
channel-group 2 mode active
no shutdown
interface port-channel2
description UF: UnifiedUplink
switchport mode trunk
switchport trunk native vlan 4049
switchport trunk allowed vlan 1,970,4048-4049
pinning border
speed 10000
VLAN 970 is trunked via port channel 2.
QoS Configuration
As mentioned, in the QoS configuration on the Nexus 5000 an end-to-end QoS configuration allows PFC to work.
In this example, only two classes are enabled. The default class which carries jumbo frames (MTU 9216) and the FC class.
This matches what was configured on the Nexus side in previous steps. Other classes are present, but not enabled.
NetApp
In this example NetApp operates in 7-mode. A step-by-step guide on how to set up the NetApp side can be found in the NetApp Storage Deployment Guide.
License
At least the FCP (FC/FCoE) license must be present in order to enable the FCoE to work.
Configuration
The FCoE VLANs do not need to be added anywhere on the NetApp. They are discovered in the VLAN discovery stage of FCoE negotiation.
As mentioned previously, during Nexus 5000 configuration the NFS is handled by same Uplinks also.
bdsol-3220-01-A> rdfile /etc/rc
ifgrp create lacp dvif -b ip e1a e1b
vlan create dvif 3380
ifconfig e0M `hostname`-e0M flowcontrol full netmask 255.255.255.128 partner
e0M mtusize 1500
ifconfig dvif-3380 `hostname`-dvif-3380 netmask 255.255.255.0 partner dvif-3380
mtusize 9000 trusted
route add default 10.48.43.100 1
routed on
The configuration shows that LACP portchannel (ifgroup) is created from interface e1a and e1b and the virtual interface for VLAN 3380 is added.
The same configuration from the GUI shows:
Interface Group Configuration
bdsol-3220-01-A> ifgrp status
default: transmit 'IP Load balancing', Ifgrp Type 'multi_mode', fail 'log'
dvif: 2 links, transmit 'IP Load balancing', Ifgrp Type 'lacp' fail 'default'
Ifgrp Status Up Addr_set
up:
e1a: state up, since 28Sep2014 00:48:23 (142+11:23:01)
mediatype: auto-10g_twinax-fd-up
flags: enabled
active aggr, aggr port: e1b
input packets 766192514, input bytes 2560966346135
input lacp packets 410301, output lacp packets 410438
output packets 615632, output bytes 81875375
up indications 9, broken indications 5
drops (if) 0, drops (link) 1
indication: up at 28Sep2014 00:48:23
consecutive 0, transitions 14
e1b: state up, since 28Sep2014 00:48:22 (142+11:23:02)
mediatype: auto-10g_twinax-fd-up
flags: enabled
active aggr, aggr port: e1b
input packets 246965410, input bytes 702882508932
input lacp packets 410301, output lacp packets 410442
output packets 615646, output bytes 81876343
up indications 9, broken indications 5
drops (if) 0, drops (link) 2
indication: up at 28Sep2014 00:48:22
consecutive 0, transitions 15
Verification
Use this section to confirm that your configuration works properly. The Cisco CLI Analyzer supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.
The Verify and Troubleshoot sections of this document center around Nexus 5000 switches since they are central to this configuration.
Check the State of VFC Interfaces
bdsol-n5548-05# show int vfc 105
vfc105 is trunking
Bound interface is port-channel105
Hardware is Ethernet
Port WWN is 20:68:00:2a:6a:28:68:7f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 70
Trunk vsans (admin allowed and active) (70)
Trunk vsans (up) (70)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
65 frames input, 6904 bytes
0 discards, 0 errors
65 frames output, 9492 bytes
0 discards, 0 errors
The previous output shows that VSAN 70 is up and trunks on VFC and that it does indeed operate in the expected mode - TF.
Verify NetApp and UCS are Logged Into the Fabric
Check the flogi database on the switches.
bdsol-n5548-05# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc103 70 0xa00020 22:c8:54:7f:ee:f2:d6:bf 20:46:54:7f:ee:f2:d6:81
vfc105 70 0xa00000 50:0a:09:81:98:fd:66:a5 50:0a:09:80:88:fd:66:a5
[bdsol-3220-01-B-1a]
vfc106 70 0xa00001 50:0a:09:81:88:fd:66:a5 50:0a:09:80:88:fd:66:a5
[bdsol-3220-01-A-1a]
The previous output shows that each VFC has one device logged in. This is expected in a very basic setup.
In case there is doubt, you can check the status on the UCS side. In end host mode the NPV facility can be used.
bdsol-6248-03-A(nxos)# show npv status
npiv is enabled
disruptive load balancing is disabled
External Interfaces:
====================
Interface: vfc713, State: Trunking
VSAN: 1, State: Waiting For VSAN Up
VSAN: 70, State: Up, FCID: 0xa00020
VSAN1 is not up on the trunk nor configured, so it can be ignored. Here is verification of the VFC 713 on the UCS.
bdsol-6248-03-A(nxos)# show interface vfc 713
vfc713 is trunking (Not all VSANs UP on the trunk)
Bound interface is port-channel2
Hardware is Ethernet
Port WWN is 22:c8:54:7f:ee:f2:d6:bf
Admin port mode is NP, trunk mode is on
snmp link state traps are enabled
Port mode is TNP
Port vsan is 1
Trunk vsans (admin allowed and active) (1,70)
Trunk vsans (up) (70)
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1)
1 minute input rate 2523680 bits/sec, 315460 bytes/sec, 163 frames/sec
1 minute output rate 1114056 bits/sec, 139257 bytes/sec, 69 frames/sec
134861480 frames input, 159675768364 bytes
0 discards, 0 errors
405404675 frames output, 792038498124 bytes
0 discards, 0 errors
This output shows that VSAN 70 is up and that port operates in Node port in NPV mode (NP). This is expected since UCS operates in End Host host mode.
NetApp
As mentioned previously, this setup runs in 7-mode. The cluster mode command can be different.
The FCP status on interfaces is:
bdsol-3220-01-A> fcp topology show
Switches connected on adapter 1a:
Switch Name: bdsol-n5548-05
Switch Vendor: Cisco Systems, Inc.
Switch Release: 5.2(1)N1(1b)
Switch Domain: 160
Switch WWN: 20:46:00:2a:6a:28:68:41
Port Count: 5
(...)
Switches connected on adapter 1b:
Switch Name: bdsol-n5548-06
Switch Vendor: Cisco Systems, Inc.
Switch Release: 5.2(1)N1(1b)
Switch Domain: 35
Switch WWN: 20:50:00:2a:6a:28:6e:41
Port Count: 5
Troubleshoot
This section provides information you can use to troubleshoot your configuration. The Cisco CLI Analyzer supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.
Note: Only registered Cisco users can access internal tools and information.
Features Configured
Check to ensure that all required features are enabled.
bdsol-n5548-05# show feature | i npiv|fcoe|lacp
fcoe 1 enabled
fcoe-npv 1 disabled
lacp 1 enabled
npiv 1 enabled
VLAN and VSAN Configuration
Check the existence of the FCoE VLAN and where it is sent.
show vlan fcoe
show vlan id X
An example for the left side of the fabric is shown here:
bdsol-n5548-05# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
970 70 Operational
bdsol-n5548-05# show vlan id 970
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
970 VLAN0970 active Po1, Po103, Po105, Po106, Po107
Po202, Po203, Po204, Eth1/3
Eth1/5, Eth1/6, Eth1/7, Eth1/8
VLAN 970 exists and is trunked to the ports outlined in the configuration, as well as confirmation that it carries VSAN 70.
Spanning Tree and FCoE VLANs
FCoE VLANs are a special case and must not exist in the spanning tree.
bdsol-n5548-05# show spanning-tree vlan 970
Spanning tree instance(s) for vlan does not exist.
A special case exists for Multiple Spanning Tree (MST) and FCoE VLANs. An instance of MST needs to be created in order to carry FCoE VLANs, and not other VLANs. See MST Instances For Dual Fabric FCoE Deployments.
VPC and FCoE VLANs
In the previous example, FCoE VLANs are added on VPCs and VPCs are on two different devices. The configuration for trunking is slightly different for each VPC.
The FCoE VLANs cannot show up as allowed VLANs in the VPC consistency check.
bdsol-n5548-05# show vpc consistency-parameters vpc 105
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
lag-id 1 [(1, 2-a0-98-40-ba-fa, [(1, 2-a0-98-40-ba-fa,
1, 0, 0), (7f9b, 1, 0, 0), (7f9b,
0-23-4-ee-be-35, 8069, 0-23-4-ee-be-35, 8069,
0, 0)] 0, 0)]
mode 1 active active
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Admin port mode 1
Allowed VLANs - 3380 3380
Local suspended VLANs - - -
This is expected.
In order to verify that the FCoE VLANs are trunked, verify per-interface trunking.
bdsol-n5548-05# show interface port-channel 105 trunk
--------------------------------------------------------------------------------
Port Native Status Port
Vlan Channel
--------------------------------------------------------------------------------
Po105 1 trunking --
--------------------------------------------------------------------------------
Port Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Po105 970,3380
--------------------------------------------------------------------------------
Port STP Forwarding
--------------------------------------------------------------------------------
Po105 3380
As mentioned, FCoE VLANs do not participate in spanning tree and are not visible as an STP forwarding VLAN. However they can show up as trunking.
Priority Flow Control Status
Priority flow control is crucial for the operation of FCoE. In order to have the PFC operational, the QoS configuration must match end to end.
In order to verify the state on a per interface basis:
bdsol-n5548-05# show interface ethernet 1/3 priority-flow-control
============================================================
Port Mode Oper(VL bmap) RxPPP TxPPP
============================================================
Ethernet1/3 Auto On (8) 2852 13240218
The previous output shows that PFC is operational (On) and that Per Priority Pause (PPP) frames have been both received and transmitted.
For a device-wide view the administrator can enter the show int priority-flow-control command:
bdsol-n5548-05# show int priority-flow-control
============================================================
Port Mode Oper(VL bmap) RxPPP TxPPP
============================================================
Ethernet1/1 Auto Off 4 0
Ethernet1/2 Auto Off 4 0
Ethernet1/3 Auto On (8) 2852 13240218
Ethernet1/5 Auto On (8) 0 0
Ethernet1/6 Auto On (8) 0 0
(...)
The interfaces which were outlined in this document, Ethernet 1/3, 1/5 and 1/6, have PFC enabled and operational.
VSAN Membership
The associated configured VFCs need to be a part of the right VSAN. This example shows that the configured virtual interfaces are part of the correct VSAN.
bdsol-n5548-05# show vsan membership
vsan 1 interfaces:
fc2/15
vsan 70 interfaces:
fc2/16 vfc103 vfc105 vfc106
FCoE Initiation
FCoE Initiation Protocol goes through several steps in order to bring up the link between the end host and the switch.
What is important to remember is that to bring up the interface properly, a couple of different actions need to happen as depicted in this diagram.
Enode is an end node, while FCF is the switch.
In order to verify the steps from one of the devices, the fcoe_mgr (FCoE manager) facility can be used. In this case it is vfc 105.
bdsol-n5548-05# show platform software fcoe_mgr info interface vfc 105
vfc105(0x841e4c4), if_index: 0x1e000068, VFC RID vfc105
FSM current state: FCOE_MGR_VFC_ST_PHY_UP
PSS Runtime Config:-
Type: 3
Bound IF: Po105
FCF Priority: 128 (Global)
Disable FKA: 0
PSS Runtime Data:-
IOD: 0x00000000, WWN: 20:68:00:2a:6a:28:68:7f
Created at: Sat Sep 27 22:45:05 2014
FC Admin State: up
Oper State: up, Reason: down
Eth IF Index: Po105
Port Vsan: 70
Port Mode: F port
Config Vsan: 70
Oper Vsan: 70
Solicits on vsan: 70
Isolated Vsan:
FIP Capable ? : TRUE
UP using DCBX ? : FALSE
Peer MAC : 00:c0:dd:22:79:39
PSS VN Port data:-
FC ID 0xA00000 -
vfc index 503316584 vfc name vfc105
vsan id 70
enode_mac 00:c0:dd:22:79:39
vfc wwn 50:0a:09:81:98:fd:66:a5
The output is continued.
The output contains a few useful pieces of information. It verifies current status, shows the actual binding between VFC and portchannel and the actual state of the VFC on the VSAN. It shows transitions between different states outlined in the diagram before the interface comes up. VLAN discovery is performed and FIP solicitation is received and answered which allows the VFC to come up.
Continuation of the previous output:
17) FSM:<vfc105> Transition at 554094 usecs after Sat Sep 27 22:48:06 2014
Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
Triggered event: [FCOE_MGR_VFC_EV_BRING_UP]
Next state: [FSM_ST_NO_CHANGE]
18) FSM:<vfc105> Transition at 685820 usecs after Sat Sep 27 22:48:07 2014
Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
Next state: [FSM_ST_NO_CHANGE]
19) FSM:<vfc105> Transition at 686781 usecs after Sat Sep 27 22:48:07 2014
Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
Triggered event: [FCOE_MGR_VFC_EV_FIP_SOLICITATION]
Next state: [FSM_ST_NO_CHANGE]
Curr state: [FCOE_MGR_VFC_ST_PHY_UP]
Known and Common Problems
There a couple of known and common problems with the FCoE configuration that the Data Center Networking Solutions Technical Assistance Center team sees reoccur.
- Trunk protocol - The trunk protocol needs to be up.
bdsol-n5548-05# show trunk protocol
Trunk Protocol is enabled.
- NetApp 8.2.2 and Nexus 5500/5600 that run NX-OS 7.x software. At the time that the document was written, a known problem between these two versions exists and is under investigation by Cisco and NetApp. Versions 6.x and 5.x of NX-OS work fine with NetApp 8.2.2.