FCoE Port Configuration Examples
This appendix describes port configuration examples relating to FCoE topologies and it includes the following sections:
VE Port Configuration Example
This section provides a sample configuration of the Cisco Nexus 5000 Series switch FCoE VE Port implementation. The configuration covers the switches in switch mode. FCoE initiators are used in this lab. You can attach either FC F Port storage directly to a Nexus 5000 Series switch FC GEMs, or use an FCoE target.
Note This example can be used for configuring VE ports between two Cisco Nexus 5000 Series switches in both fabrics. It does not include server configurations.
FCoE VE Port Topology Example
Figure 3-1 shows the topology that was used for the configuration example. The following configuration parameters are used in this topology:
- FCoE VLAN for Fabric A: 10
- FCoE VSAN for Fabric A: 10
- FCoE VLAN for Fabric B: 20
- FCoE VSAN for Fabric B: 20
- Ethernet Only VLAN across both fabrics: 200
You should choose these values before the time of configuration.
Figure 3-1 FCoE VE Port Topology
Note The FCoE VLAN/VSAN numbering does not have to be the same within the fabric. As a best practice, use different FCoE VLANs and VSAN numbers between the two fabrics to avoid confusion. Configurations have often been set up to assign ODD VLAN/VSANs for one fabric and EVEN VLANs/VSANs for the other fabric. This is just one example of keeping the numbers separate between the two fabrics
Enabling FCoE and Verifying QoS Configuration
Step 1 Enable FCoE.
switch# configure terminal
switch(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Step 2 (Optional) If you do not want to use the default Quality of Service (QoS) settings, specify your own policies:
Note Note: if you use custom policies, class-fcoe must be included in your QoS policies.
switch(config) system qos
switch(config-sys-qos)# service-policy type qos input fcoe-customized-in-policy-name
switch(config-sys-qos)# service-policy type queuing input fcoe-customized-in-policy-name
switch(config-sys-qos)# service-policy type queuing output fcoe-customized-out-policy-name
switch(config-sys-qos)# service-policy type network-qos fcoe-customized-nq-policy-name
Step 3 Verify that the FCoE policy maps can be found in the running configuration:
Note Note: If you specified customized QoS policy map names in Step 2, make sure you replace the default map names with your customized map names.
switch(config-sys-qos)# show policy-map system
Type network-qos policy-maps
===============================
policy-map type network-qos system
class type network-qos class-fcoe
class type network-qos class-default
Service-policy (qos) input: system
policy statistics status: disabled
Class-map (qos): class-fcoe (match-any)
Class-map (qos): class-default (match-any)
Service-policy (queuing) input: default-in-policy
policy statistics status: disabled
Class-map (queuing): class-fcoe (match-any)
Class-map (queuing): class-default (match-any)
Service-policy (queuing) output: default-out-policy
policy statistics status: disabled
Class-map (queuing): class-fcoe (match-any)
Class-map (queuing): class-default (match-any)
Quality of Service configuration on the Nexus 5000 series consists of three main constructs:
- Class-map and policy-map type qos: for classification purposes
- Class-map and policy-map type network: for network properties such as drop and no drop, queue size
- Class-map and policy-map type queueing: for bandwidth allocation
This exercise consists of changing the bandwidth allocation and the COS settings for FCoE.
Without proper configuration of class-fcoe in QoS, the following problems may occur:
- vFC interfaces do not come up (CNAs require advertisement of DCB parameters for FCoE)
- Drops noticed for I/Os
Note QoS has the following guidelines:
- A classification policy-map only applies in input
- A network policy-map applies globally (system)
- A queueing policy-map normally is meaningful in output, but since the exercise uses it to control the bandwidth allocation from CNA to the Cisco Nexus 5000 Series switch, in this case it is applied in input
Beginning in Cisco NX-OS Release 5.0(2)N1(1), you can modify the buffer allocation for no-drop classes:
switch(config-pmap-nq)# policy-map type network-qos nqos_policy
switch(config-pmap-nq)# class type network-qos nqos_class
switch(config-pmap-nq-c)# pause no-drop buffer-size <size> pause-threshold <threshold> resume-threshold <threshold>
Step 4 Verify the FCoE system class is active:
switch(config-sys-qos)# show queuing interface ethernet 1/1
Ethernet1/1 queuing information:
qos-group sched-type oper-bandwidth
q-size: 370240, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 2314
Pkts received over the port : 0
Ucastpkts sent to the cross-bar : 0
Mcastpkts sent to the cross-bar : 0
Ucastpkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
q-size: 79360, HW MTU: 2158 (2158 configured)
drop-type: no-drop, xon: 128, xoff: 252
Pkts received over the port : 0
Ucastpkts sent to the cross-bar : 0
Mcastpkts sent to the cross-bar : 0
Ucastpkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
Total Multicast crossbar statistics:
Mcastpkts received from the cross-bar : 0
Step 5 Repeat Step 1 through Step 4 on both upstream Cisco Nexus 5000 Series switches (CORE_N5k-1 and CORE_N5k-2 in this example).
Configuring VE Ports
FCoE VLAN and VSAN numbering in this example is as follows:
- Fabric A uses FCoE VLAN 10 and VSAN 10
- Fabric B uses FCoE VLAN 20 and VSAN 20
Note There are two switches in Fabric A and two switches in Fabric B. The FCoE VLAN/VSANs must match between the switches in the same fabric in order to bring up the VE port between them.
Step 1 Configure the VSAN on the Nexus 5000 Series switch for Fabric A:
switch(config)# vsan database
switch(config-vsan-db)# vsan 10
Step 2 Configure the FCoE VLAN to VSAN mapping and verify that it is up and operational for Fabric A:
switch(config-vlan)# fcoe vsan 10
switch(config-vlan)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
Step 3 Repeat Step 1 and Step 2 on the upstream Nexus 5000 Series switch in Fabric A.
Step 4 Configure the VSAN on the Nexus 5000 for Fabric B:
switch(config)# vsan database
switch(config-vsan-db)# vsan 20
Step 5 Configure the FCoE VLAN to VSAN mapping and verify that it is up and operational for Fabric B
switch(config-vlan)# fcoe vsan 20
switch(config-vlan)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
Step 6 Repeated Step 1 and Step 2 on the upstream Nexus 5000 Series switch in Fabric B.
Step 7 Configure the underlying 10-Gigabit Ethernet port that the vFC interface will be bound to. The VE port will use this interface as the physical transport for FCoE traffic between the two switches. This interface needs to be configured to trunk the appropriate FCoE VLAN as well as the Ethernet VLAN (in this example, we are using VLAN 200 to carry Ethernet traffic).
The 10-Gigabit Ethernet interfaces connecting the switches in this lab are shown in the topology above:
- Fabric A uses FCoE VLAN 10 and VSAN 10
- Fabric B uses FCoE VLAN 20 and VSAN 20
- PODX-N5K-1 (Fabric A) uses Ethernet 1/15 to connect to CORE N5K1
- PODX-N5K-2 (Fabric B) uses Ethernet 1/16 to connect to CORE N5K2
Configuration for both switches in Fabric A:
switch(config)# interface ethernet 1/15
switch(config-if)# switchport mode trunk
switch(config-if)# switchport trunk allowed vlan 10, 200
Configuration for both switches in Fabric A:
switch(config)# interface ethernet 1/16
switch(config-if)# switchport mode trunk
switch(config-if)# switchport trunk allowed vlan 20, 200
Step 8 Configure the vFC interface on the switch that will be bound to the VE port and add this vFC interface to VSAN 44 in the VSAN database:
The vFC numbers for the VE ports are as follows:
- Fabric A uses FCoE VLAN 10 and VSAN 10
- Fabric B uses FCoE VLAN 20 and VSAN 20
- POD1-N5K-1 (Fabric A) uses Ethernet 1/15 to connect to CORE N5K1
- POD1-N5K-2 (Fabric B) uses Ethernet 1/16 to connect to CORE N5K2
- POD1-N5K-1 (Fabric A) uses vfc 15 and binds it to Ethernet 1/15
- POD1-N5K-2 (Fabric B) uses vfc 16 and binds it to Ethernet 1/16
Configuration for both switches in Fabric A:
switch(config)# int vfc 15
switch(config-if)# switchport mode e
switch(config-if)# switchport trunk allowed vsan 10
switch(config-if)# bind interface eth 1/15
switch(config-if)# no shutdown
switch(config)# vsan database
switch(config-vsan-db)# vsan 10 interface vfc 15
switch(config-vsan-db)# show vsan membership
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
switch(config-vsan-db)# exit
Configuration for both switches in Fabric B:
switch(config)# int vfc 16
switch(config-if)# switchport mode e
switch(config-if)# switchport trunk allowed vsan 20
switch(config-if)# bind interface eth 1/16
switch(config-if)# no shutdown
switch(config)# vsan database
switch(config-vsan-db)# vsan 20 interface vfc 16
switch(config-vsan-db)# show vsan membership
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
switch(config-vsan-db)# exit
Note Don’t forget that these interface configurations must be configured on both sides of the ISL connecting the two switches in the same fabric.
Step 9 Verify that the vFC is up and operational. By default, the vFC will show as trunking. Make sure that it is bound to the correct physical interface and that VSAN 44 shows as allowed and active as well as up on the vFC interface.
Verify both switches in Fabric A:
switch(config)# show int vfc 15
Bound interface is Ethernet1/15
Hardware is Virtual Fibre Channel
Port WWN is 20:0e:00:0d:ec:b4:43:7f
Peer port WWN is 00:00:00:00:00:00:00:00
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Trunk vsans (admin allowed and active) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
13 frames input, 1028 bytes
13 frames output, 1180 bytes
last clearing of "show interface" counters never
Interface last changed at Sat Nov 6 17:58:39 2010
Verify both switches in Fabric A:
switch(config)# show int vfc 16
Bound interface is Ethernet1/16
Hardware is Virtual Fibre Channel
Port WWN is 20:0e:00:0d:ec:b4:43:7d
Peer port WWN is 00:00:00:00:00:00:00:00
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Trunk vsans (admin allowed and active) (20)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
13 frames input, 1028 bytes
13 frames output, 1180 bytes
last clearing of "show interface" counters never
Interface last changed at Sat Nov 6 17:58:39 2010