Document ID: 116075
Updated: Apr 29, 2013
Contributed by Dmitri Filenko and Andreas Nikas, Cisco TAC Engineers.
This document is intended to assist Unified Computing Systems (UCS) administrators who configure direct attached storage on the Cisco UCS platform.
There are no specific requirements for this document.
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Refer to Cisco Technical Tips Conventions for information on document conventions.
UCS appliance ports are used to directly connect a storage appliance to UCS fabric interconnects.
An appliance port behave similar to virtual ethernet (vEthernet or vEth) ports:
- It contains a list of allowed virtual LANs (VLANs).
- MAC addresses on these interfaces are learned by the fabric interconnect.
- It requires an uplink for pinning.
When a VLAN is created for an appliance port in the Appliances section of LAN tab, make sure to create the same VLAN in the LAN Cloud tab.
Why Appliance Port VLANs Should be Allowed on Uplinks
There are number of reasons why an upstream switch should allow storage appliance port traffic. These include:
- If storage must be accessed outside of the UCS domain.
- If storage and servers are located in different subnets.
- If storage is configured in Active/Passive mode and both fabric interconnects require communication to the same controller.
- In certain failover scenarios.
For an explanation of specific failover scenarios that require upstream switches refer to Cisco Unified Computing System (UCS) Storage Connectivity Options and Best Practices with NetApp Storage white paper and begin at Figure 8.
Definition of a Unified Storage Port
Traditional appliance ports for UCS Release 2.1(1a) and earlier can only pass Ethernet-based traffic, such as Network File System (NFS) and Internet Small Computer System Interface (iSCSI). Unified Storage ports introduced in UCS Manager Release 2.1(1a) can pass both Fibre Channel over Ethernet (FCoE) and Ethernet traffic over the same link. In order to use this feature, the storage controller must have a converged network adapter (CNA) capable of FCoE and traditional Ethernet on the same port.
For configuration of a Unified Storage port, refer to the UCS Manager Release 2.1(1) Configuration Guide.
In order to verify that the port is configured as a unified storage port, login to the Cisco NX-OS shell and verify the port's current configuration with this command:
ucs01-A(nxos)# show running-config interface eth 1/5
description AF: UnifiedStorage
Appliance Port Port-Channel
UCS supports Static and Link Aggregation Control Protocol (LACP) port channels for appliance port configuration. However, there is no virtual Port-Channel (vPC) support.
When to Use Trunk or Access Mode
In order to decide when to use Trunk or Access Mode depends on the storage appliance capabilities. If the storage appliance is can add VLAN tags, then it is advisable to configure appliance ports in Trunk mode and configure the VLAN tagging on the storage side for maximum flexibility. Multiple VLANs can be used on the same link in this case, which add flexibility to separate different protocols on the wire. If only one VLAN is required, or if the storage controller is not capable of VLAN tagging, then use of an access port is recommended.
Situations to Avoid
Never configure VLAN tagging both on the storage side and on the UCS side simultaneously. Double VLAN tagging breaks communication over this VLAN. For example, if a VLAN tag is added in the appliance port configuration on the UCS side (Access mode or Native VLAN in Trunk mode), then do not configure tagging for the same VLAN on the storage controller side.
Another example is a multi-protocol configuration on a storage controller. As a best practice, each protocol should be placed into a separate VLAN, and VLANs can be identified in the storage controller port configuration. Then on the UCS side, the appliance port must be set as a trunk, which allows the list of configured VLANs on storage controller side. Native VLAN is not required in this configuration.
Appliance Port Failover
Failover cannot be configured in the UCS side for appliance ports. By design, UCS fabric interconnects operate as two independent fabrics. Failover must be configured on the storage side and must be implemented with the correct network design while specific failover behavior for storage controllers are kept in mind, dependent on the storage controller model.
For more information on redundancy configuration on NetApp controllers, refer to this white paper.
With default behavior, appliance ports are shut down if the uplink that is pinned to it goes down.
Appliance Port Troubleshooting
In majority of implementations, communication between blade servers and storage controllers plugged into appliance ports are over same broadcast domain (Layer 2 of the Open Systems Interconnect (OSI) model).
The most powerful tool in troubleshooting appliance port communication is to view the MAC address table on the UCS fabric interconnects. In order to check the MAC address table, login to the UCS console, connect to the NX-OS shell and verify the list of allowed VLANs on the appliance port. Then, view the MAC address entries for the VLAN that are to be used for communication with the appliance port. Here are the commands and output:
F340-31-14-UCS-2-A# connect nxos a
F340-31-14-UCS-2-A(nxos)# show run int eth 1/11
!! Command: show running-config interface Ethernet1/11
!! Time: Fri Mar 29 07:02:29 2013
description A: Appliance
no pinning server sticky
pinning server pinning-failure link-down
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 170
F340-31-14-UCS-2-A(nxos)# show mac address-table vlan 170
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports
* 170 0025.b500.004f static 0 F F Veth780
* 170 0025.b500.005f static 0 F F Veth779
* 170 010a.84ff.e4fe dynamic 0 F F Eth1/11
In the this output, there are two blade servers, vEth780 and vEth779, and the storage controller's MAC address learned on Eth1/11. These devices should be able to communicate with each other if there are no other configuration issues on the end devices.
This is the most critical step when you verify successful communication between devices. If no MAC addresses are learned on the appliance port while the correct VLAN is specified, then return to the appliance port configuration and re-confirm the trunk configuration. Also, make sure that the communication link on the storage appliance is in Active mode in case of Active/Passive link configuration. You can also check the MAC address table on fabric interconnect B, dependent on what link is active on the storage controller side.
When the MAC address of the server and the MAC address of storage controller are learned on the fabric interconnect within the same VLAN, fabric interconnect switches traffic locally without the use of the upstream switches.
At this juncture, you can use Internet Control Message Protocol (ICMP) request (ping) to test communication between the end points.
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.