Configure Disjoint Layer 2 in UCS Manager

Available Languages

Download Options

PDF (2.1 MB)
View with Adobe Reader on a variety of devices
ePub (2.1 MB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (1.2 MB)
View on Kindle device or Kindle app on multiple devices

Updated:July 21, 2023

Document ID:220611

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Background Information

Disjoint Layer 2 (DL2)

Network Diagram

Configure

Create the uplinks in the Fabric Interconnects.

Introduction

This document describes the configuration for Disjoint Layer 2 in the Unified Computing System Manager Domain (UCSM).

Prerequisites

At least one available link on each Fabric Interconnect and two available links on your upstream switch
The links between the Fabric Interconnects and the upstream switch must be up, they must be configured as uplinks. If they are not, check this guide to configure them on UCSM: Using the LAN Uplinks Manager
The VLANs to be used must be created on UCSM already. If they are not, check this guide: Creating a Named VLAN with the LAN Uplinks Manager
The VLANs to be used must be created on the upstream switch already.
The VLANs to be used cannot exist on any other virtual Network Interface Cards (vNIC) on the Service Profiles.

Requirements

Cisco recommends that you have knowledge of these topics:

Unified Computing System Manager Domain (UCSM).
Basic understanding of Disjoint Layer 2 networks.
Networking configuration.
VNIC Configuration.

Components used

UCS Manager.
Firmware version 4.2(3e).
6454 Fabric Interconnect.
Cisco B200 M5.
Nexus 5672UP 16G-FC Chassis.
Catalyst WS-C3650-12X48UR-E.

Background Information

Disjoint Layer 2 (DL2)

Disjoint layer-2 networks are required if you have two or more Ethernet clouds that never connect, but must be accessed by servers or virtual machines located in the same Cisco UCS domain.

They are also required in a multi-tenant environment if servers or virtual machines for more than one client are located in the same Cisco UCS domain and they need to access the L2 networks for both clients.

Network Diagram

diagram

Configure

Log into Cisco UCS Manager GUI as an administrative user.

Create the uplinks in the Fabric Interconnects.

Step 1. Navigate to Equipment tab > Fabric interconnects > Fabric Interconnect A or B.

Step 2. Right-click on the desired port and select Configure as an uplink port. This uplink port is used to connect to the Disjoint network (in this example VLAN 80).

Ethernetuplink 1.1

Ethernetuplink 2.2

Assign VLANs

In order to recreate, VLAN 80 was created.

Step 1. In order to add the VLAN 80 navigate to LAN tab > LAN Cloud > VLANs click on add and complete the fields.

Create vlan80

Create vNIC Template

Step 1. Navigate to LAN tab > Policies > root > vNIC Template and click Add.

Select the vNIC template. Configure the name and select the appropriate Fabric ID.
As a best practice, it is recommended to have a redundant vNIC configuration.

Create VNICtemp

Select the VLANs to be configured accordingly. In this example, the base VLAN is VLAN 470, and the disjoint VLAN is VLAN 80.

Using VNIC temp Eth2

Using VNIC temp Eth00

Repeat the same steps for Fabric B.

Step 2. Navigate to LAN tab > Policies > root > LAN Connectivity Policies and create the new policy, click in Add, and create the vNICs.

Write the name for the vNIC, select the MAC pool, and mark the check box Use vNIC Template.

Select VNICTemp

Step 3. Use the vNIC Template previously configured, select the desired Adapter Policy, and click OK. Repeat the same procedure to Fabric Interconnect B.

Using VNIC temp Eth0

Using VNIC temp Eth2

LAN DL2

Step 4. Go back to the Service profile and select the LAN Connectivity Policy.

Reboot the Server

Step 1. To apply the previously configured changes, reboot the server.

Note: From vCenter, ensure the node is in maintenance mode.

Step 2. Once the server finishes boot up, verify if the vNIC is present. Navigate to Server > Service Profiles > root > Name of your Service Profile > Network tab.

VNICS

Create VLAN Group

Step 1. Navigate to LAN > LAN Cloud > VLAN Groups > Create VLAN Groups.

Step 2. Select the name for the VLAN Group and choose the required VLAN, finally, add single Uplinks in step 2.

VLANGroup name

Step 3. Optionally move to step 3 in the VLAN group steps if you need to add Port Channels.

VLANGroup uplinks

ESXi Configuration

Step 1. Log in to the ESXi host and navigate to the Networking tab > Virtual Switches and click Add standard virtual Switch, name the virtual switch, and select the uplink.

Virtualswitch

Step 2. Navigate to Networking > Port Group > Add Port Group. Name your port group, select the desired VLAN, and use the virtual switch previously configured.

PORTGROUP

Step 3. Navigate to Networking, select the vSwitch previously configured, and click on add uplink. In order to have redundancy add a new uplink that includes the VLAN used for the Disjoint Layer 2.

In this case, VLAN 80 was allowed in VNIC Eth2 (Fabric Interconnect A) and VNIC Eth3 (Fabric Interconnect B).

Verify in UCSM

Verify the VLAN in the CLI

Open an SSH session to the Fabric Interconnects and run the command.

FI-A(nx-os)# show vlan brief

This command displays the information for the VLANs created and you can confirm the VLAN created for the Disjoint Layer 2.

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Po1, Eth1/5, Eth1/6, Eth1/8
                                                Eth1/9, Eth1/10, Eth1/11
                                                Eth1/12, Eth1/13, Eth1/16
                                                Eth1/17, Eth1/18, Eth1/19
                                                Eth1/20, Eth1/21, Eth1/22
                                                Eth1/23, Eth1/24, Eth1/26
                                                Eth1/27, Eth1/28, Eth1/29
                                                Eth1/30, Eth1/31, Eth1/32
                                                Eth1/33, Eth1/34, Eth1/35
                                                Eth1/36, Eth1/37, Eth1/38
                                                Eth1/39, Eth1/40, Eth1/41
                                                Eth1/42, Eth1/43, Eth1/44
                                                Eth1/45, Eth1/46, Eth1/47
                                                Eth1/48, Eth1/49, Eth1/50
                                                Veth876, Veth877, Veth1084
                                                Veth1119, Veth1120, Veth1122
                                                Eth1/1/10, Eth1/1/12, Eth1/1/13
                                                Eth1/1/15, Eth1/1/18, Eth1/1/20
                                                Eth1/1/22, Eth1/1/24, Eth1/1/26
                                                Eth1/1/28, Eth1/1/29, Eth1/1/30
                                                Eth1/1/31, Eth1/1/32
80   VLAN0080                         active    Eth1/47
470  VLAN0470                         active    Po1, Eth1/5, Eth1/6, Eth1/32
                                                Eth1/45, Eth1/46, Eth1/48
                                                Veth1084, Veth1090, Veth1092
                                                Veth1094, Veth1108, Veth1119
                                                Veth1120, Veth1122, Veth1131
                                                Veth1133

Verify the Virtual Interface (VIF) Path

Navigate to the SSH session and use the command:

FI-A# show service-profile circuit <server number>
Server: 1/6
    Fabric ID: A
        Path ID: 1
        VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport
        ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
              1131 Eth0            Up          Active     No Protection Unprotected 0/0/0      0/0/1      Ether
              1133 Eth2            Up          Active     No Protection Unprotected 0/0/0      1/0/47     Ether
              1135 fc0             Up          Active     No Protection Unprotected 0/0/0      1/0/3      Fc
              9327                 Up          Active     No Protection Unprotected 0/0/0      0/0/0      Ether
    Fabric ID: B
        Path ID: 1
        VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport
        ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
              1132 Eth1            Up          Active     No Protection Unprotected 0/0/0      0/0/2      Ether
              1134 Eth3            Up          Active     No Protection Unprotected 0/0/0      1/0/47     Ether
              1136 fc1             Up          Active     No Protection Unprotected 0/0/0      1/0/3      Fc
              9328                 Up          Active     No Protection Unprotected 0/0/0      0/0/0      Ether

This command displays the VIF Paths, the interface that is pinned, and the corresponding vNICs.

In the output obtained, it can be seen that the corresponding VIF is VIF 1134 which corresponds to the vNIC Eth3 and is pinned to interface 1/0/47 in Fabric Interconnect B.

Also, VIF 1133 corresponds to vNIC Eth2 and is pinned to 1/0/47 in Fabric Interconnect A.

Verify the pinning border interfaces.

Run the command to verify the pinning to the Uplink Ports.

UCS-AS-MXC-P25-02-A(nx-os)# show pinning border-interfaces 

--------------------+---------+----------------------------------------
Border Interface     Status    SIFs                                    
--------------------+---------+----------------------------------------
Po1                  Active    Veth1084 Veth1090 Veth1092 Veth1094 
                               Veth1108 Veth1119 Veth1120 Veth1131 
Eth1/32              Down                                        
Eth1/45              Down                                        
Eth1/46              Down                                        
Eth1/47              Active    sup-eth1 Veth1133                 
Eth1/48              Down                                        
Eth1/51              Down                                        
Eth1/52              Down                                        
Eth1/53              Down                                        
Eth1/54              Down

Verify the Designated Receiver

Run this command to verify the port that receives the multicast traffic for the VLAN.

FI-A(nx-os)# show platform software enm internal info vlandb id <VLAN-ID>
vlan_id 80
-------------
Designated receiver: Eth1/47
Membership:
Eth1/47

This output shows the correct uplink.

Verify the upstream Switch

Open an SSH session to the upstream switch and run the command.

NEXUS-01# show vlan brief
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Po1, Po2, Po4, Po5, Po6, Po7
                                                Po8, Po9, Po50, Po100, Eth1/1
                                                Eth1/2, Eth1/3, Eth1/4, Eth1/5
                                                Eth1/6, Eth1/8, Eth1/9, Eth1/10
                                                Eth1/12, Eth1/13, Eth1/14
                                                Eth1/15, Eth1/18, Eth1/19
                                                Eth1/20, Eth1/21, Eth1/22
                                                Eth1/23, Eth1/24, Eth2/1, Eth2/2
                                                Eth2/3, Eth2/4, Eth2/5, Eth2/6
                                                Eth2/7, Eth2/8, Eth2/10, Eth2/11
                                                Eth2/12, Eth2/13, Eth2/14
                                                Eth2/15, Eth2/16, Eth2/17
                                                Eth2/18, Eth2/19, Eth2/20
                                                Eth2/21, Eth2/22, Eth2/23
                                                Eth3/1, Eth3/2, Eth3/3, Eth3/4
                                                Eth3/5, Eth3/6
                                                Eth2/18
80   DL2                              active    Po1, Po2, Po6, Po7, Po8, Po9
                                                Po50, Po100, Eth1/1, Eth1/3
                                                Eth1/4, Eth1/5, Eth1/6, Eth1/17
                                                Eth1/19, Eth1/20, Eth1/21
                                                Eth1/22, Eth1/23, Eth1/24
                                                Eth2/1, Eth2/2, Eth2/3, Eth2/4
                                                Eth2/5, Eth2/17, Eth2/18
470  VLAN_470                         active    Po1, Po2, Po3, Po4, Po5, Po6
                                                Po7, Po8, Po9, Po50, Po100
                                                Eth1/1, Eth1/3, Eth1/4, Eth1/5
                                                Eth1/6, Eth1/7, Eth1/9, Eth1/10
                                                Eth1/16, Eth1/19, Eth1/20
                                                Eth1/21, Eth1/22, Eth1/23
                                                Eth1/24, Eth2/1, Eth2/2, Eth2/3
                                                Eth2/4, Eth2/5, Eth2/9, Eth2/17
                                                Eth2/18, Eth2/24

This output shows the port that is associated with VLAN 80. In this case, the desired port is the Ethernet 1/17 which is associated with the uplink 1/47.

On the other hand, you can verify the MAC address table to check what Virtual Machine (VM) shows.

NEXUS-01(config)# show mac address-table vlan 80 
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 80       000c.2937.2cc7    dynamic   150        F    F  Eth1/17

Verify the connection between the VLAN network in the ESXi

Open the Terminal in the Virtual Machine and ping to the default gateway of the VLAN network; you then see a successful ping.