Contents

Introduction

Prerequisites

Requirements

Components used

Background Information

Disjoint Layer 2 (DL2)

Network Diagram

Configure

Create the uplinks in the Fabric Interconnects.

Assign VLANs

Create vNIC Template

Reboot the Server

Create VLAN Group

ESXi Configuration

Verify in UCSM

Related Information

Introduction

This document describes the configuration for Disjoint Layer 2 in the Unified Computing System Manager Domain (UCSM).

Prerequisites

- At least one available link on each Fabric Interconnect and two available links on your upstream switch

- The links between the Fabric Interconnects and the upstream switch must be up, they must be configured as uplinks. If they are not, check this guide to configure them on UCSM: Using the LAN Uplinks Manager

- The VLANs to be used must be created on UCSM already. If they are not, check this guide: Creating a Named VLAN with the LAN Uplinks Manager

- The VLANs to be used must be created on the upstream switch already.

- The VLANs to be used cannot exist on any other virtual Network Interface Cards (vNIC) on the Service Profiles.

Requirements

Cisco recommends that you have knowledge of these topics:

- Unified Computing System Manager Domain (UCSM).

- Basic understanding of Disjoint Layer 2 networks.

- Networking configuration.

- VNIC Configuration.

Components used

- UCS Manager.

- Firmware version 4.2(3e).

- 6454 Fabric Interconnect.

- Cisco B200 M5.

- Nexus 5672UP 16G-FC Chassis.

- Catalyst WS-C3650-12X48UR-E.

Background Information

Disjoint Layer 2 (DL2)

Disjoint layer-2 networks are required if you have two or more Ethernet clouds that never connect, but must be accessed by servers or virtual machines located in the same Cisco UCS domain.

They are also required in a multi-tenant environment if servers or virtual machines for more than one client are located in the same Cisco UCS domain and they need to access the L2 networks for both clients.

Network Diagram

Configure

Log into Cisco UCS Manager GUI as an administrative user.

Create the uplinks in the Fabric Interconnects.

Step 1. Navigate to Equipment tab > Fabric interconnects > Fabric Interconnect A or B.

Step 2. Right-click on the desired port and select Configure as an uplink port. This uplink port is used to connect to the Disjoint network (in this example VLAN 80).

Assign VLANs

In order to recreate, VLAN 80 was created.

Step 1.  In order to add the VLAN 80 navigate to LAN tab > LAN Cloud > VLANs click on add and complete the fields.

Create vNIC Template

Step 1. Navigate to LAN tab > Policies > root > vNIC Template and click Add.

Select the vNIC template. Configure the name and select the appropriate Fabric ID.

As a best practice, it is recommended to have a redundant vNIC configuration.

Select the VLANs to be configured accordingly. In this example, the base VLAN is VLAN 470, and the disjoint VLAN is VLAN 80.

Repeat the same steps for Fabric B.

Step 2. Navigate to LAN tab > Policies > root > LAN Connectivity Policies and create the new policy, click in Add, and create the vNICs.

Write the name for the vNIC, select the MAC pool, and mark the check box Use vNIC Template.

Step 3. Use the vNIC Template previously configured, select the desired Adapter Policy, and click OK. Repeat the same procedure to Fabric Interconnect B.

Step 4. Go back to the Service profile and select the LAN Connectivity Policy.

Reboot the Server

Step 1. To apply the previously configured changes, reboot the server.

Note: From vCenter, ensure the node is in maintenance mode.

Step 2. Once the server finishes boot up, verify if the vNIC is present. Navigate to Server > Service Profiles > root > Name of your Service Profile > Network tab.

Create VLAN Group

Step 1. Navigate to LAN > LAN Cloud > VLAN Groups > Create VLAN Groups.

Step 2. Select the name for the VLAN Group and choose the required VLAN, finally, add single Uplinks in step 2.

Step 3. Optionally move to step 3 in the VLAN group steps if you need to add Port Channels.

ESXi Configuration

Step 1. Log in to the ESXi host and navigate to the Networking tab > Virtual Switches and click Add standard virtual Switch, name the virtual switch, and select the uplink.

Step 2. Navigate to Networking > Port Group > Add Port Group. Name your port group, select the desired VLAN, and use the virtual switch previously configured.

Step 3. Navigate to Networking, select the vSwitch previously configured, and click on add uplink. In order to have redundancy add a new uplink that includes the VLAN used for the Disjoint Layer 2.

In this case, VLAN 80 was allowed in VNIC Eth2 (Fabric Interconnect A) and VNIC Eth3 (Fabric Interconnect B).

Verify in UCSM

Verify the VLAN in the CLI

Open an SSH session to the Fabric Interconnects and run the command.

FI-A(nx-os)# show vlan brief

This command displays the information for the VLANs created and you can confirm the VLAN created for the Disjoint Layer 2.

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Po1, Eth1/5, Eth1/6, Eth1/8

Eth1/9, Eth1/10, Eth1/11

Eth1/12, Eth1/13, Eth1/16

Eth1/17, Eth1/18, Eth1/19

Eth1/20, Eth1/21, Eth1/22

Eth1/23, Eth1/24, Eth1/26

Eth1/27, Eth1/28, Eth1/29

Eth1/30, Eth1/31, Eth1/32

Eth1/33, Eth1/34, Eth1/35

Eth1/36, Eth1/37, Eth1/38

Eth1/39, Eth1/40, Eth1/41

Eth1/42, Eth1/43, Eth1/44

Eth1/45, Eth1/46, Eth1/47

Eth1/48, Eth1/49, Eth1/50

Veth876, Veth877, Veth1084

Veth1119, Veth1120, Veth1122

Eth1/1/10, Eth1/1/12, Eth1/1/13

Eth1/1/15, Eth1/1/18, Eth1/1/20

Eth1/1/22, Eth1/1/24, Eth1/1/26

Eth1/1/28, Eth1/1/29, Eth1/1/30

Eth1/1/31, Eth1/1/32

80   VLAN0080                         active    Eth1/47

470  VLAN0470                         active    Po1, Eth1/5, Eth1/6, Eth1/32

Eth1/45, Eth1/46, Eth1/48

Veth1084, Veth1090, Veth1092

Veth1094, Veth1108, Veth1119

Veth1120, Veth1122, Veth1131

Veth1133

Verify the Virtual Interface (VIF) Path

Navigate to the SSH session and use the command:

FI-A# show service-profile circuit <server number>

Server: 1/6

Fabric ID: A

Path ID: 1

VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport

---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------

1131 Eth0            Up          Active     No Protection Unprotected 0/0/0      0/0/1      Ether

1133 Eth2            Up          Active     No Protection Unprotected 0/0/0      1/0/47     Ether

1135 fc0             Up          Active     No Protection Unprotected 0/0/0      1/0/3      Fc

9327                 Up          Active     No Protection Unprotected 0/0/0      0/0/0      Ether

Fabric ID: B

Path ID: 1

VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport

---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------

1132 Eth1            Up          Active     No Protection Unprotected 0/0/0      0/0/2      Ether

1134 Eth3            Up          Active     No Protection Unprotected 0/0/0      1/0/47     Ether

1136 fc1             Up          Active     No Protection Unprotected 0/0/0      1/0/3      Fc

9328                 Up          Active     No Protection Unprotected 0/0/0      0/0/0      Ether

This command displays the VIF Paths, the interface that is pinned, and the corresponding vNICs.

In the output obtained, it can be seen that the corresponding VIF is VIF 1134 which corresponds to the vNIC Eth3 and is pinned to interface 1/0/47 in Fabric Interconnect B.

Also, VIF 1133 corresponds to vNIC Eth2 and is pinned to 1/0/47 in Fabric Interconnect A.

Verify the pinning border interfaces.

Run the command to verify the pinning to the Uplink Ports.

UCS-AS-MXC-P25-02-A(nx-os)# show pinning border-interfaces

--------------------+---------+----------------------------------------

Border Interface     Status    SIFs

--------------------+---------+----------------------------------------

Po1                  Active    Veth1084 Veth1090 Veth1092 Veth1094

Veth1108 Veth1119 Veth1120 Veth1131

Eth1/32              Down

Eth1/45              Down

Eth1/46              Down

Eth1/47              Active    sup-eth1 Veth1133

Eth1/48              Down

Eth1/51              Down

Eth1/52              Down

Eth1/53              Down

Eth1/54              Down

Verify the Designated Receiver

Run this command to verify the port that receives the multicast traffic for the VLAN.

FI-A(nx-os)# show platform software enm internal info vlandb id <VLAN-ID>

vlan_id 80

-------------

Designated receiver: Eth1/47

Membership:

Eth1/47

This output shows the correct uplink.

Verify the upstream Switch

Open an SSH session to the upstream switch and run the command.

NEXUS-01# show vlan brief

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Po1, Po2, Po4, Po5, Po6, Po7

Po8, Po9, Po50, Po100, Eth1/1

Eth1/2, Eth1/3, Eth1/4, Eth1/5

Eth1/6, Eth1/8, Eth1/9, Eth1/10

Eth1/12, Eth1/13, Eth1/14

Eth1/15, Eth1/18, Eth1/19

Eth1/20, Eth1/21, Eth1/22

Eth1/23, Eth1/24, Eth2/1, Eth2/2

Eth2/3, Eth2/4, Eth2/5, Eth2/6

Eth2/7, Eth2/8, Eth2/10, Eth2/11

Eth2/12, Eth2/13, Eth2/14

Eth2/15, Eth2/16, Eth2/17

Eth2/18, Eth2/19, Eth2/20

Eth2/21, Eth2/22, Eth2/23

Eth3/1, Eth3/2, Eth3/3, Eth3/4

Eth3/5, Eth3/6

Eth2/18

80   DL2                              active    Po1, Po2, Po6, Po7, Po8, Po9

Po50, Po100, Eth1/1, Eth1/3

Eth1/4, Eth1/5, Eth1/6, Eth1/17

Eth1/19, Eth1/20, Eth1/21

Eth1/22, Eth1/23, Eth1/24

Eth2/1, Eth2/2, Eth2/3, Eth2/4

Eth2/5, Eth2/17, Eth2/18

470  VLAN_470                         active    Po1, Po2, Po3, Po4, Po5, Po6

Po7, Po8, Po9, Po50, Po100

Eth1/1, Eth1/3, Eth1/4, Eth1/5

Eth1/6, Eth1/7, Eth1/9, Eth1/10

Eth1/16, Eth1/19, Eth1/20

Eth1/21, Eth1/22, Eth1/23

Eth1/24, Eth2/1, Eth2/2, Eth2/3

Eth2/4, Eth2/5, Eth2/9, Eth2/17

Eth2/18, Eth2/24

This output shows the port that is associated with VLAN 80. In this case, the desired port is the Ethernet 1/17 which is associated with the uplink 1/47.

On the other hand, you can verify the MAC address table to check what Virtual Machine (VM) shows.

NEXUS-01(config)# show mac address-table vlan 80

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 80       000c.2937.2cc7    dynamic   150        F    F  Eth1/17

Verify the connection between the VLAN network in the ESXi

Open the Terminal in the Virtual Machine and ping to the default gateway of the VLAN network; you then see a successful ping.

Related Information

Technical Support & Documentation - Cisco Systems

Disjoint Layer 2

Cisco UCS Manager Network Management Guide, Release 4.0

Using the LAN Uplinks Manager