Prerequisites
The following sections detail the prerequisites for installing VTS 1.5:
- Hardware Requirement
- Software Requirements
- Setting up the VTC VM
- Day Zero Underlay Configuration - XRv as RR
- Day Zero Underlay Configuration - External RR (Spine as RR)
Hardware Requirement
The following are the hardware requirements:
|
ToR |
Cisco Nexus 93xxTX Switch |
|
Spine |
Cisco Nexus 93xxPX Switch; Cisco Nexus 95xxPX Switch |
|
Controller / Compute host |
Cisco UCS C220 Server; Cisco UCS C240 Server |
|
Border leaf |
Cisco Nexus 93xxTX Switch |
|
Data Center Interconnect |
Cisco ASR 9000 Series Aggregation Services Routers; Cisco Nexus 7000 Series Switches |
Software Requirements
VTS 1.5 has been qualified for OpenStack as well for VMware vCenter VMM. The following tables provide information about the software requirements on OpenStack and VMware environments:
|
|
|
|---|---|
|
OpenStack release |
OpenStack Ice house. Cisco VTS has been tested using RedHat OpenStack Platform 5 (RHEL OSP5), which is based on OpenStack Icehouse release. |
|
RedHat Requirement for VTC controller host |
VTS has been qualified using RHEL 7.0. |
|
|
|
|---|---|
|
Hypervisor requirements |
All nodes must be running VMware ESXi 5.5. |
|
VMware vCenter server requirements |
VMware vCenter 5.5 server Enterprise Plus must be installed on Windows 2008 x 64 server. |
Setting up the VTC VM
Given is a sample of the bridge configuration on the controller in a Linux host OS environment:
Bridge br-ex
Port "vnet0"
Interface "vnet0"
Port "vnet1"
Interface "vnet1"
Port br-ex
Interface br-ex
type: internal
Port "eth0"
Interface "eth0"
Day Zero Underlay Configuration - XRv as RR
The following table provides details about day zero configuration required on XRv, ToR1 & ToR2 (leafs), DC GW – ToR3 and DCI with reference to the image below.
|
Configuration Items |
Example Configuration (Includes Descriptions) |
| On XRv1 |
hostname XRv1 telnet vrf VTS_VIP ipv4 server max-servers 10 username admin password cisco123 group root-system group cisco-support vrf VTS_VIP address-family ipv4 unicast ! interface Loopback0 ipv4 address 1.1.1.1 255.255.255.255 interface MgmtEth0/0/CPU0/0 vrf VTS_VIP ipv4 address 209.165.201.1 255.255.255.254 interface GigabitEthernet0/0/0/0 Description: Interface to peer node spine eth1/1 ipv4 address 209.165.202.129 255.255.255.254 router static vrf VTS_VIP address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router ospf 100 router-id 1.1.1.1 address-family ipv4 unicast area 0 interface Loopback0 interface GigabitEthernet0/0/0/0 router bgp 100 |
|
On XRv2 |
hostname XRv2 telnet vrf VTS_VIP ipv4 server max-servers 10 username admin password cisco123 group root-system group cisco-support vrf VTS_VIP address-family ipv4 unicast ! interface Loopback0 ipv4 address 10.10.10.10 255.255.255.255 interface MgmtEth0/0/CPU0/0 vrf VTS_VIP ipv4 address 209.165.201.1 255.255.255.254 interface GigabitEthernet0/0/0/0 Description: Interface to peer node spine eth1/2 ipv4 address 10.6.46.2 255.255.255.0 router static vrf VTS_VIP address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router ospf 100 router-id 10.10.10.10 address-family ipv4 unicast area 0 interface Loopback0 interface GigabitEthernet0/0/0/0 router bgp 100 |
|
On ToR1 |
hostname ToR1 vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature lldp feature vtp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 vpc domain 50 peer-keepalive destination 209.165.201.3 peer-gateway ip arp synchronize interface Ethernet1/1 description: Interface connected to Compute1 eth1. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/2 Description: Interface connected to Controller1 eth1 for dhcp. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/3 Description: Interface connected to Compute1 eth2 for vPC link. switchport mode trunk speed 1000 channel-group 100 no shutdown interface Ethernet1/4 Description: Interface connected to ToR2 eth1/4 for vPC peer link. switchport mode trunk channel-group 20 no shutdown interface port-channel20 Description: Port channel link connected to ToR2 vPC peer link. switchport mode trunk spanning-tree port type network speed 1000 vpc peer-link interface port-channel00 Description: Port channel link connected to compute1 link. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable vpc 50 interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 209.165.200.225/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.4/26 interface loopback0 ip address 2.2.2.2/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 2.2.2.2 area 0.0.0.0 default-cost 10 |
|
On ToR 2 |
vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature lldp feature vtp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 vpc domain 50 peer-keepalive destination 209.165.201.4 peer-gateway ip arp synchronize interface Ethernet1/1 Description: Interface connected to Compute2 eth1. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/3 Description: Interface connected to Compute1 eth3 for vPC link. switchport mode trunk speed 1000 channel-group 100 no shutdown interface Ethernet1/4 Description: Interface connected to ToR2 eth1/4 for vPC peer link. switchport mode trunk channel-group 20 no shutdown interface port-channel20 Description: Port channel link connected to ToR1 vPC peer link. switchport mode trunk spanning-tree port type network speed 1000 vpc peer-link interface port-channel00 Description: Port channel link connected to compute2 link. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable vpc 50 interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 198.51.100.1/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.3/26 interface loopback0 ip address 3.3.3.3/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 3.3.3.3 area 0.0.0.0 default-cost 10 |
|
On ToR 3 - DC Gateway |
hostname ToR3 vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature vtp feature lldp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 interface Ethernet1/1 Description: Interface connected to Compute3 eth1. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/2 Description: Interface connected to DCI G0/0/1/19. no switchport ip address 192.0.2.1/24 no shutdown interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 209.165.202.130/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.4/26 interface loopback0 ip address 4.4.4.4/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 4.4.4.4 area 0.0.0.0 default-cost 10 |
|
On Spine |
hostname SolTB1-Spine1 vdc SolTB1-Spine1 id 1 allocate interface Ethernet1/1-48 allocate interface Ethernet2/1-12 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature vtp feature lldp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 vlan 1 vrf context management ip route 0.0.0.0/0 209.165.201.5 interface Ethernet1/1 Description: Interface connected to XRv1 G0/0/0/0. no switchport ip address 10.6.45.1/24 no shutdown interface Ethernet1/2 Description: Interface connected to XRv2 G0/0/0/0. no switchport ip address 209.165.202.131/24 no shutdown interface Ethernet2/1 Description: Interface connected to ToR1 eth2/1. no switchport ip address 209.165.200.226/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface Ethernet2/2 Description: Interface connected to ToR2 eth2/1. no switchport ip address 198.51.100.1/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface Ethernet2/3 Description: Interface connected to ToR3 DC GW eth2/1. no switchport ip address 13.1.1.1/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.5/26 interface loopback0 ip address 5.5.5.5/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.6.1.2.I3.1.bin router ospf 100 router-id 5.5.5.5 area 0.0.0.0 default-cost 10 |
|
On DCI ASR 9 K |
service unsupported-transceiver hostname asr9k1 username admin password cisco123 group root-system group cisco-support interface MgmtEth0/0/CPU0/0 ipv4 address 209.165.201.6 255.255.255.254 interface GigabitEthernet0/0/1/19 Description: Interface to peer node DC GW ToR3 eth1/2 ipv4 address 209.165.202.131 255.255.255.254 router static address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router bgp 44 bgp-router-id 6.6.6.6 address-family ipv4 unicast address-family vpnv4 unicast lldp |
On Linux Hosts/Servers
- Install lldpad on the server.
-
Run the LLDP Daemon: #lldpad -d
-
Run the following script: for i in `ls /sys/class/net/ | grep eth` ; do echo "enabling lldp for interface: $i" ; lldptool set-lldp -i $i adminStatus=rxtx ; lldptool -T -i $i -V sysName enableTx=yes; lldptool -T -i $i -V portDesc enableTx=yes ; lldptool -T -i $i -V sysDesc enableTx=yes; lldptool -T -i $i -V sysCap enableTx=yes; lldptool -T -i $i -V mngAddr enableTx=yes; done
Server Configuration for vPC – Data-bond in case of OpenStack Compute host which has OVS bridge configured.
Server connected in port channel mode must have a NIC bonding between their Ethernet interfaces. In the example, Server1 - eth2 and eth3 is bonded as data-bond and attached to OVS bridge br-data.
Eth2 - /etc/sysconfig/network-scripts/ifcfg-eth2 DEVICE=eth2 BOOTPROTO=none ONBOOT=yes MASTER=data-bond SLAVE=yes USERCTL=no NM_CONTROLLED=no
Eth1 – /etc/sysconfig/network-scripts/ifcfg-eth3 DEVICE=eth3 BOOTPROTO=none ONBOOT=yes MASTER=data-bond SLAVE=yes USERCTL=no NM_CONTROLLED=no
Data bond - /etc/sysconfig/network-scripts/ifcfg-data-bond DEVICE=data-bond DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-data ONBOOT=yes BOOTPROTO=none NM_CONTROLLED=no USERCTL=no BONDING_OPTS="mode=4"
Some topologies have br-ctrl/br-control instead of br-data. Make changes according to your bridge name. Replace br-data with your bridge name everywhere.
Br-data - /etc/sysconfig/network-scripts/ifcfg-br-data PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no DOMAIN=cisco.com UUID=3ab2bc85-1093-4bda-920a-411662efcdb1 ONBOOT=yes DEVICE=br-data DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge NM_CONTROLLED=no
Add data-bond to ovs bridge br-data:
ovs-vsctl add-port br-data data-bond
[root@vts-tb1-server3 ~]# ovs-vsctl show
96f00f61-b8e5-4634-a29c-d6a11105a84d
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port int-br-data
Interface int-br-data
Bridge br-data
Port br-data
Interface br-data
type: internal
Port phy-br-data
Interface phy-br-data
Port data-bond
Interface data-bond
Restart
networking: Disable NetworkManager if you have. You need to disable
NetworkManager from CIMC. Instead of NetworkManager service enable Network
service. See RedHat docuemntation for steps to disable NetworkManager/enable
Network.
system network restartIf you cannot reach your server after network restart, then you might have lost the default route. So, add it back from your CIMC:
route add default gw <gw_ip>
VTC Configuration
Day Zero Underlay Configuration - External RR (Spine as RR)
The following sections provide details about the day zero configuration required on ToR1 and ToR2 (leafs), DC GW – ToR3, and DCI, with reference to the below example topology, with Spine as BGP Route Reflector.
|
Configuration Items |
Example Configuration (Includes Descriptions) |
| On XRv1 |
hostname XRv1 telnet vrf VTS_VIP ipv4 server max-servers 10 username admin password cisco123 group root-system group cisco-support vrf VTS_VIP address-family ipv4 unicast ! interface MgmtEth0/0/CPU0/0 vrf VTS_VIP ipv4 address 209.165.201.1 255.255.255.254 router static vrf VTS_VIP address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router bgp 100 |
|
On XRv 2 |
hostname XRv2 telnet vrf VTS_VIP ipv4 server max-servers 10 username admin password cisco123 group root-system group cisco-support vrf VTS_VIP address-family ipv4 unicast ! interface MgmtEth0/0/CPU0/0 vrf VTS_VIP ipv4 address 209.165.201.1 255.255.255.254 router static vrf VTS_VIP address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router bgp 100 |
|
On ToR1 |
hostname ToR1 vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature lldp feature vtp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 vpc domain 50 peer-keepalive destination 209.165.201.3 peer-gateway ip arp synchronize interface Ethernet1/1 Description:Interface connected to Compute1 eth1 switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/2 description ***Interface connected to Controller1 eth1 for dhcp*** switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/3 Description: Interface connected to Compute1 eth2 for vPC link. switchport mode trunk speed 1000 channel-group 100 no shutdown interface Ethernet1/4 Description: Interface connected to ToR2 eth1/4 for vPC peer link. switchport mode trunk channel-group 20 no shutdown interface port-channel20 Description: Port channel link connected to ToR2 vPC peer link. switchport mode trunk spanning-tree port type network speed 1000 vpc peer-link interface port-channel00 Description: Port channel link connected to compute1 link. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable vpc 50 interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 209.165.200.225/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.4/26 interface loopback0 ip address 2.2.2.2/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 2.2.2.2 area 0.0.0.0 default-cost 10 router bgp 100
router-id 2.2.2.2
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 5.5.5.5
remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
|
|
On ToR 2 |
vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature lldp feature vtp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 vpc domain 50 peer-keepalive destination 209.165.201.4 peer-gateway ip arp synchronize interface Ethernet1/1 Description: Interface connected to Compute2 eth1. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/3 Description: Interface connected to Compute1 eth3 for vPC link. switchport mode trunk speed 1000 channel-group 100 no shutdown interface Ethernet1/4 Description: Interface connected to ToR2 eth1/4 for vPC peer link. switchport mode trunk channel-group 20 no shutdown interface port-channel20 Description: Port channel link connected to ToR1 vPC peer link. switchport mode trunk spanning-tree port type network speed 1000 vpc peer-link interface port-channel00 Description: Port channel link connected to compute2 link. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable vpc 50 interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 12.1.1.2/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.3/26 interface loopback0 ip address 3.3.3.3/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 3.3.3.3 area 0.0.0.0 default-cost 10 router bgp 100
router-id 3.3.3.3
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 5.5.5.5
remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
|
|
On ToR 3 - DC Gateway |
hostname ToR3 vdc ToR1 id 1 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature vtp feature lldp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup spanning-tree mode mst snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 2001.2001.2001 ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 route-map subnet-policy permit vrf context management ip route 0.0.0.0/0 209.165.201.2 interface Ethernet1/1 Description: Interface connected to Compute3 eth1. switchport mode trunk spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree bpdufilter enable interface Ethernet1/2 Description: Interface connected to DCI G0/0/1/19. no switchport ip address 192.0.2.1/24 no shutdown interface Ethernet2/1 Description: Interface connected to Spine eth2/1. no switchport ip address 198.51.100.2/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.4/26 interface loopback0 ip address 4.4.4.4/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.7.0.3.I1.1.bin router ospf 100 router-id 4.4.4.4 area 0.0.0.0 default-cost 10 router bgp 100
router-id 4.4.4.4
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 5.5.5.5
remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
|
|
On Spine |
vdc SolTB1-Spine1 id 1 allocate interface Ethernet1/1-48 allocate interface Ethernet2/1-12 limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 512 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 feature telnet feature nxapi feature bash-shell cfs eth distribute nv overlay evpn feature ospf feature bgp feature pim feature isis feature interface-vlan feature vn-segment-vlan-based feature lacp feature vpc feature vtp feature lldp feature nv overlay username admin password cisco123 role network-admin no password strength-check ip domain-lookup snmp-server user admin network-admin auth md5 cisco123 priv cisco123 localizedkey rmon event 1 log trap public description FATAL(1) owner PMON@FATAL rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL rmon event 3 log trap public description ERROR(3) owner PMON@ERROR rmon event 4 log trap public description WARNING(4) owner PMON@WARNING rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO ip pim rp-address 2.2.2.2 group-list 224.0.0.0/4 ip pim ssm range 232.0.0.0/8 vlan 1 vrf context management ip route 0.0.0.0/0 209.165.201.5 interface Ethernet1/1 Description: Interface connected to XRv1 G0/0/0/0. no switchport ip address 209.165.202.130/24 no shutdown interface Ethernet1/2 Description: Interface connected to XRv2 G0/0/0/0. no switchport ip address 209.165.202.131/24 no shutdown interface Ethernet2/1 Description: Interface connected to ToR1 eth2/1. no switchport ip address 209.165.200.226/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface Ethernet2/2 Description: Interface connected to ToR2 eth2/1. no switchport ip address 198.51.100.1/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface Ethernet2/3 Description: Interface connected to ToR3 DC GW eth2/1. no switchport ip address 198.51.100.2/24 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode no shutdown interface mgmt0 vrf member management ip address 209.165.201.5/26 interface loopback0 ip address 5.5.5.5/32 ip router ospf 100 area 0.0.0.0 ip pim sparse-mode line console line vty boot nxos bootflash:/n9000-dk9.6.1.2.I3.1.bin router ospf 100 router-id 5.5.5.5 area 0.0.0.0 default-cost 10 router bgp 100
router-id 5.5.5.5
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 2.2.2.2
remote-as 100
update-source loopback0
address-family ipv4 unicast
send-community extended
route-reflector-client
address-family l2vpn evpn
send-community extended
route-reflector-client
neighbor 3.3.3.3
remote-as 100
update-source loopback0
address-family ipv4 unicast
send-community extended
route-reflector-client
address-family l2vpn evpn
send-community extended
route-reflector-client
neighbor 4.4.4.4
remote-as 100
update-source loopback0
address-family ipv4 unicast
send-community extended
route-reflector-client
address-family l2vpn evpn
send-community extended
route-reflector-client
|
|
On DCI ASR 9 K |
service unsupported-transceiver hostname asr9k1 username admin password cisco123 group root-system group cisco-support interface MgmtEth0/0/CPU0/0 ipv4 address 209.165.201.6 255.255.255.0 interface GigabitEthernet0/0/1/19 Description: Interface to peer node DC GW ToR3 eth1/2 ipv4 address 209.165.202.131 255.255.255.0 router static address-family ipv4 unicast 0.0.0.0/0 209.165.201.2 router bgp 44 bgp-router-id 6.6.6.6 address-family ipv4 unicast address-family vpnv4 unicast lldp |
On Linux Hosts/Servers
- Install lldpad on the server.
-
Run the LLDP Daemon: #lldpad -d
-
Run the following script: for i in `ls /sys/class/net/ | grep eth` ; do echo "enabling lldp for interface: $i" ; lldptool set-lldp -i $i adminStatus=rxtx ; lldptool -T -i $i -V sysName enableTx=yes; lldptool -T -i $i -V portDesc enableTx=yes ; lldptool -T -i $i -V sysDesc enableTx=yes; lldptool -T -i $i -V sysCap enableTx=yes; lldptool -T -i $i -V mngAddr enableTx=yes; done
Server Configuration for vPC – Data-bond in case of OpenStack Compute host which has OVS bridge configured.
Server connected in port channel mode must have a NIC bonding between their Ethernet interfaces. In the example, Server1 - eth2 and eth3 is bonded as data-bond and attached to OVS bridge br-data.
Eth2 - /etc/sysconfig/network-scripts/ifcfg-eth2 DEVICE=eth2 BOOTPROTO=none ONBOOT=yes MASTER=data-bond SLAVE=yes USERCTL=no NM_CONTROLLED=no
Eth1 – /etc/sysconfig/network-scripts/ifcfg-eth3 DEVICE=eth3 BOOTPROTO=none ONBOOT=yes MASTER=data-bond SLAVE=yes USERCTL=no NM_CONTROLLED=no
Data bond - /etc/sysconfig/network-scripts/ifcfg-data-bond DEVICE=data-bond DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-data ONBOOT=yes BOOTPROTO=none NM_CONTROLLED=no USERCTL=no BONDING_OPTS="mode=4"
Some topologies have br-ctrl/br-control instead of br-data. Make changes according to your bridge name. Replace br-data with your bridge name everywhere.
Br-data - /etc/sysconfig/network-scripts/ifcfg-br-data PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no DOMAIN=cisco.com UUID=3ab2bc85-1093-4bda-920a-411662efcdb1 ONBOOT=yes DEVICE=br-data DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge NM_CONTROLLED=no
Add data-bond to ovs bridge br-data:
ovs-vsctl add-port br-data data-bond
[root@vts-tb1-server3 ~]# ovs-vsctl show
96f00f61-b8e5-4634-a29c-d6a11105a84d
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port int-br-data
Interface int-br-data
Bridge br-data
Port br-data
Interface br-data
type: internal
Port phy-br-data
Interface phy-br-data
Port data-bond
Interface data-bond
Restart
networking: Disable NetworkManager if you have. You need to disable
NetworkManager from CIMC. Instead of NetworkManager service enable Network
service. See RedHat docuemntation for steps to disable NetworkManager/enable
Network.
system network restartIf you cannot reach your server after network restart, then you might have lost the default route. So, add it back from your CIMC:
route add default gw <gw_ip>
Feedback