Cisco Virtual Topology System (VTS) 2.6.4 Installation Guide

Enabling VTS L2 High Availability

To enable VTC L2 HA, VTC1 and VTC2 must be on the same subnet.

Spawn two VTC VMs. At a minimum, you would need to have three IP addresses for VTC. One for VTC1, One for VTC2, one for the public Virtual IP (VIP). If you are using VTFs, you will also need one for the private VIP, which other devices on the private network such as the VTF can reach.

Note

Cisco VTS supports dual stack clusters for L2 HA. Have both the VTCs (vts01 and vts02) installed and configured with IPv6 & IPv4 address for dual stack to be supported. Both of the VTCs should be reachable by any means with IPv6 address or IPv4 address.

Note

Before enabling HA, make sure that both VTC 1 and VTC 2 have the same password. If not, go to the VTC GUI and do a change password on newly brought up VTC, to make the password identical with that of the other VTC . When you upgrade a VTC / bring up a new VTC / do a hardware upgrade of VTC host, you should make sure that password is the same.

Enabling VTS L2 HA involves:

Setting up the VTC Environment
Enabling VTC High Availability
Registering vCenter to VTC
Enabling VTSR High Availability

Setting up the VTC Environment

You need to set up the VTC environment before you run the high availability script.

Procedure

Step 1

Create a copy of cluster.conf file from cluster.conf.tmpl, which is under the /opt/vts/etc directory. For example:

admin@vts01:~$ cd /opt/vts/etc

admin@vts01:~$ sudo copy cluster.conf.tmpl cluster.conf

Step 2

Specify the VIP address and the details of the two nodes in cluster.conf file . For example:

admin@vts01:/var/# cd /opt/vts/etc/

admin@vts01/etc# sudo vi cluster.conf

 

###Virtual Ip of VTC Master on the public interface. Must fill in at least 1
vip_public=172.23.92.202
vip_public_ipv6=2001:420:10e:2015:c00::202

 

###VTC1 Information. Must fill in at least 1 ip address
master_name=vts01
master_ip=172.23.92.200
master_ipv6=2001:420:10e:2015:c00::200

 

###VTC2 Information. Must fill in at least 1 ip address
slave_name=vts02
slave_ip=172.23.92.201
slave_ipv6=2001:420:10e:2015:c00:201

 

###In the event that a network failure occurs evenly between the two routers, the cluster needs an outside ip to determine where the failure lies
###This can be any external ip such as your vmm ip or a dns but it is recommended to be a stable ip within your environment
###Must fill in at least 1 ip address
external_ip=171.70.168.183
external_ipv6=2001:420:200:1::a

Note

The two nodes communicate each other using VIP IP, and user can use VIP address to login to Cisco VTS UI. You will be directly logged in to the master node, when you use VIP IP address. Make sure that you specify the correct host name, IP Address, and interface type.

Enabling VTC High Availability

You must run the cluster_install.sh script on both VTCs to enable high availability.

Procedure

Step 1

Run the cluster installer script /opt/vts/bin/cluster_install.sh on both VTC1 and VTC2 . For example:

admin@vts02:/opt/vts/etc$ sudo su  - 

[sudo] password for admin:

root@vts02:/opt/vts/etc$ cd ../bin

root@vts02:/opt/vts/bin# ./cluster_install.sh
172.23.92.200 vts01
172.23.92.201 vts02
2001:420:10e:2015:c00::200 vts01
2001:420:10e:2015:c00::201 vts02

Change made to ncs.conf file. Need to restart ncs

Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /lib/systemd/system/pacemaker.service.

Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /lib/systemd/system/corosync.service.

Both nodes are online. Configuring master

Configuring Pacemaker resources

Master node configuration finished

HA cluster is installed

Step 2

Check the status on both the nodes to verify whether both nodes online, and node which got installed first is the master, and the other, slave. For example:

admin@vts02:/opt/vts/log/nso$ sudo crm status

[sudo] password for admin:

Last updated: Mon Apr 10 18:43:52 2017          Last change: Mon Apr 10 17:15:21 2017 by root via crm_attribute on vts01

Stack: corosync

Current DC: vts01 (version 1.1.14-70404b0) - partition with quorum

2 nodes and 4 resources configured

 

Online: [ vts01 vts02 ]

 

Full list of resources:

 

Master/Slave Set: ms_vtc_ha [vtc_ha]

     Masters: [ vts02 ]

     Slaves: [ vts01 ]

ClusterIP      (ocf::heartbeat:IPaddr2):       Started vts02

ClusterIPV6    (ocf::heartbeat:IPaddr2):       Started vts02

Registering vCenter to VTC

To do this:

Procedure

Step 1

Log in to VCSA.

Step 2

Go to Networking > Distributed Virtual Switch > Manage > VTS.

Note

For vCenter 6.5, the VTS comes under Configure tab.

Step 3

Click on System Configuration

Step 4

Enter the following:

VTS IP—This is the Virtual public IP address.
VTS GUI Username
VTS GUI Password

Step 5

Click Update.

Enabling VTSR High Availability

You need to enable VTSR HA, if you have VTFs in your setup. For information about enabling VTSR HA, see Installing VTSR in High Availability Mode.

Enabling VTS L3 High Availability for Underlay and Management Networks

This section describes the procedure to enable VTC L3 HA. In L3 HA environments, VTCs are located on separate overlay networks.

Before enabling VTC L3 HA, make sure that the following requirements are met:

Procedure

Step 1

Both VTC 1 and VTC 2 must have the same password. If they are not the same, change the password via the VTC GUI to make it identical with that of the other VTC. When you upgrade a VTC, bring up a new VTC, or do a hardware upgrade of a VTC host, remember to make sure that the passwords match.

Step 2

VTC1 and VTC2 are deployed in two different networks VTC1 is able to ping VTC 2 and wise versa via both underlay and overlay networks.

We recommend that you add static routes in /etc/network/interfaces on the VTCs. For example:

post-up route add -net 44.44.44.0/24 gw 10.10.10.1 dev eth1

post-up route add -net 10.10.10.0/24 gw 44.44.44.1 dev eth1

Step 3

VTSR installation is mandatory for VTC L3 HA to work. Although the VTSR(s) will not be able to register to the master VTC’s Virtual IP Address (VIP) initially, the master VTC must still be able to reach all VTSRs in order to configure them to support L3HA.

Step 4

A site-id is required and need to be updated in VTSR_template file before generating the VTSR iso file and is generated by either:

Creating a site directly from the VTC1 GUI.
Using the uuidgen linux command to generate a 32 character site ID.

Note

In Dual stack scenario VTC supports dual stack but VTSR can only be either IPV4 or IPv6.

In both cases, note down the site id for use in the below section Deploying VTSR on VMware and Installing VTSR. In the second case the site id will also be required to update the VTC master site configuration and associate the UUID to the site after the HA process is complete.

The implementation of L3 HA is done for both management and underlay networks or management network only. For VIP ip reachability across networks the master VTC will configure BGP and route policies between the VTSRs and their respective directly connected TORs based on the user network settings provided in cluster.conf. Routing policy including tagging helps to migrate VIP IPs across the VTSRs during VTC HA.

Setting up the VTC Environment

You need to set up the VTC environment before you run the high availability script.

Procedure

Step 1

Modify the /opt/vts/etc/cluster.conf_tmpl file on both the VTCs. And rename to cluster.conf .A sample modified file is given below:

Both the VTCs must have the identical information in the cluster.conf file.

###Virtual Ip of VTC Master on the public interface. Must fill in at least 1
vip_public= 192.168.10.254
vip_public_ipv6=
###VTC1 Information. Must fill in at least 1 ip address
master_name=Onion-VTC1
master_ip= 60.60.60.10
master_ipv6=
###VTC2 Information. Must fill in at least 1 ip address
slave_name= Onion-VTC2
slave_ip= 70.70.70.10
slave_ipv6=
###In the event that a network failure occurs evenly between the two routers, the cluster needs an outside ip to determine where the failure lies
###This can be any external ip such as your vmm ip or a dns but it is recommended to be a stable ip within your environment
###Must fill in at least 1 ip address
external_ip= 81.81.81.1
external_ipv6=
###If you intend to use a virtual topology forwarder (VTF) in your environment, please fill in the vip for the underlay as well as the underlay gateway. Otherwise leave blank.
###Virtual Ip of VTC Master on the private interface. You can fill in ipv4 configuration, ipv6, or both if you use both
vip_private= 45.45.45.10
private_gateway=10.10.10.1
vip_private_ipv6=
private_gateway_ipv6=
###If you have your vtc's in different subnets, vtsr will need to be configured to route traffic and the below section needs to be filled in
###If you have your vtc's on the same subnet, the below section should be skipped
###Name of your vrf. Example: VTS_VIP
vrf_name= mgmt-vrf
###Ip of your first Vtsr. Example: 11.1.1.5
vtsr1_mgmt_ip=60.60.60.15
vtsr1_mgmt_ipv6=
###List of neighbors for vtsr1, separated by comma. Example: 11.1.1.1,11.1.1.2
vtsr1_bgp_neighbors=  11.11.11.11
vtsr1_bgp_neighbors_ipv6=
###Ip of your second Vtsr. Example: 12.1.1.5
vtsr2_mgmt_ip= 70.70.70.15
vtsr2_mgmt_ipv6=
###List of neighbors for vtsr2, separated by comma. Example: 12.1.1.1,12.1.1.2
vtsr2_bgp_neighbors= 12.12.12.12
vtsr2_bgp_neighbors_ipv6=
###Username for Vtsr
vtsr_user= admin
###Vtsr ASN information
remote_ASN= 6500
local_ASN= 6501
###Vtsr BGP information
bgp_keepalive= 10
bgp_hold= 30
###Update source for Vtsr1 (i.e. loopback)
vtsr1_update_source=loopback0
###Update source for Vtsr2 (i.e. loopback)
vtsr2_update_source=loopback0
###Router BGP Id for Vtsr1
vtsr1_router_id= 21.21.21.21
###Router BGP Id for Vtsr2
vtsr2_router_id= 31.31.31.31
###Ipv4 Route Distinguisher Loopback for IPv6 Vtsr1 (if VTSR has a defined IPv6 management address, an ipv4 loopback address will be needed for the route distinguisher)
vtsr1_rd_loopback_name=
vtsr1_rd_loopback_address=
###Ipv4 Route Distinguisher Loopback for IPv6 Vtsr2
vtsr2_rd_loopback_name=
vtsr2_rd_loopback_address=

###XRVR1 name
vtsr1_name= vtsr01
###XRVR2 name
vtsr2_name= vtsr02
###If you plan on having your VTC's on different subnets and intend to use a virtual topology forwarder (VTF) in your environment,
### please fill out the following fields. Otherwise, leave blank
###List of neighbors for vtsr1, separated by comma. Example: 2.2.2.2,2.2.2.3
vtsr1_underlay_neighbors= 2.2.2.2
vtsr1_underlay_neighbors_ipv6=
###List of neighbors for vtsr2, separated by comma. Example: 3.3.3.2,3.3.3.3
vtsr2_underlay_neighbors= 6.6.6.6,9.9.9.9
vtsr2_underlay_neighbors_ipv6=
###OSPF Parameters
ospf_id_v4=100
ospf_id_v6=
area=0.0.0.0
default_cost=10
###ISIS Parameters
isis_id= 
is_type=
lsp_mtu=
key_chain_id=
key_id=
cryptographic_algorithm=
#Network name (consist of an even number of octets and be of the form 01.2345.6789.abcd.ef)
vtsr1_network_entity=
vtsr2_network_entity=

###Directly connected Tor information for Vtsr1
vtsr1_directly_connected_device_ip= 10.10.50.3
vtsr1_directly_connected_device_ipv6=
vtsr1_directly_connected_device_user= admin
vtsr1_directly_connected_device_neighbors= 21.21.21.21
vtsr1_directly_connected_device_neighbors_ipv6=
vtsr1_directly_connected_ospf=
vtsr1_directly_connected_router_id=2.2.2.2
vtsr1_directly_connected_update_source=loopback0
###Directly connected Tor information for Vtsr2
vtsr2_directly_connected_device_ip= 10.10.50.7
vtsr2_directly_connected_device_ipv6=
vtsr2_directly_connected_device_user=admin
vtsr2_directly_connected_device_neighbors=31.31.31.31
vtsr2_directly_connected_device_neighbors_ipv6=
vtsr2_directly_connected_ospf=
vtsr2_directly_connected_router_id=6.6.6.6
vtsr2_directly_connected_update_source=loopback0
###VPC Peer information if any. Otherwise leave blank
vtsr1_vpc_peer_ip=
vtsr1_vpc_peer_ipv6=
vtsr1_vpc_peer_user=
vtsr1_vpc_peer_ospf=
vtsr1_vpc_peer_router_id=
vtsr1_vpc_peer_update_source=
vtsr2_vpc_peer_ip=
vtsr2_vpc_peer_ipv6=
vtsr2_vpc_peer_user=
vtsr2_vpc_peer_ospf=
vtsr2_vpc_peer_router_id=
vtsr2_vpc_peer_update_source=
###VTC Underlay Addresses
vtc1_underlay= 10.10.10.10
vtc2_underlay= 44.44.44.44
vtc1_underlay_ipv6=
vtc2_underlay_ipv6=
##Gateway of secondary L3 underlay
vtc2_private_gateway=44.44.44.1
#vtc2_private_gateway_ipv6=

Step 2

Make sure VTSR configuration ISOs are up to date with above configurations. For example:

Note

# The VTS_REGISTRATION_PASSWORD and VTS_SITE_UUID values are VTC UI password and SITE ID those are created on VTC above respectively

# This is a sample VTSR configuration file

# Please protect the generated ISO, as it contains authentication data

# in plain text.

# VTS Registration Information:
#  VTS_ADDRESS should be the IP for VTS. The value must be either an ip or a mask.
#  VTS_ADDRESS is mandatory. If only the V4 version is specified,
#  The V4 management interface for the VTSR (NODE1_MGMT_NETWORK_IP_ADDRESS)
#  will be used. If the V6 version is specified, the V6 management interface
#  for the VTSR (NODE1_MGMT_NETWORK_IPV6_ADDRESS) must be specified and will be used.
VTS_ADDRESS="192.168.10.254"
VTS_IPV6_ADDRESS=
# VTS_REGISTRATION_USERNAME used to login to VTS.
VTS_REGISTRATION_USERNAME="admin"
# VTS_REGISTRATION_PASSWORD is in plaintext.
VTS_REGISTRATION_PASSWORD="<password>"
# VTSR VM Admin user/password
USERNAME="<username>"
PASSWORD="<password>"

# Mandatory Management-VRF name for VTSR.                
VTS_MANAGEMENT_VRF="mgmt-vrf"

# VTSR VM Network Configuration for Node 1:
#  NETWORK_IP_ADDRESS, NETWORK_IP_NETMASK, and NETWORK_IP_GATEWAY
#     are required to complete the setup. Netmask can be in the form of
#     "24" or "255.255.255.0"
#  The first network interface configured with the VTC VM will be used for
#  underlay connectivity; the second will be used for the management network.
#  For both the MGMT and UNDERLAY networks, a <net-name>_NETWORK_IP_GATEWAY
#  variable is mandatory; they are used for monitoring purposes.
#
#  V6 is only supported on the mgmt network and dual stack is
#  currently not supported, so if both are specified V6 will take priority (and
#  requires VTS_IPV6_ADDRESS to be set).
#  The *V6* parameters for the mgmt network are optional. Note that if V6 is used for mgmt
#  it must be V6 on both nodes. Netmask must be the prefix length for V6.
NODE1_MGMT_NETWORK_IP_ADDRESS="60.60.60.15"
NODE1_MGMT_NETWORK_IP_NETMASK="255.255.255.0"
NODE1_MGMT_NETWORK_IP_GATEWAY="60.60.60.1"
NODE1_MGMT_NETWORK_IPV6_ADDRESS=
NODE1_MGMT_NETWORK_IPV6_NETMASK=
NODE1_MGMT_NETWORK_IPV6_GATEWAY=
NODE1_UNDERLAY_NETWORK_IP_ADDRESS="10.10.10.33"
NODE1_UNDERLAY_NETWORK_IP_NETMASK="255.255.255.0"
NODE1_UNDERLAY_NETWORK_IP_GATEWAY="10.10.10.1"
# AUX network is optional
#NODE1_AUX_NETWORK_IP_ADDRESS="169.254.20.100"
#NODE1_AUX_NETWORK_IP_NETMASK="255.255.255.0"
#NODE1_AUX_NETWORK_IP_GATEWAY="169.254.20.1"
# XR Hostname
NODE1_XR_HOSTNAME="vtsr01"
# Loopback IP and netmask
NODE1_LOOPBACK_IP_ADDRESS="128.0.0.10"
NODE1_LOOPBACK_IP_NETMASK="255.255.255.255"

# Operational username and password - optional
# These need to be configured to start monit on VTSR

VTSR_OPER_USERNAME="admin"
# Password needs an encrypted value
# Example : "openssl passwd -1 -salt <salt-string> <password>"
VTSR_OPER_PASSWORD="$1$cisco$Qv2TLtPNI3jqwXMOA3M3f0/"

# VTSR monit interval - optional - default is 30 seconds
VTSR_MONIT_INTERVAL="30"

# VTSR VM Network Configuration for Node 2:
#  If there is no HA then the following Node 2 configurations will remain commented and
#  will not be used and Node 1 configurations alone will be applied
#  For HA , the following Node 2 configurations has to be uncommented
#  VTSR VM Network Configuration for Node 2
#  NETWORK_IP_ADDRESS, NETWORK_IP_NETMASK, and NETWORK_IP_GATEWAY
#     are required to complete the setup. Netmask can be in the form of
#     "24" or "255.255.255.0"
#  The first network interface configured with the VTC VM will be used for
#  underlay connectivity; the second will be used for the management network.
#  For both the MGMT and UNDERLAY networks, a <net-name>_NETWORK_IP_GATEWAY
#  variable is mandatory; they are used for monitoring purposes.
#
#  V6 is only supported on the mgmt network and dual stack is
#  currently not supported, so if both are specified V6 will take priority (and
#  requires VTS_IPV6_ADDRESS to be set).
#  The *V6* parameters for the mgmt network are optional. Note that if V6 is used for mgmt
#  it must be V6 on both nodes. Netmask must be the prefix length for V6.
NODE2_MGMT_NETWORK_IP_ADDRESS="70.70.70.15"
NODE2_MGMT_NETWORK_IP_NETMASK="255.255.255.0"
NODE2_MGMT_NETWORK_IP_GATEWAY="70.70.70.1"
NODE2_MGMT_NETWORK_IPV6_ADDRESS=
NODE2_MGMT_NETWORK_IPV6_NETMASK=
NODE2_MGMT_NETWORK_IPV6_GATEWAY=
NODE2_UNDERLAY_NETWORK_IP_ADDRESS="44.44.44.15"
NODE2_UNDERLAY_NETWORK_IP_NETMASK="255.255.255.0"
NODE2_UNDERLAY_NETWORK_IP_GATEWAY="44.44.44.1"
#  AUX network is optional
#  Although Aux network is optional it should be either present in both nodes
#  or not present in both nodes.
#  It cannot be present on Node1 and not present on Node2 and vice versa
#NODE2_AUX_NETWORK_IP_ADDRESS="179.254.20.200"
#NODE2_AUX_NETWORK_IP_NETMASK="255.255.255.0"
#NODE2_AUX_NETWORK_IP_GATEWAY="179.254.20.1"
# XR Hostname
NODE2_XR_HOSTNAME="vtsr02"
# Loopback IP and netmask
NODE2_LOOPBACK_IP_ADDRESS="130.0.0.1"
NODE2_LOOPBACK_IP_NETMASK="255.255.255.255"

# VTS site uuid
VTS_SITE_UUID="abcdefab-abcd-abcd-abcd-abcdefabcdef"

Enabling VTC High Availability

To enable VTC high availability, perform the following steps.

Note

Step 1 to 3 have to be run on both VTCs. Step 4 must be run only on the node that you want to make the active VTC.

Procedure

Step 1	SSH to VTC 1 and VTC 2
Step 2	Go to the following directory: `cd /opt/vts/etc/`
Step 3	Copy the cluster.conf file to /opt/vts/etc on both VTC 1 and VTC 2.
Step 4	Go to the following directory: `cd /opt/vts/bin` Run the sudo ./cluster_install.sh command. For example: `admin@Onion-VTC1:/opt/vts/bin#sudo ./cluster_install.sh` You will be asked to provide the vtsr password. vtsr password is the password for VTSR1 and VTSR2. In addition to this, you will also be asked for the passwords for the switches directly connected to VTSR1 and VTSR2. And you will be prompted to run cluster_install.sh on VTC2 as well A message similar to below: `Please run cluster_install.sh on Onion-VTC2. Will wait until finished ==> At this point on VTC2 run the cluster install script` Run the sudo ./cluster_install.sh command. For example: `admin@Onion-VTC2:/opt/vts/bin#sudo ./cluster_install.sh` You will be asked to provide the vtsr password. vtsr password is the password for VTSR1 and VTSR2. In addition to this, you will also be asked for the passwords for the switches directly connected to VTSR1 and VTSR2.An output similar to what is given below is displayed: Change made to ncs.conf file. Need to restart ncs Finding running docker container ID 263f311dbdff Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /lib/systemd/system/pacemaker.service. Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /lib/systemd/system/corosync.service. Retrieving and storing node certificates Retrieving master certificate Retrieving slave certificate Importing master certificate Importing slave certificate HA cluster is installed admin@Onion-VTC2:/opt/vts/bin# Once this is over on VTC2 ,Automatically on VTC1 the process will resume and you will see an output similar to what is given below: `2018-08-25 19:33:31,358 - INFO - Setup finished at 19:33:31 Configuring Pacemaker resources Master node configuration finished Retrieving and storing node certificates Retrieving master certificate Retrieving slave certificate Importing master certificate Importing slave certificate HA cluster is installed. admin@Onion-VTC1:/opt/vts/bin#`

Enabling VTS L3 High Availablity Management Network Only

This section describes the procedure to enable VTC L3 HA. In L3 HA environments, VTCs are located on separate overlay networks.

Before enabling VTC L3 HA, make sure that the following requirements are met

:

Procedure

Step 1	Both VTC 1 and VTC 2 must have the same password. If they are not the same, change the password via the VTC GUI to make it identical with that of the other VTC. When you upgrade a VTC, bring up a new VTC, or do a hardware upgrade of a VTC host, remember to make sure that the passwords match.
Step 2	VTC1 and VTC2 are deployed in two different networks VTC1 is able to ping VTC 2 and wise versa via both underlay and overlay networks. We recommend that you add static routes in /etc/network/interfaces on the VTCs. For example: `sudo vi /etc/network/interfaces post-up route add -net 44.44.44.0/24 gw 10.10.10.1 dev eth1 post-up route add -net 10.10.10.0/24 gw 44.44.44.1 dev eth1`
Step 3	VTSR installation is mandatory for VTC L3 HA to work. Although the VTSR(s) will not be able to register to the master VTC’s Virtual IP Address (VIP) initially, the master VTC must still be able to reach all VTSRs in order to configure them to support L3HA.
Step 4	A site-id is required and need to be updated in VTSR_template file before generating the VTSR iso file and is generated by either Creating a site directly from the VTC1 GUI. Using the uuidgen linux command to generate a 32 character site ID. In both cases, note down the site id for use in the below section. In the second case the site id will also be required to update the VTC master site configuration and associate the UUID to the site after the HA process is complete. The implementation of L3 HA is done for both management and underlay networks or management networks only. For VIP ip reachability across networks the master VTC will configure BGP and route policies between the VTSRs and their respective directly connected TORs based on the user network settings provided in cluster.conf. Routing policy including tagging helps to migrate VIP IPs across the VTSRs during VTC HA.

Enabling IOS XRv High Availability

You need to enable IOS XRv HA, if you have VTFs in your setup. For information about enabling IOS XRv HA, see Installing VTSR in High Availability Mode.

Registering vCenter to VTC

To do this:

Procedure

Step 1

Log in to VCSA.

Step 2

Go to Networking > Distributed Virtual Switch > Manage > VTS.

Note

For vCenter 6.5, the VTS comes under the Configure tab.

Step 3

Click on System Configuration

Step 4

Enter the following:

VTS IP—This is the Virtual public IP address.
VTS GUI Username
VTS GUI Password

Step 5

Click Update.

Enabling VTS L3 High Availability for Management Network only

Setting up the VTC Environment for L3 High Availability Management

Use tthis procedure only to set up the VTC environment for L3 Hig Availabitiy (HA) management.

Procedure

Step 1

Copy the /opt/vts/etc/cluster.conf_tmpl file to /opt/vts/etc/cluster.conf on both VTCs. The cluster.conf files must be identical on both VTCs. A sample modified file is given below:

###Virtual Ip of VTC Master on the public interface. Must fill in at least 1
vip_public= 192.168.10.254
vip_public_ipv6=
 
###VTC1 Information. Must fill in at least 1 ip address
master_name=Onion-VTC1
master_ip=60.60.60.10
master_ipv6=
 
###VTC2 Information. Must fill in at least 1 ip address
slave_name= Onion-VTC2
slave_ip=70.70.70.10
slave_ipv6=
 
###In the event that a network failure occurs evenly between the two routers, the cluster needs an outside ip to determine where the failure lies
###This can be any external ip such as your vmm ip or a dns but it is recommended to be a stable ip within your environment
###Must fill in at least 1 ip address
external_ip=81.81.81.1
external_ipv6=
 
###If you intend to use a virtual topology forwarder (VTF) in your environment, please fill in the vip for the underlay as well as the underlay gateway. Otherwise leave blank.
###Virtual Ip of VTC Master on the private interface. You can fill in ipv4 configuration, ipv6, or both if you use both
vip_private=
private_gateway=
 
vip_private_ipv6=
private_gateway_ipv6
###If you have your vtc's in different subnets, vtsr will need to be configured to route traffic and the below section needs to be filled in
###If you have your vtc's on the same subnet, the below section should be skipped
 
###Name of your vrf. Example: VTS_VIP
vrf_name=mgmt-vrf
 
###Ip of your first Vtsr. Example: 11.1.1.5
vtsr1_mgmt_ip=60.60.60.15
vtsr1_mgmt_ipv6=
 
###List of neighbors for vtsr1, separated by comma. Example: 11.1.1.1,11.1.1.2
vtsr1_bgp_neighbors= 11.11.11.11
vtsr1_bgp_neighbors_ipv6=
 
###Ip of your second Vtsr. Example: 12.1.1.5
vtsr2_mgmt_ip=70.70.70.15
vtsr2_mgmt_ipv6=
 
###List of neighbors for vtsr2, separated by comma. Example: 12.1.1.1,12.1.1.2
vtsr2_bgp_neighbors= 12.12.12.12
vtsr2_bgp_neighbors_ipv6=
 
###Username for Vtsr
vtsr_user= admin
 
###Vtsr ASN information
remote_ASN= 6500
local_ASN= 6501

###Vtsr BGP information
bgp_keepalive= 10
bgp_hold= 30
 
###Update source for Vtsr1 (i.e. loopback)
vtsr1_update_source=loopback0
 
###Update source for Vtsr2 (i.e. loopback)
vtsr2_update_source=loopback0
 
###Router BGP Id for Vtsr1
vtsr1_router_id= 21.21.21.21
 
###Router BGP Id for Vtsr2
vtsr2_router_id= 31.31.31.31
 
###Ipv4 Route Distinguisher Loopback for IPv6 Vtsr1 (if VTSR has a defined IPv6 management address, an ipv4 loopback address will be needed for the route distinguisher)
vtsr1_rd_loopback_name=loopback5
vtsr1_rd_loopback_address=41.41.41.41
 
###Ipv4 Route Distinguisher Loopback for IPv6 Vtsr2
vtsr2_rd_loopback_name=loopback5
vtsr2_rd_loopback_address=51.51.51.51
 
 
###XRVR1 name
vtsr1_name= vtsr01
 
###XRVR2 name
vtsr2_name= vtsr02
 
###If you plan on having your VTC's on different subnets and intend to use a virtual topology forwarder (VTF) in your environment,
### please fill out the following fields. Otherwise, leave blank
 
###List of neighbors for vtsr1, separated by comma. Example: 2.2.2.2,2.2.2.3
vtsr1_underlay_neighbors=
vtsr1_underlay_neighbors_ipv6=
 
###List of neighbors for vtsr2, separated by comma. Example: 3.3.3.2,3.3.3.3
vtsr2_underlay_neighbors=
vtsr2_underlay_neighbors_ipv6=
 
###OSPF Parameters
ospf_id_v4=
ospf_id_v6=
area=
default_cost=
 
###ISIS Parameters
isis_id= 
is_type= 
lsp_mtu=
key_chain_id= 
key_id= 
cryptographic_algorithm=
Network name (consist of an even number of octets and be of the form 01.2345.6789.abcd.ef)
vtsr1_network_entity=
vtsr2_network_entity=

###Directly connected Tor information for Vtsr1
vtsr1_directly_connected_device_ip=
vtsr1_directly_connected_device_ipv6=
vtsr1_directly_connected_device_user=
vtsr1_directly_connected_device_neighbors=
vtsr1_directly_connected_device_neighbors_ipv6=
vtsr1_directly_connected_ospf=
vtsr1_directly_connected_router_id=
vtsr1_directly_connected_update_source=
 
###Directly connected Tor information for Vtsr2
vtsr2_directly_connected_device_ip=
vtsr2_directly_connected_device_ipv6=
vtsr2_directly_connected_device_user=
vtsr2_directly_connected_device_neighbors=
vtsr2_directly_connected_device_neighbors_ipv6=
vtsr2_directly_connected_ospf=
vtsr2_directly_connected_router_id=
vtsr2_directly_connected_update_source=
 
###VPC Peer information if any. Otherwise leave blank
vtsr1_vpc_peer_ip=
vtsr1_vpc_peer_ipv6=
vtsr1_vpc_peer_user=
vtsr1_vpc_peer_ospf=
vtsr1_vpc_peer_router_id=
vtsr1_vpc_peer_update_source=
 
vtsr2_vpc_peer_ip=
vtsr2_vpc_peer_ipv6=
vtsr2_vpc_peer_user=
vtsr2_vpc_peer_ospf=
vtsr2_vpc_peer_router_id=
vtsr2_vpc_peer_update_source=
 
###VTC Underlay Addresses
vtc1_underlay=
vtc2_underlay=
vtc1_underlay_ipv6=
vtc2_underlay_ipv6=
 
##Gateway of secondary L3 underlay
vtc2_private_gateway=
vtc2_private_gateway_ipv6=

Step 2

Make sure VTSR configuration ISOs are up to date with above configurations. For example:

Note

The VTSR values VTS_REGISTRATION_PASSWORD and VTS_SITE_UUID map to the VTC UI password and SITE ID on the VTCs, respectively.

# This is a sample VTSR configuration file
# Copyright (c) 2015 cisco Systems
 
# Please protect the generated ISO, as it contains authentication data
# in plain text.
 
# VTS Registration Information:
#  VTS_ADDRESS should be the IP for VTS. The value must be either an ip or a mask.
#  VTS_ADDRESS is mandatory. If only the V4 version is specified,
#  The V4 management interface for the VTSR (NODE1_MGMT_NETWORK_IP_ADDRESS)
#  will be used. If the V6 version is specified, the V6 management interface
#  for the VTSR (NODE1_MGMT_NETWORK_IPV6_ADDRESS) must be specified and will be used.
VTS_ADDRESS="192.168.10.254"
VTS_IPV6_ADDRESS=
# VTS_REGISTRATION_USERNAME used to login to VTS.
VTS_REGISTRATION_USERNAME="admin"
# VTS_REGISTRATION_PASSWORD is in plaintext.
VTS_REGISTRATION_PASSWORD="Cisco123!"
# VTSR VM Admin user/password
USERNAME="admin"
PASSWORD="cisco123"
 
# Mandatory Management-VRF name for VTSR.                
VTS_MANAGEMENT_VRF="mgmt-vrf"
 
# VTSR VM Network Configuration for Node 1:
#  NETWORK_IP_ADDRESS, NETWORK_IP_NETMASK, and NETWORK_IP_GATEWAY
#     are required to complete the setup. Netmask can be in the form of
#     "24" or "255.255.255.0"
#  The first network interface configured with the VTC VM will be used for
#  underlay connectivity; the second will be used for the management network.
#  For both the MGMT and UNDERLAY networks, a <net-name>_NETWORK_IP_GATEWAY
#  variable is mandatory; they are used for monitoring purposes.
#
#  V6 is only supported on the mgmt network and dual stack is
#  currently not supported, so if both are specified V6 will take priority (and
#  requires VTS_IPV6_ADDRESS to be set).
#  The *V6* parameters for the mgmt network are optional. Note that if V6 is used for mgmt
#  it must be V6 on both nodes. Netmask must be the prefix length for V6.
NODE1_MGMT_NETWORK_IP_ADDRESS="60.60.60.15"
NODE1_MGMT_NETWORK_IP_NETMASK="255.255.255.0"
NODE1_MGMT_NETWORK_IP_GATEWAY="60.60.60.1"
NODE1_MGMT_NETWORK_IPV6_ADDRESS=
NODE1_MGMT_NETWORK_IPV6_NETMASK=
NODE1_MGMT_NETWORK_IPV6_GATEWAY=
NODE1_UNDERLAY_NETWORK_IP_ADDRESS="10.10.10.33"
NODE1_UNDERLAY_NETWORK_IP_NETMASK="255.255.255.0"
NODE1_UNDERLAY_NETWORK_IP_GATEWAY="10.10.10.1"
# AUX network is optional
#NODE1_AUX_NETWORK_IP_ADDRESS="169.254.20.100"
#NODE1_AUX_NETWORK_IP_NETMASK="255.255.255.0"
#NODE1_AUX_NETWORK_IP_GATEWAY="169.254.20.1"
# XR Hostname
NODE1_XR_HOSTNAME="vtsr01"
# Loopback IP and netmask
NODE1_LOOPBACK_IP_ADDRESS="128.0.0.10"
NODE1_LOOPBACK_IP_NETMASK="255.255.255.255"
 
# Operational username and password - optional
# These need to be configured to start monit on VTSR
 
VTSR_OPER_USERNAME="admin"
# Password needs an encrypted value
# Example : "openssl passwd -1 -salt <salt-string> <password>"
VTSR_OPER_PASSWORD="$1$cisco$Qv2TLtPNI3jqwXMOA3M3f0/"
 
# VTSR monit interval - optional - default is 30 seconds
VTSR_MONIT_INTERVAL="30"
 
# VTSR VM Network Configuration for Node 2:
#  If there is no HA then the following Node 2 configurations will remain commented and
#  will not be used and Node 1 configurations alone will be applied
#  For HA , the following Node 2 configurations has to be uncommented
#  VTSR VM Network Configuration for Node 2
#  NETWORK_IP_ADDRESS, NETWORK_IP_NETMASK, and NETWORK_IP_GATEWAY
#     are required to complete the setup. Netmask can be in the form of
#     "24" or "255.255.255.0"
#  The first network interface configured with the VTC VM will be used for
#  underlay connectivity; the second will be used for the management network.
#  For both the MGMT and UNDERLAY networks, a <net-name>_NETWORK_IP_GATEWAY
#  variable is mandatory; they are used for monitoring purposes.
#
#  V6 is only supported on the mgmt network and dual stack is
#  currently not supported, so if both are specified V6 will take priority (and
#  requires VTS_IPV6_ADDRESS to be set).
#  The *V6* parameters for the mgmt network are optional. Note that if V6 is used for mgmt
#  it must be V6 on both nodes. Netmask must be the prefix length for V6.
NODE2_MGMT_NETWORK_IP_ADDRESS="70.70.70.15"
NODE2_MGMT_NETWORK_IP_NETMASK="255.255.255.0"
NODE2_MGMT_NETWORK_IP_GATEWAY="70.70.70.1"
NODE2_MGMT_NETWORK_IPV6_ADDRESS=
NODE2_MGMT_NETWORK_IPV6_NETMASK=
NODE2_MGMT_NETWORK_IPV6_GATEWAY=
NODE2_UNDERLAY_NETWORK_IP_ADDRESS="44.44.44.15"
NODE2_UNDERLAY_NETWORK_IP_NETMASK="255.255.255.0"
NODE2_UNDERLAY_NETWORK_IP_GATEWAY="44.44.44.1"
#  AUX network is optional
#  Although Aux network is optional it should be either present in both nodes
#  or not present in both nodes.
#  It cannot be present on Node1 and not present on Node2 and vice versa
#NODE2_AUX_NETWORK_IP_ADDRESS="179.254.20.200"
#NODE2_AUX_NETWORK_IP_NETMASK="255.255.255.0"
#NODE2_AUX_NETWORK_IP_GATEWAY="179.254.20.1"
# XR Hostname
NODE2_XR_HOSTNAME="vtsr02"
# Loopback IP and netmask
NODE2_LOOPBACK_IP_ADDRESS="130.0.0.1"
NODE2_LOOPBACK_IP_NETMASK="255.255.255.255"
 
# VTS site uuid
VTS_SITE_UUID="abcdefab-abcd-abcd-abcd-abcdefabcdef"

Deploying VTSR VMs

To deploy VTSR 1 and VTSR 2, follow the below steps:

Procedure

Step 1	Generate an ISO file for the VTSR VM. See Generating an ISO for VTSR.
Step 2	After a VTSR is up and running, ssh to it’s management ip address and make sure that the routes are added and both VTC 1 and VTC 2 can reach VTSR 1 and VTSR 2, and vice versa.

Day Zero Configuration for High Availability

The following example shos Day zero configuration for L3 High Availability:

VTSR1 #
 
vrf mgmt-vrf
address-family ipv4 unicast
!
address-family ipv6 unicast
!
!
tpa
vrf default
  address-family ipv4
   update-source dataports GigabitEthernet0/0/0/0
  !
!
vrf mgmt-vrf
  address-family ipv4
   update-source dataports GigabitEthernet0/0/0/1
  !
  address-family ipv6
   update-source dataports GigabitEthernet0/0/0/1
  !
!
!
interface Loopback0
ipv4 address 128.0.0.10 255.255.255.255
!
interface Loopback1
ipv4 address 169.254.10.1 255.255.255.255
!
interface MgmtEth0/RP0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 10.10.10.33 255.255.255.0
!
interface GigabitEthernet0/0/0/1
vrf mgmt-vrf
ip address 60.60.60.15/24
!
interface GigabitEthernet0/0/0/2
shutdown
!
router static
address-family ipv4 unicast
   0.0.0.0/0  60.60.60.1
  !
!
!
 
VTSR2 #
 
vrf mgmt-vrf
address-family ipv4 unicast
!
address-family ipv6 unicast
!
!
tpa
vrf default
  address-family ipv4
   update-source dataports GigabitEthernet0/0/0/0
  !
!
vrf mgmt-vrf
  address-family ipv4
   update-source dataports GigabitEthernet0/0/0/1
  !
  address-family ipv6
   update-source dataports GigabitEthernet0/0/0/1
  !
!
!
call-home
service active
contact smart-licensing
profile CiscoTAC-1
  active
  destination transport-method http
!
!
interface Loopback0
ipv4 address 128.0.0.10 255.255.255.255
!
interface Loopback1
ipv4 address 169.254.10.1 255.255.255.255
!
interface MgmtEth0/RP0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 44.44.44.15 255.255.255.0
!
interface GigabitEthernet0/0/0/1
vrf mgmt-vrf
ipv4 address 70.70.70.15/24
!
interface GigabitEthernet0/0/0/2
shutdown
!
router static
address-family ipv4 unicast
  0.0.0.0/0 44.44.44.1
!
vrf mgmt-vrf
  address-family ipv4 unicast
   0.0.0.0/0 70.70.70.1
  !
!
!

Verifying VTSR High Availability Setup

Check the HA status with command sudo crm status . For example:

root@vtsr01:/opt/cisco/package# crm status

Last updated: Thu Aug 30 16:27:25 2018 Last change: Thu Aug 30 14:52:01 2018 by root via cibadmin on vtsr01

Stack: corosync

Current DC: vtsr01 (version 1.1.14-70404b0) - partition with quorum

2 nodes and 24 resources configured

Online: [ vtsr01 vtsr02 ]

Full list of resources:

dl_server        (ocf::heartbeat:anything):        Started vtsr01
 redis_server        (ocf::heartbeat:anything):        Started vtsr01
 Clone Set: cfg_dl_clone [cfg_dl]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: rc_clone [rc]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: sm_clone [sm]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: tunnel_clone [tunnel]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: confd_clone [confd]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: mping_clone [mgmt_ping]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: uping_clone [underlay_ping]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: monit_clone [monit]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: socat_confd_clone [socat-confd]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: socat_monit_clone [socat-monit]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: mate_tunnel_clone [mate_tunnel]
     Started: [ vtsr01 vtsr02 ]
root@vtsr01:/opt/cisco/package#

root@vtsr02:/opt/cisco/package# crm status

Last updated: Thu Aug 30 16:32:06 2018 Last change: Thu Aug 30 14:52:01 2018 by root via cibadmin on vtsr01

Stack: corosync

Current DC: vtsr01 (version 1.1.14-70404b0) - partition with quorum

2 nodes and 24 resources configured

Online: [ vtsr01 vtsr02 ]

Full list of resources:

 dl_server        (ocf::heartbeat:anything):        Started vtsr01
 redis_server        (ocf::heartbeat:anything):        Started vtsr01
 Clone Set: cfg_dl_clone [cfg_dl]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: rc_clone [rc]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: sm_clone [sm]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: tunnel_clone [tunnel]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: confd_clone [confd]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: mping_clone [mgmt_ping]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: uping_clone [underlay_ping]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: monit_clone [monit]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: socat_confd_clone [socat-confd]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: socat_monit_clone [socat-monit]
     Started: [ vtsr01 vtsr02 ]
 Clone Set: mate_tunnel_clone [mate_tunnel]
     Started: [ vtsr01 vtsr02 ]
root@vtsr02:/opt/cisco/package#

Enabling VTC High Availability

To enable VTC high availability, do the following steps:

Note

Step 1 to 3 should be run on both VTCs. Step 4 must be run only on the node that you want to make the active VTC.

Procedure

Step 1	SSH to VTC 1 and VTC 2 .
Step 2	Go to the following directory: cd /opt/vts/etc/
Step 3	Copy the cluster.conf file to /opt/vts/etc on both VTC 1 and VTC 2.
Step 4	Go to the following directory:: cd /opt/vts/bin Run the sudo ./cluster_install.sh command. For example: admin@Onion-VTC1:/opt/vts/bin#sudo ./cluster_install.sh You will be asked to provide the vtsr password. vtsr password is the password for VTSR1 and VTSR2. You will be prompted to run cluster_install.sh on VTC2 as well . A message is displyaed similar to below example: Please run cluster_install.sh on Onion-VTC2. Will wait until finished ==> At this point on VTC2 run the cluster install script Run the sudo ./cluster_install.sh command. For example: admin@Onion-VTC2:/opt/vts/bin#sudo ./cluster_install.sh You will be asked to provide the vtsr password. vtsr password is the password for VTSR1 and VTSR2. An output similar to what is given below is displayed: Change made to ncs.conf file. Need to restart ncs Finding running docker container ID 263f311dbdff Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /lib/systemd/system/pacemaker.service. Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /lib/systemd/system/corosync.service. Retrieving and storing node certificates Retrieving master certificate Retrieving slave certificate Importing master certificate Importing slave certificate HA cluster is installed admin@Onion-VTC2:/opt/vts/bin# Once this is over on VTC2 ,Automatically on VTC1 the process will resume and you will see an output similar to what is given below : 2018-08-25 19:33:31,358 - INFO - Setup finished at 19:33:31 Configuring Pacemaker resources Master node configuration finished Retrieving and storing node certificates Retrieving master certificate Retrieving slave certificate Importing master certificate Importing slave certificate HA cluster is installed. admin@Onion-VTC1:/opt/vts/bin#

Verifying the VTC High Availabilty

To verify the

Procedure

Step 1

Run the sudo crm status command. For example

admin@Onion-VTC1:~$ sudo crm status

[sudo] password for admin:

Last updated: Mon Aug 27 17:34:47 2018 Last change: Sat Aug 25 20:20:11 2018 by root via crm_attribute on Onion-VTC2

Stack: corosync

Current DC: Onion-VTC1 (version 1.1.14-70404b0) - partition with quorum

2 nodes and 5 resources configured

Online: [ Onion-VTC1 Onion-VTC2 ]

Full list of resources:

Master/Slave Set: ms_vtc_ha [vtc_ha]

Masters: [ Onion-VTC1 ]

Slaves: [ Onion-VTC2 ]

ClusterIP (ocf::heartbeat:IPaddr2): Started Onion-VTC1

ClusterIP2 (ocf::heartbeat:IPaddr2): Started Onion-VTC1

admin@Onion-VTC1:~$

Step 2

Verify on which VTC the virtual IP is configured. For example:

root@Onion-VTC1:/home/admin# ip addr s

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 52:54:00:c7:92:c2 brd ff:ff:ff:ff:ff:ff

inet 60.60.60.10/24 brd 60.60.60.255 scope global eth0