Ensure Proper Virtual WSA HA Group Functionality in a VMware Environment

Available Languages

Updated:July 30, 2015
Document ID:119188

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes the process that must be completed so that the Cisco Web Security Appliance (WSA) High Availability (HA) feature works properly on a Virtual WSA that runs in a VMware environment.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

  • Cisco WSA

  • HTTP

  • Multicast traffic

  • Common Address Resolution Protocol (CARP)

Components Used

The information in this document is based on these software and hardware versions:

  • AsyncOS for Web Version 8.5 or later

  • VMware ESXi Version 4.0 or later

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

A Virtual WSA that is configured with one or more HA groups always has the HA in the backup state, even when the priority is the highest.

The system logs show constant flapping, as shown in this log snippet:

Tue May 19 08:05:52 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)
Tue May 19 08:05:52 2015 Info: Interface Failover Group 94 is down
Tue May 19 08:06:01 2015 Info: Interface Failover Group 94 is up
Tue May 19 08:06:01 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)
Tue May 19 08:06:01 2015 Info: Interface Failover Group 94 is down
Tue May 19 08:06:10 2015 Info: Interface Failover Group 94 is up
Tue May 19 08:06:10 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)
Tue May 19 08:06:10 2015 Info: Interface Failover Group 94 is down
Tue May 19 08:06:19 2015 Info: Interface Failover Group 94 is up
Tue May 19 08:06:19 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)
Tue May 19 08:06:19 2015 Info: Interface Failover Group 94 is down
Tue May 19 08:06:28 2015 Info: Interface Failover Group 94 is up
Tue May 19 08:06:28 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)
Tue May 19 08:06:28 2015 Info: Interface Failover Group 94 is down
Tue May 19 08:06:37 2015 Info: Interface Failover Group 94 is up
Tue May 19 08:06:37 2015 Info: Interface Failover Group 94 has changed
 role from Master to Backup (more frequent advertisement received)

If you take a packet capture (for multicast IP address 224.0.0.18 in this example), you might observe an output similar to this:

13:49:04.601713 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283
13:49:04.601931 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283
13:49:04.602798 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283
13:49:04.602809 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283
13:49:13.621706 IP (tos 0x10, ttl 255, id 24801, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178284
13:49:13.622007 IP (tos 0x10, ttl 255, id 24801, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178284
13:49:13.622763 IP (tos 0x10, ttl 255, id 24801, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178284
13:49:13.622770 IP (tos 0x10, ttl 255, id 24801, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178284
13:49:22.651653 IP (tos 0x10, ttl 255, id 44741, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178285

Problem Analysis

The WSA system logs that are provided in the previous section indicate that when the HA group becomes a Master in the CARP negotiation, there is an advertisement that is received with a better priority.

You can verify this also from the packet capture. This is the packet that is sent from the virtual WSA:

13:49:04.601713 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283

In a milliseconds time-frame, you can see another set of packets from the same source IP address (the same virtual WSA appliance):

13:49:04.602798 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283
13:49:04.602809 IP (tos 0x10, ttl 255, id 4785, offset 0, flags [DF],
 proto VRRP (112), length 56)
   192.168.0.131 > 224.0.0.18: carp 192.168.0.131 > 224.0.0.18: CARPv2-advertise 36:
    vhid=94 advbase=3 advskew=1 authlen=7 counter=15790098039517178283

In this example, the source IP address of 192.168.0.131 is the IP address of the problematic virtual WSA. It appears that the multicast packets are looped back to the virtual WSA.

This issue occurs due to a defect on the VMware side, and the next section explains the steps that you must complete in order to resolve the issue.

Solution

Complete these steps in order to resolve this issue and stop the loop of multicast packets that are sent in the VMware environment:

  1. Enable promiscuous mode on the Virtual Switch (vSwitch).

  2. Enable MAC Address changes.

  3. Enable Forged transmits.

  4. If multiple physical ports exist on the same vSwitch, then the Net.ReversePathFwdCheckPromisc option must be enabled in order to work around a vSwitch bug where the multicast traffic loops back to the host, which causes the CARP to not function with link states coalesced messages. (Refer to the next section for additional information).

Modify the Net.ReversePathFwdCheckPromisc Option

Complete these steps in order to modify the Net.ReversePathFwdCheckPromisc option:

  1. Log into the VMware vSphere client.

  2. Complete these steps for each VMware host:

    1. Click host, and navigate to the Configuration tab.

    2. Click Software Advanced Settings from the left pane.

    3. Click Net and scroll down to the Net.ReversePathFwdCheckPromisc option.

    4. Set the Net.ReversePathFwdCheckPromisc option to 1.

    5. Click OK.

The interfaces that are in Promiscuous mode must now be set, or turned off and then back on. This is completed on a per-host basis.

Complete these steps in order to set the interfaces:

  1. Navigate to the Hardware section and click Networking.

  2. Complete these steps for each vSwitch and/or Virtual Machine (VM) port group:

    1. Click Properties from the vSwitch.

    2. By default, Promiscuous mode is set to Reject. In order to change this setting, click edit and navigate to the Security tab.

    3. Select Accept from drop-down menu.

    4. Click OK.

Note: This setting is usually applied on a per-VM port group basis (which is more secure), where the vSwitch is left at the default setting (Reject).

Complete these steps in order to disable and then re-enable Promiscuous mode:

  1. Navigate to Edit > Security > Policy Exceptions.

  2. Uncheck the Promiscuous Mode checkbox.

  3. Click OK.

  4. Navigate to Edit > Security > Policy Exceptions.

  5. Check the Promiscuous Mode checkbox.

  6. Select Accept from the drop-down menu.

Related Information

TAC Authored

Contributed by Cisco Engineers

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products