This document describes the method to install and configure the Cisco AnyConnect Network Visibility Module (NVM) on an end-user system using AnyConnect 4.2.x or higher.
The Cisco AnyConnect NVM is used as a medium for deploying security analytics. NVM empowers organizations to see endpoint & user behavior on their network, collects flows from endpoints both on and off-premise along with additional context like users, applications, devices, locations and destinations.
This technote is a configuration example using AnyConnect NVM with Splunk.
Cisco recommends that you have knowledge of these topics:
AnyConnect 4.2.01022 or higher with NVM
AnyConnect APEX license
ASDM 7.5.1 or higher
The information in this document is based on these software and hardware versions:
Cisco AnyConnect Security Mobility Client 4.2 or later
Cisco AnyConnect Profile Editor
Cisco Adaptive Security Appliance (ASA), version 9.5.2
Cisco Adaptive Security Device Manager (ASDM), version 7.5.1
Splunk Enterprise 6.3
Ubuntu 14.04.3 LTS as a collector device
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Cisco Anyconnect Secure Mobility Client
Cisco Anyconnect is a unified agent that delivers multiple security services to protect the enterprise. Anyconnect is most commonly used as an enterprise VPN client, but it also supports additional modules that cater to different aspects of enterprise security. The additional modules enable security features like posture assessment, web security, malware protection, network visibility and more.
This technote is about Network Visibility Module (NVM), which integrates with Cisco Anyconnect to provide administrators the ability to monitor endpoint application usage.
For more information regarding Cisco Anyconnect, refer to:
IPFIX is an IETF protocol to define a standard for exporting IP flow information for various purposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, though not directly compatible.
Cisco vzFlow is a protocol specification extended based on the IPFIX protocol. IPFIX doesn’t have enough standard Information Elements to support all the parameters can be collected as part of AC NVM. Cisco vzFlow protocol extends the IPFIX standard and defines new Information Elements as well as defines a standard set of IPFIX templates that will be used by AC NVM for exporting IPFIX data.
A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk. Eg. Lancope.
Cisco also provide its home-grown IPFIX collector.
Splunk is a powerful tool that collects and analyses diagnostic data to give meaningful information about the IT infrastructure. It provides a one-stop location for administrators to collect data that is crucial in understanding the health of the network.
Splunk is not owned or maintained by Cisco Systems, however Cisco provides Cisco AnyConnect NVM App for Splunk.
For more information regarding Spunk, please visit their website.
IP address conventions in this technote :
Collector IP address: 192.0.2.123
Splunk IP address: 192.0.2.113
This section covers configuration of Cisco NVM components.
Anyconnect NVM client profile
Anyconnect NVM configuration is saved in an XML file that contains information about the collector IP address and port number, along with other information. The collector IP address and port number need to be correctly configured on NVM client profile.
For correct operation of the NVM module, the XML file is required to be placed in this directory:
For Windows 7 and later: %ALLUSERSPROFILE%\Cisco\Cisco AnyConnect Secure Mobility Client\NVM
For Mac OSX: /opt/cisco/anyconnect/nvm
If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed along with Anyconnect NVM deployment.
3. Give the profile a name. In Profile Usage, select Network Visibility Service Profile
4. Assign it to the group-policy being used by Anyconnect users. Click OK.
5. The new policy is created. Click Edit
6. Fill information regarding the Collector IP address and port number. Click OK.
7. Click Apply.
Configure NVM client profile via Anyconnect Profile Editor
This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM is being deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to Cisco ISE, or copied directly to endpoints.
For detailed information on Anyconnect Profile Editor, refer to:
The NVM sends flow information only when it is on a Trusted Network. It uses the TND feature of Anyconnect client to learn if the endpoint is in a trusted network. TND uses DNS/domain information to determine if the endpoint is in a trusted network. When VPN is connected, it is considered to be in a trusted network, and flow information is sent to the collector.
TND needs to be correctly configured for correct functioning of NVM. For details on TND configuration, refer to:
Deploying Anyconnect NVM solution involves these steps:
1. Configure Anyconnect NVM on Cisco ASA/ISE
2. Set up IPFIX Collector component
3. Set up Splunk with Cisco NVM App
Step 1. Configure Anyconnect NVM on Cisco ASA/ISE
This step has been covered in detail in the Configure section.
Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.
Step 2. Set up IPFIX Collector component
The Collector Component is responsible for collecting and translating all IPFIX data from the endpoints and forwarding it to the Splunk App. There are various third-party collector tools available, and Cisco NVM is compatible with any collector that understands IPFIX. This technote uses Cisco homegrown collector tool running on 64-bit Linux. CentOS and Ubuntu configuration scripts are included in with the splunk application. The CentOS install scripts and configuration files can also be used in Fedora and Redhat distributions as well. The collector should be run on either a standalone 64-bit Linux system or a Splunk Forwarder running on 64-bit Linux.
In order to install the collector you will need to copy the application in the CiscoNVMCollector_TA.tar file, located in the $APP_DIR$/appserver/addon/ directory to the system you plan to install it on.
Splunk, for this technote, is installed on Windows workstation on the E: drive.
CiscoNVMCollector_TA.tar file can be located in the following directory :
Extract the tar file on the system where you plan to install the collector and execute the install.sh script with super user privileges. It is recommended to read the $PLATFORM$_README file in the .tar bundle before executing the install.sh script. The $PLATFORM$_README file provides information on relevant configuration settings that need to be verified and modified (if necessary) before the install.sh script is executed.
The information needs to be configured in the configuration file (acnvm.conf):
1. IP address and listening port of Splunk instance.
2. Listening port for collector (incoming IPFIX data).
Per Flow Data Port, Endpoint Identity Data Port and Collector Port are pre-configured to default settings in the configuration file. Ensure that these values are changed if non-default ports are being used.
This information is added in the configuration file (acnvm.conf):
Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from end points in usable reports, and correlates user and endpoint behavior.
Navigate to Splunk > Apps and install the tar.gz file downloaded from the Splunkbase or search within the Apps section.
By default, Splunk receives two data input feeds for Per Flow Data and Endpoint Identity Data, on UDP ports 20519 and 20520 respectively. The collector component sends these feeds on these ports by default. The default ports can be changed on the splunk, but the same ports also need to be specifed in the collector configuration (see Step 2)
In order to change default ports, navigate to Splunk > Settings > Data Input > UDP
Validate Anyconnect NVM installation
After successful installation, the Network Visibility Module should be listed in Installed Modules, within in the Information section of Anyconnect Secure Mobility client.
Also, verify if the nvm service is running on the end point and profile is in the required directory.
Validate Collector status as Running
Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflow from the endpoints at all times.
root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status
* acnvmcollector is running
Ensure that Splunk and its relevant services are running. For documentation on troubleshooting Splunk, please refer to their website.
1. IPFIX packets are generated on client endpoints by Anyconnect NVM module.
2. Client endpoints forward IPFIX packets to the Collector IP address
3. Collector collects the information and forwards it to Splunk
4. Collector sends traffic to Splunk on two different streams: Per Flow Data and Endpoint Identity Data
All traffic is UDP based on there is no acknowledgement of traffic.
Default port for traffic:
IPFIX data 2055
Per Flow Data 20519
Per Flow Data 20520
NVM module caches IPFIX data and sends it to collector when it is in Trusted Network. This can either be when the laptop is connected to the corporate network (on-prem) or when it is connected via VPN.
Basic troubleshoot steps
Ensure network connectivity between client endpoint and collector.
Ensure network connectivity between collector and splunk.
Ensure that NVM is correctly installed on client endpoint.
Apply captures on endpoint to see if IPFIX traffic is being generated.
Apply captures on collector to see if it is recieving IPFIX traffic, and if it is forwarding traffic to Splunk.
Apply captures on Splunk to see if it is recieving traffic.
IPFIX traffic as seen in Wireshark:
Trusted Network Detection (TND)
NVM relies on TND for detecting when the endpoint is within trusted network. If the TND configuration is incorrect, this will cause issues with NVM.
TND works based on information received via DHCP: domain-name and DNS server. If the DNS server and/or domain-name match the configured values, then the network is deemed to be trusted.
If NVM is not forwarding traffic to collector, then it could be an issue with TND.
IPFIX flow templates are sent to collector at the start of the IPFIX communication. These templates help the collector to make sense of the IPFIX data. If this information is not sent to the collector, then the collector can not collect the IPFIX data. This causes issues with data collection.
Such issues are seen if the collector is configured later, or if the first few IPFIX packets are dropped in the network (common over VPN). In order to mitigate this, one of the below events should occur:
There is a change in the NVM client profile.
There is a network change event.
The nvmagent service is restarted.
End point is rebooted/restarted.
This issue can be recovered by rebooting the endpoint, or reconnecting VPN.
The issue can be identified by observing no template found in a packet capture on the end point, or no templates for flowset in the collector logs.
Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
=================> flowsetid=258 flowsetlen=218
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
Cisco always recommends the latest software version of AnyConnect at the time of use or updating. While choosing AnyConnect version, please use the latest 4.2.x or 4.3.x client. This will give the latest enhancements with resepect NVM, defect fixes and mitigate recent changes with Microsoft Code Signing Certificates enforcements. More details here.
CSCva21660 - Anyconnect NVM Handles/Leak for acnvmagent.exe*32 process