This document describes how to install and configure the Cisco AnyConnect Network Visibility Module (NVM) on an end-user system using AnyConnect 4.7.x or higher as well as how to install and configure the associated Splunk Enterprise components and NVM Collector.
NVM Collector (bundled in a zip file with the Splunk Application)
Cisco recommends that you have knowledge of these topics:
AnyConnect 4.3.x or higher with NVM
AnyConnect APEX license
ASDM 7.5.1 or higher
Familiarity with Splunk Enterprise and how to install Splunk Apps and Add-ons
The information in this document is based on these software and hardware versions:
Cisco AnyConnect Security Mobility Client 4.3.x or later
Cisco AnyConnect Profile Editor
Cisco Adaptive Security Appliance (ASA), version 9.5.2
Cisco Adaptive Security Device Manager (ASDM), version 7.5.1
Splunk Enterprise 6.3 or later
Ubuntu 14.04.3 LTS as a collector device
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
This is a high level overview of deployment in its simplest form. This is an all-in-one configuration running on 64-bit Linux.
This configuration is how most demonstrations are setup and is also useful in a small production deployment.
This is a more comprehensive set of options that are available for deployment. Typically, a production setup is distributed and has several Splunk Enterprise nodes.
The Cisco AnyConnect Network Visibility Module provides a continuous feed of high value endpoint telemetry. NVM empowers organizations to see endpoint and user behavior on their network, collects flows from endpoints both on and off-premise along with valuable contexts like users, applications, devices, locations, and destinations. Splunk Enterprise consumes the telemetry data and provides the analytics capabilities and reports.
This technote is a configuration example for AnyConnect NVM with Splunk Enterprise as part of the new CESA solution.
Cisco Anyconnect Secure Mobility Client - More than VPN
Cisco Anyconnect is a unified agent that delivers multiple security services to protect the enterprise. AnyConnect is most commonly used as an enterprise VPN client, but it also supports additional modules that cater to different aspects of enterprise security. The additional modules enable security features like posture assessment, web security, malware protection, network visibility and more.
This technote is about Network Visibility Module (NVM), which integrates with Cisco Anyconnect to provide administrators the ability to monitor endpoint application usage.
For more information regarding Cisco Anyconnect, refer to:
IPFIX is an IETF protocol to define a standard for exporting IP flow information for various purposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, though not directly compatible. Cisco nvzFlow is a protocol specification based on the IPFIX protocol. By design, IPFIX is an extensible protocol allowing one to define new parameters to convey information. Cisco nvzFlow protocol extends the IPFIX standard and defines new Information Elements as well as defines a standard set of IPFIX templates that are conveyed as part of the telemetry used by AnyConnect NVM.
A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk.
Cisco provides a collector specifically designed for the nvzFlow protocol and bundled with the Splunk App.
Splunk Enterprise is a powerful tool that collects and analyses diagnostic data to give meaningful information about the IT infrastructure. It provides a one-stop location for administrators to collect data that is crucial in understanding the health of the network.
Splunk is a partner of Cisco's and the CESA solution was created in collaboration with them.
IP address conventions in this technote :
Collector IP address: 192.0.2.123
Splunk IP address: 192.0.2.113
This section covers the configuration of Cisco NVM components.
Anyconnect NVM Client Profile
Anyconnect NVM configuration is saved in an XML file that contains information about the collector IP address and port number, along with other information. The collector IP address and a port number need to be correctly configured on the NVM client profile.
For correct operation of the NVM module, the XML file is required to be placed in this directory:
For Windows 7 and later: %ALLUSERSPROFILE%\Cisco\Cisco AnyConnect Secure Mobility Client\NVM
For Mac OSX: /opt/cisco/anyconnect/nvm
If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed along with Anyconnect NVM deployment.
3. Give the profile a name. In Profile Usage, select Network Visibility Service Profile.
4. Assign it to the group-policy being used by Anyconnect users and click on OK, as shown in the image.
5. The new policy is created, click on Edit, as shown in the image.
6. Enter the information regarding the Collector IP address and port number and click on OK.
7. Now click on Apply, as shown in the image.
Configure NVM Client Profile via Anyconnect Profile Editor
This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM is being deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to Cisco ISE, or copied directly to endpoints.
For detailed information on Anyconnect Profile Editor, refer to:
AnyConnect NVM sends flow information only when it is on a Trusted Network. It uses the TND feature of AnyConnect client to learn if the endpoint is in a trusted network or not.
Trusted Network Detection is configured in the AnyConnect Client Profile (XML) used for VPN regardless of whether the VPN component is being used in the environment or not. TND is enabled by configuring the Automatic VPN Policy section in the profile. At a minimum, a single Trusted DNS Domain or Trusted DNS Server must be populated. The actions taken by AnyConnect when the client has determined that it is on a Trusted Network can be set to DoNothing mode using the pull-down for the Trusted and Untrusted Network Policy.
For additional details on TND configuration, refer to:
Deploy Anyconnect NVM solution involves these steps:
1. Configure Anyconnect NVM on Cisco ASA/ISE.
2. Set up IPFIX Collector component.
3. Set up Splunk with Cisco NVM App and Add-On.
Step 1. Configure Anyconnect NVM on Cisco ASA/ISE.
This step has been covered in detail in the Configure section.
Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.
Step 2. Set up IPFIX Collector Component.
The Collector Component is responsible for collecting and translating all IPFIX data from the endpoints and forwarding it to the Splunk Add-On. The NVM collector runs on 64-bit Linux. CentOS, Ubuntu and Docker configuration scripts are included. The CentOS install scripts and configuration files can also be used in Fedora and Redhat distributions as well.
In a typical distributed Splunk Enterprise deployment, the collector should be run on either a standalone 64-bit Linux system or aSplunk Forwarder node running on 64-bit Linux.
Note: The solution can also be run on a single 64-bit Linux system that includes the NVM collector and Splunk Enterprise components for use in a small deployment or for demonstration purposes.
In order to install the collector, you need to copy the application in theacnvmcollector.zipfile, located in the $APP_DIR$/appserver/addon/ directory to the system you plan to install it on. Extract the files on the system where you plan to install the collector on and execute the install.sh script with superuser privileges. It is recommended to read the $PLATFORM$_README file in the .zip bundle before it executes the install.sh script. The $PLATFORM$_README file provides information on the relevant configuration settings that need to be verified and modified (if necessary) before the install.sh script is executed. At a minimum, you need to configure the address of the Splunk instance that you forward data to. Failing to properly configure the system can cause the collector to operate incorrectly.
Note: Ensure that network and host firewalls are properly configured to allow the UDP traffic for the source and destination addresses and ports
A single NVM collector instance can handle a minimum of 5000 flows per second on a properly sized system. The collector needs to be configured and running before the Splunk App can be used.
By default, the collector receives flows from AnyConnect NVM endpoints on UDP port 2055.
Additionally, the collector produces three data feeds for Splunk,Per Flow Data, Endpoint Identity Data, andEndpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively.
The receive and data feed ports can be changed by altering the acnvm.conf file and restarting the collector instance. Make sure that any host/network firewalls between endpoints and the collector or between the collector and Splunk system(s) are open for the configured UDP ports and addresses. Also, ensure that your AnyConnect NVM configuration matches your collector configuration.
Once all components are installed and running, refer to the Help files section from within the Splunk application for detailed information about the pre-configured reports, data model and information elements that are created by the solution.
You may want to restart one of your AnyConnect endpoints and validate that data is being sent to the solution.
The information needs to be configured in the configuration file (acnvm.conf):
1. The IP address and listening port of Splunk instance.
2. Listening port for the collector (incoming IPFIX data).
Per Flow Data Port, Endpoint Identity Data Port, Endpoint Interface Data and Collector Port are pre-configured to default settings in the configuration file. Ensure that these values are changed if non-default ports are being used.
This information is added in the configuration file (acnvm.conf):
Step 3. Set up Splunk with Cisco NVM App and Add-On for Splunk.
Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from end points in usable reports and correlates user and endpoint behavior.
Step 1. Navigate to Splunk > Apps and install the tar.gz file downloaded from the Splunkbase or search within the Apps section.
Step 2. Next, you need to install the Add-On following the same process. Confirm that both are installed by viewing Splunk Apps page:
The default configuration receives three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively. (see Step 2)
The Add-On then maps these to Splunk sourcetypes cisco:nvm:flowdata, cisco:nvm:sysdata and cisco:nvm:ifdata.
In order to change default ports, navigate to Splunk > Settings > Data Input > UDP, as shown in the image.
Validate Anyconnect NVM Installation
After successful installation, the Network Visibility Module should be listed in Installed Modules, within in the Information section of Anyconnect Secure Mobility Client.
Also, verify if the nvm service is running on the endpoint and profile is in the required directory.
Validate Collector status as Running
Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflow from the endpoints at all times.
root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status
* acnvmcollector is running
Ensure that Splunk and its relevant services are running. For documentation on troubleshooting Splunk, please refer to their website.
1. IPFIX packets are generated on client endpoints by Anyconnect NVM module.
2. The client endpoints forward IPFIX packets to the Collector IP address.
3. The collector collects the information and forwards it to Splunk.
4. Collector sends traffic to Splunk on two different streams: Per Flow Data and Endpoint Identity Data.
All traffic is UDP based on there is no acknowledgment of traffic.
Default port for traffic:
IPFIX data 2055
Per Flow Data 20519
Endpoint Data 20520
Interface Data 20521
NVM module caches IPFIX data and sends it to a collector when it is in Trusted Network. This can either be when the laptop is connected to the corporate network (on-prem) or when it is connected via VPN.
Basic Steps to Troubleshoot
Ensure network connectivity between client endpoint and collector.
Ensure network connectivity between collector and splunk.
Ensure that NVM is correctly installed on client endpoint.
Apply captures on endpoint to see if IPFIX traffic is being generated.
Apply captures on collector to see if it is recieving IPFIX traffic, and if it is forwarding traffic to Splunk.
Apply captures on Splunk to see if it is recieving traffic.
IPFIX traffic as seen in Wireshark:
Trusted Network Detection (TND)
NVM relies on TND for detecting when the endpoint is within a trusted network. If the TND configuration is incorrect, this will cause issues with NVM.
TND works based on information received via DHCP: domain-name and DNS server. If the DNS server and/or domain-name match the configured values, then the network is deemed to be trusted.
If NVM is not forwarding traffic to the collector, then it could be an issue with TND.
IPFIX flow templates are sent to the collector at the start of the IPFIX communication. These templates help the collector to make sense of the IPFIX data.
The collector also preloads templates to ensure that even if the client has not sent them that the data can be parsed. If a newer version of the client is released with protocol changes, the new templates sent by the client will be used.
A template is sent out under the following conditions:
There is a change in the NVM client profile.
There is a network change event.
The nvmagent service is restarted.
The endpoint is rebooted/restarted.
In rare circumstances, a template may not be found. This can be easily remedied by restarting one of the endpoints.
The issue can be identified by observing no template found in a packet capture on the endpoint, or no templates for flowset in the collector logs.
Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
=================> flowsetid=258 flowsetlen=218
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
Cisco always recommends the latest software version of AnyConnect at the time of use or updating. While choosing the AnyConnect version, please use the latest 4.7.x client or later. This gives the latest enhancements with respect to NVM.
CSCva21660 - Anyconnect NVM Handles/Leak for acnvmagent.exe*32 process (fixed in 4.3 MR1).