This document describes how to install and configure the Cisco AnyConnect Network Visibility Module (NVM) on an end-user system using AnyConnect 4.7.x or higher as well as how to install and configure the associated Splunk Enterprise components and NVM Collector.
For a complete overview POV of CESA with Splunk please reference cs.co/cesa-pov
NVM Collector (bundled in a zip file with the NVM TA Add-on)
Cisco recommends that you have knowledge of these topics:
AnyConnect 4.7.x or higher with NVM
ASDM 7.5.1 or higher
Familiarity with Splunk Enterprise and how to install Splunk Apps and Add-ons
The information in this document is based on these software and hardware versions:
Cisco AnyConnect Security Mobility Client 4.7.x or later
Cisco AnyConnect Profile Editor
Cisco Adaptive Security Appliance (ASA), version 9.5.2
Cisco Adaptive Security Device Manager (ASDM), version 7.5.1
Splunk Enterprise 7.x or later (installed as all-in-one on any supported linux, CentOS preferred)
Any supported linux install as a collector device (collector can run on the same server as well, refer cs.co/cesa-pov for more information)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
This is a high-level overview of deployment in its simplest form. This is an all-in-one configuration running on 64-bit Linux.
This configuration is how most demonstrations are set up and is also useful in a small production deployment.
This is a more comprehensive set of options that are available for deployment. Typically, a production setup is distributed and has several Splunk Enterprise nodes.
The Cisco AnyConnect Network Visibility Module provides a continuous feed of high-value endpoint telemetry. NVM empowers organizations to see endpoint and user behavior on their network, collects flows from endpoints both on and off-premise along with valuable contexts like users, applications, devices, locations, and destinations. Splunk Enterprise consumes the telemetry data and provides the analytics capabilities and reports.
This technote is a configuration example for AnyConnect NVM with Splunk Enterprise as part of the new CESA solution.
Cisco Anyconnect Secure Mobility Client - More than VPN
Cisco Anyconnect is a unified agent that delivers multiple security services to protect the enterprise. AnyConnect is most commonly used as an enterprise VPN client, but it also supports additional modules that cater to different aspects of enterprise security. The additional modules enable security features like posture assessment, web security, malware protection, network visibility, and more.
This technote is about Network Visibility Module (NVM), which integrates with Cisco Anyconnect to provide administrators the ability to monitor endpoint application usage.
IPFIX is an IETF protocol to define a standard for exporting IP flow information for various purposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, though not directly compatible. Cisco nvzFlow is a protocol specification based on the IPFIX protocol. By design, IPFIX is an extensible protocol allowing one to define new parameters to convey information. Cisco nvzFlow protocol extends the IPFIX standard and defines new Information Elements as well as defines a standard set of IPFIX templates that are conveyed as part of the telemetry used by AnyConnect NVM.
A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk.
Cisco provides a collector specifically designed for the nvzFlow protocol and bundled with the Splunk App (NVM TA Add-On).
Splunk Enterprise is a powerful tool that collects and analyses diagnostic data to give meaningful information about the IT infrastructure. It provides a one-stop location for administrators to collect data that is crucial in understanding the health of the network.
Splunk is a partner of Cisco's and the CESA solution was created in collaboration with them.
IP address conventions in this technote :
Collector IP address: 192.0.2.123
Splunk IP address: 192.0.2.113
This section covers the configuration of Cisco NVM components.
NVM can now be configured to send data securely to the collector, over DTLS. This mode is configurable in the NVM Profile Editor. When the 'Secure' check box is checked, NVM shall use DTLS as transport. For the DTLS connection to go through, the DTLS server (collector) certificate should be trusted by the endpoint. Untrusted certificates are silently rejected. DTLS 1.2 is the minimum supported version. The collector as part of CESA Splunk App v3.1.2+ is required for DTLS support. The collector will only work in one mode, secure or unsecure.
Collector certificate needs to be trusted by the client (need to make sure the certificate chain is trusted), there is no configuration on Anyconnect.
The certificate needs to be in PEM format.
Doesn't support certificate key password (Cisco ISE internal CA requires one)
Any certificate may be used on the collector as long as the Anyconnect client machine trusts it (internal PKI, well-known, etc).
After the configuration file is updated will need to restart anyconnect NVM service (for single client testing). For profiles pushed from ISE/ASA will need to disconnect/reconnect to the network.
AC NVM Profile collector config should be IP or FQDN. This depends on what is used in the CN of the certificate. FQDN is always preferred in case of IP address changes. If you use an IP address then the collector certificate CN or SAN should have that IP. If you have FQDN as CN in the certificate then the NVM profile should have the same FQDN as a collector.
Anyconnect config (4.9.3043 and higher) - see collector info
NVM profile there is a new checkbox under collector IP/port called Secure.
For those who do not have an AnyConnect deployment or are using another VPN solution, you can install the NVM standalone package for your NVM needs. This package works independently but provides the same level of flow collection from an endpoint as the existing AnyConnect NVM solution. If you install the standalone NVM, the active processes (such as the Activity Monitor on macOS) indicate the use.
Standalone NVM is configured with theNVM Profile Editor, and Trusted Network Detection (TND) configuration is mandatory. Using the TND configuration, NVM determines if the endpoint is on the corporate network and then applies the appropriate policies.
Troubleshooting and logging are still done by AnyConnect DART, which can be installed from the AnyConnect package.
Prior to the stand-alone, you had to have the Core VPN module installed to take advantage of Trusted Network Detection which also resulted in the user seeing the core VPN tile in the UI which may confuse the end users especially if they use another vendors VPN solution.
When using the stand-alone you do not use the core VPN profile to configure TND. The NVM profile can now be configured directly for TND.
Anyconnect NVM Client Profile
Anyconnect NVM configuration is saved in an XML file that contains information about the collector IP address and port number, along with other information. The collector IP address and a port number need to be correctly configured on the NVM client profile.
For correct operation of the NVM module, the XML file is required to be placed in this directory:
For Windows 7 and later: %ALLUSERSPROFILE%\Cisco\Cisco AnyConnect Secure Mobility Client\NVM
For Mac OSX: /opt/cisco/anyconnect/nvm
If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed along with Anyconnect NVM deployment.
This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM is being deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to Cisco ISE, or copied directly to endpoints.
For detailed information on Anyconnect Profile Editor, refer to:
AnyConnect NVM sends flow information only when it is on a Trusted Network. It uses the TND feature of AnyConnect client to learn if the endpoint is in a trusted network or not.
Trusted Network Detection is configured in the AnyConnect Client Profile (XML) used for VPN regardless of whether the VPN component is being used in the environment or not. TND is enabled by configuring the Automatic VPN Policy section in the profile. At a minimum, a single Trusted DNS Domain or Trusted DNS Server must be populated. The actions taken by AnyConnect when the client has determined that it is on a Trusted Network can be set to DoNothing mode using the pull-down for the Trusted and Untrusted Network Policy.
For additional details on TND configuration, refer to:
Deploy Anyconnect NVM solution involves these steps:
1. Configure Anyconnect NVM on Cisco ASA/ISE.
2. Set up IPFIX Collector component (Anyconnect NVM on Linux).
3. Set up Splunk with Cisco NVM App and Add-On.
Step 1. Configure Anyconnect NVM on Cisco ASA/ISE.
This step has been covered in detail in the Configure section.
Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.
Step 2. Set up IPFIX Collector Component (Anyconnect NVM).
The Collector Component is responsible for collecting and translating all IPFIX data from the endpoints and forwarding it to the Splunk Add-On. The NVM collector runs on 64-bit Linux. CentOS, Ubuntu and Docker configuration scripts are included. The CentOS install scripts and configuration files can also be used in Fedora and Redhat distributions as well.
In a typical distributed Splunk Enterprise deployment, the collector should be run on either a standalone 64-bit Linux system or aSplunk Forwarder node running on 64-bit Linux.
Note: The solution can also be run on a single 64-bit Linux system that includes the NVM collector and Splunk Enterprise components for use in a small deployment or for demonstration purposes. The all-in-one is easiest for up to 10k endpoints - see CESA POV sizing information,
How to Install the Collector?
Copy the acnvmcollector.zipfile, located in the /opt/splunk/etc/apps/$APP_DIR$/appserver/addon/ (included with the TA Add-On) directory to the system you plan to install it on.
Extract the files (unzip acnvmcollector.zip)
It is recommended to read the $PLATFORM$_README file in the .zip bundle before it executes the install.sh script. The $PLATFORM$_README file provides information on the relevant configuration settings that need to be verified and modified (if necessary) before the install.sh script is executed. At a minimum, you need to configure the address of the Splunk instance that you forward data to. Failing to properly configure the system can cause the collector to operate incorrectly.
Note: Ensure that network and host firewalls are properly configured to allow the UDP traffic for the source and destination addresses and ports. The IPFIX (cflow) traffic coming in from the anyconnect clients to the collector and the outgoing UDP data to Splunk (here).
A single NVM collector instance can handle a minimum of 5000 flows per second on a properly sized system. The collector needs to be configured and running before the Splunk App can be used.
By default, the collector receives flows from AnyConnect NVM endpoints on UDP port 2055.
Additionally, the collector produces three data feeds for Splunk,Per Flow Data, Endpoint Identity Data, andEndpoint Interface Data, on UDP ports 20519, 20520, and 20521 respectively.
The receive and data feed ports can be changed by altering the acnvm.conf file and restarting the collector instance. Make sure that any host/network firewalls between endpoints and the collector or between the collector and Splunk system(s) are open for the configured UDP ports and addresses. Also, ensure that your AnyConnect NVM configuration matches your collector configuration.
Once all components are installed and running, refer to the Help files section from within the Splunk application for detailed information about the pre-configured reports, data model and information elements that are created by the solution.
You may want to restart one of your AnyConnect endpoints and validate that data is being sent to the solution. Run a steady stream of data by using youtube.
The information needs to be configured in the configuration file - acnvm.conf
syslog_server_ip (forwarder or splunk instance) can point to 127.0.0.1 (don't use LOCALHOST) if it is on the same box
Listening port for the collector (incoming IPFIX data) default is ok.
NOTE: netflow_collector_ip is omitted from the config file (it uses default public interface), it should only be changed to override with a specific local IP
Per Flow Data Port, Endpoint Identity Data Port, Endpoint Interface Data, and Collector Port are pre-configured to default settings in the configuration file. Ensure that these values are changed if non-default ports are being used.
This information is added in the configuration file - /opt/acnvm.conf
(see Anyconnect NVM DTLS Information for more details)
This is done on the box hosting the collector.
Make directory /opt/acnvm/certs.
In order to apply the certificate to the collector save it along with the key in the /opt/acnvm/certs directory
change the owner and group of the folder to acnvm:acnvm with the following command: sudo chown -R acnvm:acnvm certs/:
This section below for acnvm.conf needs to be configured with the cert and key
After the config and cert are placed, restart the collector - sudo systemctl restart acnvm.service
Check collector status - sudo systemctl status acnvm.service
Step 3. Set up Splunk with Cisco NVM App (CESA Dashboard) and TA Add-On for Splunk.
Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from endpoints in usable reports and correlates user and endpoint behavior.
Note: For cloud deployments, both apps are installed in the cloud instance. Only the TA is installed on-premise (with the forwarder). The collector is installed on-premise with the forwarder or on a separate linux/docker box.
For on-premise, only you can install all components and apps on one box (or separate) see diagrams
Step 1. Navigate to Splunk > Apps , click on the gear and install the tar.gz file downloaded from the Splunkbase or search within the Apps section.
Step 2. Next, you need to install the Add-On following the same process. Confirm that both are installed by viewing Splunk Apps page:
The default configuration receives three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively. (see Step 2)
The Add-On then maps these to Splunk sourcetypes cisco:nvm:flowdata, cisco:nvm:sysdata and cisco:nvm:ifdata.
Enable UDP Inputs using Splunk Management UI
Note: You can also do this using an input.conf file, this is explained in the Cisco NVM Dashboard app gui in the help pulldown
You do not need to restart Splunk software.
Navigate to Splunk > Settings > Data Input > UDP, as shown in the image.
1. Click New Local UDP> Enter port # missing> Click Next > Select Corresponding Source Type> Click Review > Click Submit
2. Repeat for the other 2 ports (try using clone)
Validate Anyconnect NVM Installation
After successful installation, the Network Visibility Module should be listed in Installed Modules, within the Information section of Anyconnect Secure Mobility Client.
Also, verify if the nvm service is running on the endpoint and the profile is in the required directory.
Validate Collector status as Running
Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflow from the endpoints at all times. If it is not running then ensure that the acnvm account permissions for the file allow it to execute: /opt/acnvm/bin/acnvmcollector
root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status
* acnvmcollector is running
Validate Splunk - Anyconnect NVM CESA Dashboard
Ensure that Splunk and its relevant services are running. For documentation on troubleshooting Splunk, please refer to their website.
Not the dashboards for CESA won't be updated until 5 minutes after initial data is received due to an automation script. Run a manual search to validate right away:
From main dashboard, click on "Search & Reporting". In the next screen, set the correct range in order to populate the desire data then where it says "enter search here..." enter "sourcetype="cisco:nvm:flowdata"
Check the Splunk Dashboard to make sure Go to Splunk, click on Cisco NVM Dashboard, click on Device Activity by Volume and Flow Count if you want to keep the current settings, and click on Submit. It displays data in the graphics.
1. IPFIX packets are generated on client endpoints by Anyconnect NVM module.
2. The client endpoints forward IPFIX packets to the Collector IP address.
3. The collector collects the information and forwards it to Splunk.
4. Collector sends traffic to Splunk on three different streams: Per Flow Data, Endpoint Data and Interface Data.
All traffic is UDP based on there is no acknowledgment of traffic.
Default port for traffic:
IPFIX data 2055
Per Flow Data 20519
Endpoint Data 20520
Interface Data 20521
NVM module caches IPFIX data and sends it to a collector when it is in Trusted Network. This can either be when the laptop is connected to the corporate network (on-prem) or when it is connected via VPN.
You can validate the collector is receiving packets from NVM module by running a packet capture on specific UDP ports, as per your configuration to check if the packets are being received. This would be done via the Splunk system linux OS.
IPFIX flow templates are sent to the collector at the start of the IPFIX communication. These templates help the collector to make sense of the IPFIX data.
The collector also preloads templates to ensure that even if the client has not sent them that the data can be parsed. If a newer version of the client is released with protocol changes, the new templates sent by the client will be used.
A template is sent out under these conditions:
There is a change in the NVM client profile.
There is a network change event.
The nvmagent service is restarted.
The endpoint is rebooted/restarted.
Periodically (default=24 hours) as configured in the NVM Profile.
In rare circumstances, a template may not be found. This can be easily remedied by restarting one of the endpoints.
The issue can be identified by observing no template found in a packet capture on the endpoint, or no templates for flowset in the collector logs.
These are the basic steps to troubleshoot:
Ensure network connectivity between client endpoint and collector.
Ensure network connectivity between collector and splunk.
Ensure that NVM is correctly installed on the client endpoint.
Apply captures on the endpoint to see if IPFIX traffic is being generated.
Apply captures on a collector to see if it receives IPFIX traffic and if it forwards traffic to Splunk.
Apply captures on Splunk to see if it receives traffic.
anyconnect clients trust the collector certificate
NVM profile has secure enabled
collector is configured for certs
IPFIX traffic as seen in Wireshark:
Note: If running DTLS between the client and collector you will need to filter on DTLS traffic
Anyconnect Client (NVM Module)
Anyconnect NVM - Not Reporting to the Collector - CFLOW Data Packets aren't Leaving tje Endpoint
Is the NVM database file growing under C:\%ProgramData%\Cisco\Cisco Anyconnect Secure Mobility Client? If it keeps growing that means the logs aren’t being sent from the Client.If you look under the NVM folder you can see the sql database growing, the nvm.db is not documented but we talk extensively about how we cache and the controls around caching in the NVM guide. if you see that then it's not sending the data to the collector.
Trusted Network Detection (TND)
Launch the Anyconnect UI and make sure it's on a trusted network. NVM relies on TND for detecting when the endpoint is within a trusted network. If the TND configuration is incorrect, this will cause issues with NVM. NVM has its own TND configuration, which works on the TLS certificate fingerprint of the configured server. This can be configured in the NVM Profile Editor.
If NVM TND is not configured, NVM relies on the VPN module's TND configuration. VPN's TND works based on information received via DHCP: domain-name and DNS server. If the DNS server and/or domain-name match the configured values, then the network is deemed to be trusted. VPN also supports TLS certificate-based TND detection.
Ensure the Trusted Network Detection configuration is correct. NVM exports only when on a trusted network i.e. invalid TND configuration (ex: if you have 3 DNS servers you need 3 defined).
Remove the trusted domain from the TND VPN config
split tunneling (collector’s IP address is not part of the split tunnel trusted so the data is sent out the public interface). Please ensure to include the collector's IP always in the split include configuration for VPN.
Ensure the CollectionMode is configured to collect on the current network (trusted/untrusted).
Make sure the VPN.xml and NVM_ServiceProfile.xml are in the correct folders and restart
Start-stop all anyconnect services
Bounce the network connected to the inside that has a connection to DNS server.
Anyconnect Diagnostic and Reporting Tools (DART)
In order to troubleshoot what Anyconnect is doing run DART on the NVM components.
All logs needed for NVM are handled by DART, it collects log files, configuration, etc.
Windows logs - Events aren’t in a single place, there is a separate leaf in the event viewer for NVM under AnyConnect.
macOS/linux - filter logs by nvmagent
Collector (On Linux/Docker machine - all-in-one or standalone)
acnvmcollector is failing to start:
This was an issue on Ubuntu. I noticed that the code failed to execute on the acnvmcollector file: /opt/acnvm/bin/acnvmcollector.
Make sure the permissions include execute for the acnvm account.
How can i obtiain verions of the collector?
Where can i set the debug?
You can set the logging level in the ACMNVMLOG.conf - Its part of the configuration sent to the collector startup. After a change restart the collector.
log4cplus.rootLogger=DEBUG, STDOUT, NvmFileAppender < this is in ACNVMLOG.conf file
Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:
=================> flowsetid=258 flowsetlen=218
Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates
for flowset 258 for 10.150.176.167 yet
DTLS not configured (means wasn't seen in the acnvm.conf file)
The server key is invalid (this was a password key combo that isn't supported)
Splunk Console (NVM Dashboard) is not showing data
Generate data using youtube and maybe browsing to a few website
Can anyconnect client send information via UDP 2055 to the collector server (are there any firewalls in between?)
try telnet from client machine to collector machine
Run wireshark to make sure client is sending traffic (2055 cflow) data to the collector
Validate its receiving anyconnect NVM traffic
Do a tcpdump (and make sure you see packets from client to server on 25001 to 2055)
Sudo tcpdump -I any -c100 -nn host 10.1.110.7 (this will see first 100 packets coming from client host IP)
Where do you Store the Certificate for Anyconnect NVM DTLS?
This would be for lab testing where you don't have a well-known cert installed on the collector.
Install the collector certificate into the Windows Trusted Certificates
For installing the root certificate, the process is standard and well defined for macOS which is via keychain, we can use Keychain tool to import and add as trusted.
Linux - different for each distro(Ubuntu and RHEL).
RHEL Root CA IMPORT STEPS : 1) Copy the ca cert to /etc/pki/ca-trust/source/anchors 2) sudo update-ca-trust enable 3) Finally, sudo update-ca-trust extract
Ubuntu Root CA IMPORT STEPS :
1) Convert .cer file to .crt file openssl x509 -inform PEM -in RootCA.cer -out rootCa.crt 2) Copy the .crt file to /usr/local/share/ca-certificates 3) Run the command sudo update-ca-certificates
XML File Names
When using the local profile editor. The core VPN module XML profile name doesn't matter. “Save the profile as NVM_ServiceProfile.xml. You must save the profile with this exact name or NVM fails to collect and send data.”
Can vendor directory be created under root and then the ownership provided to another account?
You can create /opt/acnvm first as long as the install script has permission to copy files into it
File permissions - install.sh need permissions to execute as root
Why useradd -r, and why -s /bin/false because its a non-interactive account without a home directory
It is not a requirement for a home directory and its standard practice for the service account to not have one to keep clean
All users have uid/guid whether or not they have a home directory.
Collector OS - runs CentOS, Ubuntu, Redhat can use CentOS script.
Install script - can be modified if needed.
Needs to be run as root or with SUDO right as it creates a new user called acnvm and places everything into /opt/acnvm directory.
General note: You can also create your own script to do what you need to do as per your requirements. This script could use a different user you already had running on the system but this user would need to have SUDO rights to run the install.
To find the collector version run with -v flag
Cisco always recommends the latest software version of AnyConnect at the time of use or update. While you choose the AnyConnect version, please use the latest 4.9.x client or later. This gives the latest enhancements with respect to NVM.
AnyConnect 4.9.00086 New Features
This is a major release that includes the following features and supports updates, and that resolves the defects described in AnyConnect 4.9.00086
NVM expansion to enrich flow and endpoint data, including new NVM Collector, coordinated with Splunk app 3.x and a timestamp for flow information