The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This section provides several SD-AVC troubleshooting scenarios. If this information does not provide a solution, contact Cisco TAC for assistance.
The following tables describe troubleshooting for issues with:
SD-AVC network service
(operates on a dedicated host)
SD-AVC agent
(operates on each participating device in the network)
Connectivity
(between network service and one or more devices in the network)
Problem |
How it appears |
Troubleshooting |
---|---|---|
SD-AVC network service: installation failure |
SD-AVC not active, sd-avc status shows installation failure. |
Summary Diagnose with sd-avc status and then service sd-avc trace. Possible issues:
Troubleshooting Details |
SD-AVC network service: activation failure |
SD-AVC not active, sd-avc status shows activation failure. |
Summary Diagnose with sd-avc status and then service sd-avc trace. Possible issue: Something may be using CPU resources. Ensure that nothing is using CPU resources. Troubleshooting Details |
SD-AVC network service: configuration failure |
SD-AVC not active, sd-avc status shows configuration failure. |
Summary A VRF is attached to the interface used as the management interface on the device hosting the SD-AVC network service. Remove the VRF assignment from the management interface using: interface interface no ip vrf forwarding
Troubleshooting Details |
Problem |
How it appears |
Troubleshooting |
---|---|---|
NBAR2 is not activated on the device |
On the Dashboard > Application Visibility page, the Timeline graph of bandwidth shows no activity. |
Summary NBAR2 is not active: Activate NBAR2 on the device. Troubleshooting Details |
Problem |
How it appears |
Troubleshooting |
---|---|---|
UDP |
Warning in: Dashboard > Application Visibility page > Network Monitoring pane > Connection |
Summary Check UDP connectivity. Troubleshooting Details |
TCP |
Warning in: Dashboard > Application Visibility page > Network Monitoring pane > Update |
Summary Check TCP connectivity. Troubleshooting Details |
FTP |
Warning in: Dashboard > Application Visibility page > Network Monitoring pane > Update |
Summary
Troubleshooting Details |
Problem |
How it appears |
Troubleshooting |
---|---|---|
Failure to load Protocol Pack on a device |
When deploying Protocol Packs to one or more devices, results page shows error, such as "out of sync." |
Summary Load the Protocol Pack manually on the device to determine whether the Protocol Pack is valid. Troubleshooting Details |
The following commands are helpful for troubleshooting SD-AVC network service issues. Execute the commands on the network service host device. The output may indicate any installation or configuration problems.
Command |
Description |
---|---|
service sd-avc status |
Status of SD-AVC network service installation, configuration, and activation |
service sd-avc trace |
Memory or disk problems |
show virtual-service list |
Activation errors |
show virtual-service global |
CPU and memory usage |
Execute the command on the network service host device.
Output indicates status of SD-AVC installation, configuration, and activation.
Installation error:
Service SDAVC is uninstalled, not configured and deactivated
Activation error:
Service SDAVC is installed, configured and Activate Failed
Execute the command on the network service host device.
Output indicates memory or disk problems.
Memory problem (shown in bold below):
service sd-avc trace 2017/11/27 02:06:42.384 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MACH_PARSE_FAILURE: Virtual Service[SDAVC]::Parsing::XML parsing failure::Unable to parse VM machin e definition::Requests 3072 MB of memory which exceeds the maximum of 1024 2017/11/27 02:06:42.383 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MEMORY_LIMIT_WARN: Virtual service (SDAVC) defines 3072 MB of Memory exceeding the maximum 1024 MB. ...
Disk problem (shown in bold below):
2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM mac address binding from FDB 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get mac binding from persistent DB file 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Could not retrieve HA disk info for VM 'SDAVC' 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Unable to locate fdb attributes for vm(SDAVC) 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM storage info list from FDB 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get storage pool from persistent DB file 2017/11/27 03:36:52.499 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Install::The installation of the virtual service failed
Execute the command on the network service host device.
Output indicates activation status (failed in this example):
Virtual Service List: Name Status Package Name ------------------------------------------------------------------------------ SDAVC Activate Failed avc_iosxe_221533.ova
Execute the command on the network service host device.
Output indicates virtual service CPU and memory usage:
Example showing a service using 5% of CPU:
show virtual-service global Maximum VCPUs per virtual service : 1 Resource virtualization limits: Name Quota Committed Available -------------------------------------------------------------- system CPU (%) 75 5 70 memory (MB) 3072 800 2272 bootflash (MB) 20000 6764 10672
Device hosting the SD-AVC network service
Memory or disk allocation issues can prevent successful installation of the SD-AVC network service.
Use service sd-avc status on the network service host device to check status of installation. If installation is unsuccessful, the output shows "Service SDAVC is uninstalled."
service sd-avc status
Service SDAVC is uninstalled, not configured and deactivated
Use service sd-avc trace on the network service host device to indicate whether the installation problem is due to memory or disk.
Memory problem:
service sd-avc trace 2017/11/27 02:06:42.384 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MACH_PARSE_FAILURE: Virtual Service[SDAVC]::Parsing::XML parsing failure::Unable to parse VM machin e definition::Requests 3072 MB of memory which exceeds the maximum of 1024 2017/11/27 02:06:42.383 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MEMORY_LIMIT_WARN: Virtual service (SDAVC) defines 3072 MB of Memory exceeding the maximum 1024 MB. ...
Disk problem:
2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM mac address binding from FDB 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get mac binding from persistent DB file 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Could not retrieve HA disk info for VM 'SDAVC' 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Unable to locate fdb attributes for vm(SDAVC) 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM storage info list from FDB 2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get storage pool from persistent DB file 2017/11/27 03:36:52.499 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Install::The installation of the virtual service failed
Problem |
Solution |
---|---|
Memory error |
Increase the device memory to the amount specified in System Requirements: SD-AVC Network Service Host. |
Disk error |
Increase the size of the harddisk or bootflash (for CSR) device according to the requirements specified in System Requirements: SD-AVC Network Service Host. |
Device hosting the SD-AVC network service
The platform hosting the SD-AVC network service should not have other virtual services operating. Sharing CPU resources with other virtual services can prevent successful activation.
Use service sd-avc status on the network service host device to check status of installation. If installation has succeeded, but activation is unsuccessful, the output shows "Activate Failed."
service sd-avc status
Service SDAVC is installed, configured and Activate Failed
Use service sd-avc trace on the network service host device to troubleshoot. The following output shows a problem (shown in bold) with activation, due to shared CPU.
service sd-avc trace
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to find domain SDAVC - state query
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Domain not found: No domain with matching name 'SDAVC'
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Error from libvirt: code=42
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (note): VM (SDAVC) State Transition: next_state: LIFECYCLE_ACTIVATE_FAILED
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Activate::Internal error::Machine definition customization failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Machine definition customization failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Customization of common XML parameters failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Customize CPU tunes: Cannot commit CPU tunes
2017/11/26 15:46:48.131 [errmsg] [2224]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-CPUSHARES_LIMIT: Virtual Service[SDAVC]::CPU shares limit::The virtual service definition exceeds the maximum number of CPU shares::Defined: 75, available: 70
Use show virtual-service global to provide details. In this example, another process is using 5% of the CPU resources (shown in bold).
show virtual-service global
Maximum VCPUs per virtual service : 1
Resource virtualization limits:
Name Quota Committed Available
--------------------------------------------------------------
system CPU (%) 75 5 70
memory (MB) 3072 800 2272
bootflash (MB) 20000 6764 10672
Deactivate Interface Using CPU Resources
Check the running configuration using show run on the network service host device. If an active interface is using CPU resources, deactivate the interface.
Example
GigabitEthernet1 is using CPU resources.
show run | section csr_mgmt virtual-service csr_mgmt ip shared host-interface GigabitEthernet1 activate
Deactivate the interface.
conf t virtual-service csr_mgmt no activate no ip shared host-interface GigabitEthernet1
Repeat the installation of the SD-AVC network service.
Device hosting the SD-AVC network service
If the host interface that is used as a gateway interface for the SD-AVC network service is attached to a VRF, the SD-AVC network service installation may be successful, but a configuration step may fail.
Check VRF status of the SD-AVC network service gateway interface.
Example showing a VRF configured on the gateway interface GigabitEthernet1:
interface GigabitEthernet1
ip vrf forwarding Mgt
ip address 10.56.196.177 255.255.252.0
service sd-avc configure gateway interface gigabitEthernet 1 service-ip 10.56.196.180
% Error: VRF 'Mgt' is configured on gateway. This type of configuration is not supported.
Remove the VRF assignment from the management interface. Example:
interface GigabitEthernet1 no ip vrf forwarding
Devices in the network that are using SD-AVC
The NBAR2 component must be active on any interface that processes network traffic, in order to report on traffic handled by the interface. For details, see Configuration Prerequisites: Network Devices Using SD-AVC.
If NBAR2 is not active on an interface processing network traffic:
The device will not report on any traffic on that interface.
On the Dashboard > Application Visibility page, the Timeline graph of bandwidth will show no activity.
The device will not receive application rules packs from the SD-AVC network service.
Verify that NBAR2 is active on interfaces that process network traffic.
If necessary, activate NBAR2 on the interface(s).
SD-AVC network service
Devices in the network that use SD-AVC
The SD-AVC Network Service uses UDP over port 50000 to communicate with the devices that it manages.
If a Connection warning appears in the SD-AVC Dashboard, for a specific device in the network, check connectivity on UDP port 50000. Warnings appear here:
Dashboard > Application Visibility page > Network Monitoring pane > Connection
If no problem is found, contact Cisco TAC.
Ensure that UDP connectivity is possible on port 50000 between the affected device and the SD-AVC network service.
SD-AVC network service
Devices in the network that use SD-AVC
The SD-AVC network service uses TCP over ports 20-21 (FTP) to communicate with the devices that it manages.
If an Update warning appears in the SD-AVC Dashboard, for a specific device in the network, check connectivity on TCP ports 20-21. Warnings appear here:
Dashboard > Application Visibility page > Network Monitoring pane > Update
If no problem is found, contact Cisco TAC.
Ensure that TCP communication is possible over ports 20-21 (FTP) between the affected device and the SD-AVC network service.
SD-AVC network service
Devices in the network that use SD-AVC
The SD-AVC network service uses FTP to communicate with the devices that it manages.
A device with partial connectivity, but problems specific to FTP may show up as a warning in the SD-AVC Dashboard. An Update warning may appear while the Connection status is green.
If an Update warning appears in the SD-AVC Dashboard while the Connection status is green, for a specific device in the network, check the FTP connection status. Warnings appear here:
Dashboard > Application Visibility page > Network Monitoring pane
On the device with the connectivity issue, use show ip nbar classification cache sync summary to check the FTP connection status. "Connection: DISCONNECTED" in the output below shows an FTP connectivity problem.
show ip nbar classification cache sync summary Connection Status: Connection: DISCONNECTED Last disconnection: Never Mode : Standalone connectivityTimeout (sec) : 300 connectivityCheckInterval (sec) : 60 connectivityCheckInterval was changed: FALSE Active controller: Type : Primary IP : 10.56.196.232 Status: Disconnected Last connection: Never bypass : FALSE force down: FALSE
Ensure that FTP communication is possible between the affected device and the SD-AVC network service.
Verify that nothing is preventing FTP network connectivity between the SD-AVC network service and the network device. This includes checking ACL, firewalls, and so on.
On the device with the Update warning, ensure that FTP connectivity is possible from the routable interface to the SD-AVC network service. To enable FTP connections from a specific interface, use:
ip ftp source-interface interface-nameExample:
ip ftp source-interface GigabitEthernet1
SD-AVC network service
Cisco NBAR2 Protocol Packs
Use the SD-AVC network service to deploy Protocol Packs to one or more devices. See Deploying Protocol Packs to Devices. When deploying Protocol Packs to one or more devices, if the deployment fails, the results page may show an error.
Load the Protocol Pack manually on the device indicated by the error to verfiy that the Protocol Pack is valid and can be loaded onto the device. This rules out any problems with the Protocol Pack file.
(config)#ip nbar protocol-pack bootflash:pack_file_name.pack
If no problem is found, contact Cisco TAC.