Cisco Crosswork Network Controller 7.2.x Administration Guide

PDF

Cisco Crosswork Network Controller 7.2.x Administration Guide

Troubleshooting options and common issues in Crosswork Data Gateway

Updated: August 14, 2026

Want to summarize with AI?

Log in

Lists key troubleshooting options and common operational issues for Crosswork Data Gateway, along with guidance on diagnosing and resolving them.

This section provides information about the troubleshooting options available in Crosswork Data Gateway and outlines common issues that may arise, along with guidance on diagnosing and resolving them.

Use this section to access topics related to troubleshooting options and common issues in Crosswork Data Gateway.

  • Use the troubleshooting options available in the Crosswork Network Controller UI to diagnose and resolve issues with Crosswork Data Gateway instances. Refer to Troubleshooting actions available from Crosswork.

  • Use these procedures to quickly identify and resolve operational disruptions, restore service continuity, and maintain reliable data collection performance. Refer to Troubleshooting common issues.

Troubleshooting actions available from Crosswork

Check Data Gateway connectivity

Verify that you can reach a target destination from a Data Gateway to ensure that network connectivity is available for troubleshooting or validation.

Use the Ping and Traceroute actions provided within the Crosswork Data Gateway Management interface to check connectivity to network destinations.

Before you begin

Enable ping traffic on the network to allow successful ping requests.

Use these steps to check Data Gateway connectivity.

Procedure

1.

Go to Administration > Data Gateway Management > Data gateways in the Crosswork Network Controller interface.
2.

Click the Data Gateway name from which you want to check connectivity.
3.

On the Data Gateway details page, at the top right corner, click Actions and choose one of the options.

  • Ping: Enter values for number of packets and destination address, then click Ping.

  • Traceroute: Enter the destination address and click Traceroute.
4.

If the destination is reachable, Cisco Crosswork displays the results of the Ping or Traceroute test in the same window.

The destination's reachability status and details of the Ping or Traceroute test are displayed, confirming network connectivity.

Download the showtech logs

Download encrypted showtech logs for all collection jobs from a Data Gateway instance.

You may need to retrieve showtech logs from a Data Gateway for troubleshooting or support analysis. This task explains how to use the Cisco Crosswork UI to securely download the showtech logs. The logs are encrypted and requires a passphrase for decryption.

Procedure

1.

Go to Administration > Data Gateway Management > Data gateway instances.
2.

Click the Data Gateway name for which you want to download the service metrics.
3.

In the Data Gateway details page, at the top-right corner, click Actions > Download Showtech.

Figure 2. Download showtech
Data Gateway - Download Showtech
4.

Enter a passphrase.

Note

Ensure that you make a note of this passphrase. This passphrase is used later to decrypt the file.
5.

Click Download Showtech.

The file is downloaded to the default download folder on your system in an encrypted format.
6.

After the download is complete, run this command to decrypt it:

openssl enc -d -aes-256-ctr -pbkdf2 -md sha512 -iter 100000 -in <showtech file> -out <decrypted filename> -pass pass:<password>

Example:

openssl enc -d -aes-256-ctr -pbkdf2 -md sha3-512 -iter 100000 -in show-tech-file.tar.xz.enc -out show-tech-file.tar.xz -pass pass:myPassword
Note

  • Use OpenSSL version 1.1.1i to decrypt the file. To check the OpenSSL version on your system, use the command openssl version.

  • The <showtech file> must have a .tar.xz extension.

  • When referring to the <showtech file> and <decrypted filename>, do not enclose the filenames in quotation marks.

  • To decrypt on a MAC, you need OpenSSL 1.1.1+, as LibreSSL does not support all the necessary switches.

After completing this task, you will have securely downloaded encrypted showtech logs for all collection jobs from the selected Data Gateway instance. You can use your passphrase to decrypt the downloaded file and review logs for troubleshooting or support analysis. The decrypted metrics will be available in your system’s default download folder.

Change the log level of components

This document provides step-by-step instructions for users to change the log level of specific components in a Crosswork Data Gateway through the Cisco Crosswork UI.

Changing the log level allows users to adjust the verbosity of logs generated by individual components, such as collectors and infrastructure services, on a targeted Crosswork Data Gateway. The procedure limits log level changes to the Data Gateway selected by the user and ensures accurate configuration for troubleshooting or monitoring purposes. The instructions are intended for administrators managing data gateways within the Cisco Crosswork platform.

Procedure

1.

Go to Administration > Data Gateway Management > Data gateways.

2.

Click the Data Gateway name where you want to change the log level for collectors of Crosswork Infrastructure services.

3.

On the Crosswork Data Gateway details page, click Actions > Change Log Level in the top right corner.

The Change Log Level window appears, indicating the current log level of each container service.

Figure 3. Change log level
Change Log Level Window
4.

Select the check box of the container service for which you wish to change the log level.

5.

At the top of the table, open the Change Log Level drop-down list and select a log level: Debug, Trace, Warning, Info, or Error.

Note

To reset the log level of all logs to the default log level (Info), click Reset to Default.
6.

Click Save.

After you click Save, a UI message appears indicating that the log level of the component was changed successfully.

Troubleshooting common issues

This section provides information about the common troubleshooting issues that you might face when using Data Gateway.

Use this section to access topics related to common issues in Crosswork Data Gateway.

Troubleshoot Data Gateway not moving from assigned to unassigned state issue

This procedure describes how to remove a Data Gateway from an HA pool when it remains in the Assigned state. Use these steps to ensure the Data Gateway can be safely removed while maintaining high availability and proper pool operation.

On the Create Pool page, within the Add Data Gateway instance to pool pane, some Data Gateways in the Assigned state cannot be moved to the Unassigned state, even if they do not have any devices attached. This situation typically means the Data Gateway has a virtual IP assigned, which prevents its removal from the HA pool using standard actions.

Before you begin

Use these steps to remove a Data Gateway out of the HA pool while it is in the Assigned state.

Procedure

1.

Add an additional Data Gateway to the HA pool only if a spare is not already present.
2.

Perform a manual failover to make the assigned Data Gateway a spare.
3.

Update the HA pool to reduce the spare count, then move the spare Data Gateway out of the pool.

What to do next

Workaround: If there is an issue with manual failover in step 2 and the Data Gateway cannot be converted as spare, delete the HA pool, and re-create the pool with a different Data Gateway.

Resolve incorrect NLB health report for active Data Gateway

This procedure helps you identify and resolve incorrect Network Load Balancer (NLB) health reports for an active Crosswork Data Gateway

During the pool creation, Crosswork Data Gateway opens a health port for Network Load Balancer (NLB) to indicate Crosswork Data Gateway’s health status. However, if the NLB FQDN resolves to IP addresses that are on different subnets of eth2 then Crosswork Data Gateway adds a static route to VM. The inclusion of the static route may fail with an error due to network configuration issues. Crosswork Data Gateway disregards the failure and creates the HA pool. As a consequence, Crosswork Data Gateway does not collect any data from the device.

Procedure

1.

Log in to the system identified as NLB and view the health status of the Crosswork Data Gateway.
2.

If status is unhealthy, verify if the NLB subnet address conflicts with the interfaces such as eth1 or eth0. To resolve the conflict, perform one of the following:

  • Modify the NLB IP addresses and restart the Infra services (oam-manager).

  • Redeploy the Crosswork Data Gateway VMs using new subnet configurations.

Recover collection job from degraded state

This task guides administrators on how to recover a collection job when it enters a degraded state, ensuring continued data collection and system reliability

A collection job may enter the Degraded state on the Collection Jobs page, indicating potential issues with service status or system components. By reviewing the service status and identifying the responsible collector, administrators can diagnose the cause of degradation and apply corrective actions. Access to administrative tools and navigation through the Data Gateway Management interface are required to complete these procedures.

Before you begin

Use these steps if the collector is not listed in the Service status section.

Procedure

1.

Go to the main menu on the interactive console and select the Troubleshooting menu.

2.

Select the Remove All Non-Infra Containers and Reboot the VM menu.
3.

When the confirmation message is prompted, click Yes.
4.

If required, check the status of services in the Service status section.

Resolve Data Gateway collection issue after SNMPv3 engine ID update

Describe how to resolve a Crosswork Data Gateway collection issue that occurs after an SNMPv3 engine ID update, including the underlying system behavior and recommended workaround actions to restore appropriate data collection.

When the SNMPv3 engine ID changes or the device experiences downtime or reachability issues, the SNMP collector continues collecting data. The data gateway should pause collection when these changes occur. Data collection continues even when the Force Re-Sync USM Engine Details for SNMPv3 option is disabled.

Workaround: To resolve this issue, enable Force Re-Sync USM Engine Details for SNMPV3 in the Global Parameters window or change the device admin state from DOWN to UP. For more information about enabling the resync option, see Configure the global Data Gateway settings.

Recover LVPN service from monitoring initiated state

This document explains how to recover an LVPN service that is stuck in the monitoring initiated state. It describes the cause of the issue, when the device fails to connect properly to the Data Gateway, and outlines steps to resume data collection by detaching and reattaching devices through Crosswork Data Gateway.

If the device cannot establish a connection with Data Gateway, the gNMI collection job fails with an error. The L2VPN Point to Point service is then unable to monitor the devices, and the status in the Crosswork UI shows Monitoring initiated.

Workaround: To resume data collection, detach, and then reattach the devices using Crosswork Data Gateway.

For more information, see:

Resolve missing IPv6 address and port details in error message

Help users identify and resolve cases where IPv6 address and port details are missing or displayed in a combined format within device error messages on the Crosswork Network Controller.

You can check the status summary of devices on the Crosswork Network Controller UI by navigating to Device Management > Network Devices.

If a device is in the error state, you can see more details by hovering over the information icon next to the state in the Operational state column.

Workaround: When troubleshooting devices with an IPv6 address, the message displays the address and port number in this format: 2001:420:284:2004:4:112:165:636:22, where the address and port numbers are combined.

In these cases, the first block indicates the address followed by the port number. For example, [2001:420:284:2004:4:112:165:636] is the address, and 22 is the port number. If the IP address contains only eight segments, the port number is unavailable.

Handle DAD error in Data Gateway failover process

Resolve a persistent Duplicate Address Detection (DAD) error that may occur during the failover process between Data Gateway instances. The steps ensure that the Data Gateway transitions to the UP state by clearing the DAD error when automatic resolution does not occur in the expected timeframe.

During a Data Gateway failover, the secondary Data Gateway inherits the southbound IPv6 address that was previously assigned to the primary Data Gateway. This inheritance can cause the operating system to register a DAD error, as the address was initially tied to the primary instance. Crosswork detects this condition, logs a DAD failure event, and, under normal circumstances, the error self-resolves within approximately 5 minutes. If the DAD error persists beyond this period, manual intervention is required to clear the DAD flag and bring the Data Gateway back to the UP state.

Note

This behavior is expected and usually resolves within 5 minutes.

Once the DAD failure status is cleared by the operating system, Crosswork automatically transitions the Data Gateway to the UP state.

Before you begin

Workaround: Use these steps if the DAD failure error persists for more than 5 minutes.

Procedure

Remove the southbound VIP address from the secondary Data Gateway and reassign it using these commands.

  1. Delete the VIP address.

    ip address del {southbound_ip}/{mask} dev eth2

  2. Replace the VIP address.

    ip address replace {southbound_ip}/{mask} dev eth2

Resolve Data Gateway failover issues

Provide guidance on resolving Data Gateway failover issues by outlining necessary steps to reattempt failover and ensure standby instances are in the correct operational state.

Workaround: If the failover is not complete due to some issue, reattempt the failover after confirming you have at least one standby instance in the NOT_READY state.

Wait 10 to 30 seconds for the standby data gateway to move to the NOT_READY state before initiating a subsequent failover. If the standby instance remains in the UP state after 30 seconds, restart the oam-manager of the data gateway. This action restores the operational state to NOT_READY.