Troubleshooting Smart Switches in DPU Security mode
Troubleshoot Smart Switches in DPU Security Mode
This article covers the end-to-end packet flow, spanning from the NPU hardware programming to DPU-based inspection and High Availability (HA) state synchronization. This article also provides systematic procedures for identifying, isolating, and resolving issues within the Smart Switch environment when integrated with Hypershield.
As Hypershield introduces advanced traffic redirection and inspection capabilities—leveraging Data Processing Units (DPUs) and redirection from NPU—troubleshooting requires a multi-layered approach.
The contents in this article are intended for users who have both network-admin and netsecops-admin roles. For more information about these user roles, refer to the User Accounts and RBAC for Smart Switches chapter in the Cisco Smart Switches Configuration Guide for DPU Security Mode. It is assumed that the users have a working knowledge of NX-OS, VRF-lite architectures, and the service-acceleration feature.
The commands and outputs provided herein are based on the current NX-OS Release. Always ensure your environment is running the supported software version before performing deep-dive troubleshooting.
DPU bringup
Successful DPU bringup is required for the Smart Switch to provide security functions. These commands check the status of the DPUs after enabling the service acceleration feature.
Step 1 | Verify that the service-acceleration feature is enabled. Example:
|
Step 2 | (Optional) Verify the feature status using the show feature command. Example:
|
Step 3 | Verify that the DPUs are powered up and online. Example:
Ensure that all listed DPUs' |
Step 4 | Verify the DPU software and hardware versions. Example:
Verify the software (Sw) and hardware (Hw) versions listed for each DPU. |
Step 5 | Verify the active DPU firmware package. Example:
|
Step 6 | Verify the DPU initialization state in the DME. Example:
Ensure the |
Step 7 | Verify the DPU to service-ethernet interface mapping and DPU IP addresses. Example:
Verify the Number of DPUs and Total number of DPU <-> NPU links. For each DPU listed (for example, DPU-1), ensure the Number of links to NPU is 1, the Service Ethernet Interface (NPU) maps to the correct interface (for example, SEth1/1 for DPU-1), and a unique DPU-N IP address is assigned (in the 169.254.x.x range). |
What to do next
After performing these checks, you can confirm that the DPUs have powered up successfully and are in an operational state.
Hypershield Controller to Hypershield Agent Connection Issues
Diagnose and resolve issues preventing the Hypershield Agent (HSA) on the switch from establishing a connection with the Hypershield Controller (HSC). The primary tool for troubleshooting HSC to HSA connectivity is the service system hypershield test controller connection command, which performs a series of checks.
Step 1 | Run the controller connection test utility. Example:
This command performs several checks and indicates the status and potential reasons for connection failures. In a successful connection, verify the following:
Example:
If the agent is not running, the output explicitly states this. To resolve this, ensure the service system hypershield command is configured. Also, ensure that source-interface loopback <idx> is configured under service system hypershield and that the configured loopback interface has a valid /32 IP address in the default VRF. Verify the agent state using the show system internal service-acceleration agent status command. It should show |
Step 2 | Interpret the output if the Firewall subsystem is not Example:
If the firewall service is not enabled, the output prompts you to configure To resolve this, configure in-service under the |
Step 3 | Interpret the output if the token is not configured or an invalid token is configured. Example:
If the OTP has not been configured, the output indicates the missing token. To resolve this, retrieve the OTP from the Hypershield Management Portal (HS MP) and configure it on the switch using the service system hypershield register <token> EXEC command. You can verify if a token is configured (though not the token itself) using show system internal dme running-config all dn sys/sas/volatiledata/agent-hypershield | grep connToken . |
Step 4 | This is the output if the HSA has completed handshake and sync with all DPUs. Example:
If the HSA cannot communicate or synchronize with one or more DPUs, the In the DPU diagnostics:
Collecting the output of the show tech-support service-acceleration command is recommended for further debugging in this scenario. |
Step 5 | When there is no route to the controller the output shows the detailed network connectivity checks and routing verification. Example:
|
Step 6 | (Optional) Test reachability from the HSA container using ping. Example:
This command verifies basic IP connectivity from the HSA's perspective, as the ping is run from within the container. Specify the controller's IP address or hostname to ping. |
By analyzing the output of the service system hypershield test controller connection command and performing additional ping tests from the HSA container, you can identify common configuration or network issues preventing the HSA from connecting to the HSC.
Load Balance mode verification
Verify the configured and operational load-balance configuration. Verify that the service port-channel interfaces for the load-balance mode have been created as expected.
These sections explain troubleshooting for:
Symmetric hash
Step 1 | Identify the load-balance configuration for service-acceleration. In this example, the load-balance configuration is at its default value of symmetric-hash. Example:
|
Step 2 | Verify that the service firewall is created and is in the out-of-service state. Example:
This confirms the firewall service is configured. |
Step 3 | Verify that the service firewall is created. Verify that this shows correctly as out-of-service and the load-balance mode is set to the default mode of symmetric-hash. Example:
|
Step 4 | Verify service port-channel creation and membership for symmetric-hash. Example:
In the above example, only |
Step 5 | Use the consistency-checker for port-channel membership for any service port-channel issues. Example:
|
Step 6 | Verify that the Hypershield Agent has identified the load-balance mode correctly. Example:
|
Pinning load balance mode
Step 1 | Identify the load-balance configuration for service-acceleration. In this example, the load-balance configuration is configured as pinning. Example:
|
Step 2 | Verify that the service firewall is created. Verify that this shows correctly as out-of-service and that the load-balance mode shows the pinning mode. Example:
|
Step 3 | Verify service per-DPU port-channel creation and membership for pinning mode. The output for N9324C-SE1U that is captured here indicates that four service port-channels are created, one for each DPU. Each one of these port-channels has a single service-ethernet interface as its member link. Example:
On N9348Y2C6D-SE1U, two service-port-channels, one per DPU, are created. Each service port-channel has two service-ethernet interfaces as its member links. |
Step 4 | Use the consistency-checker for port-channel membership for any service port-channel issues. Example:
|
Step 5 | Verify that the Hypershield Agent has identified the load-balance mode correctly. Example:
|
What to do next
After performing these checks, you can confirm that the system has detected and enabled the correct load-balance mode as per the configuration.
Network configuration before firewall bringup
Verify that the system is in the expected state when the firewall service is configured but not active. In this state, VRFs configured for service acceleration should be isolated, and the VLANs should be set to drop. Traffic should not be redirected to the DPUs and must be dropped. If traffic is unexpectedly forwarded to the DPU or out of the box when VRFs are isolated or when VLANs are set to drop, perform the checks in this section.
Step 1 | Verify that the service firewall is created and is in the out-of-service state. Example:
This confirms the firewall service is configured. |
Step 2 | Check the operational status of the firewall service. Example:
The |
Step 3 | Identify the VRFs and VLANs enabled for service-acceleration. Example:
Identify the VRFs configured for service acceleration by checking the Identify the VLANs configured for service acceleration by checking the |
Step 4 | Verify that the identified VRFs are in Example:
Verify the routes advertised by the routing protocols for the VRF aligns with behavior supported during protocol isolation. VRF isolation behavior is currently supported for OSPF and BGP. |
What to do next
After performing these checks, you should be able to confirm that the VRFs configured for service acceleration are correctly identified as isolated by the system and routing protocols, which is the expected state before the firewall service is brought up (in-service). You can verify the expected states for VLANs configured for service-acceleration.
NPU packet flows before firewall bringup
When the firewall service is not active (out-of-service state), traffic intended for firewall VRFs should be dropped or bypassed (for specific protocols like BFD or multicast), and not redirected to the DPUs. These steps verify this behavior. If traffic is unexpectedly forwarded to the DPU or out of the box despite correct VRF states and clean consistency checks, perform the verification steps in this section. Traffic intended for Firewall VLANs should be dropped or bypassed (for multicast and broadcast traffic) and not redirected to the DPUs.
Step 1 | Run the Consistency checker for the VRF first for any issues where traffic is expected to land on the switch, but is not dropping or directly routing. Example:
Replace |
Step 2 | Run the Consistency checker for the VLAN first for any issues where traffic is expected to land on the switch, but is not dropping or directly forwarding Example:
Replace |
Step 3 | Verify that the internal policies generated for the intefaces of the VRF permit the BFD echo and multicast traffic and drop all other IP traffic. Example:
This example shows the internally generated IPv4 policy. Verify the ACL counters to detect if any packets are matching the access-list. |
Step 4 | Verify that the internal policies generated for the VLAN permit multicast and broadcast traffic and drop all IP traffic. Example:
|
Step 5 | Verify that the VLAN access-map is attached to the FW VLANs. Example:
|
Step 6 | Verify the ACL counters to detect if any bridged-traffic in the VLANs are matching the access-lists with the drop action. Example:
|
Step 7 | (Optional) Check counters on service-ethernet interfaces to verify no traffic is being sent to the DPU. Example:
Verify that the counters on these interfaces remain at zero or show only minimal control traffic, as traffic should not be sent to the DPU in the out-of-service state. |
By performing these checks, you can confirm that the NPU is correctly handling traffic for firewall VRFs and VLANs in the out-of-service state by dropping or bypassing it according to the programmed policies, and that traffic is not being sent to the DPUs.
Firewall Bringup
Verify that the firewall service and configured VRFs have transitioned to the expected operational states after being enabled. The service firewall becomes ready asynchronously after the in-service command is executed. These steps confirm that the system is ready to attract and process traffic for firewalling.
Step 1 | Verify that the in-service command is configured for the firewall service. Example:
Verify that the |
Step 2 | Verify the final state of the HSA and the configured VRFs and VLANs. Example:
This command provides a detailed status. Verify these:
|
After performing these checks, you can confirm that the firewall service is active and the system, including the HSA and configured VRFs, has reached the necessary operational state to begin processing traffic for firewalling.
Packet Forwarding
Verify the forwarding path for a specific IP flow, including service acceleration redirection, using the show troubleshoot l3 command. This command checks routing information in various software and hardware tables and includes service acceleration consistency checks to help troubleshoot redirection issues.
Step 1 | Run the Layer 3 troubleshooting command for a specific IPv4 flow. Example:
Specify the destination IP ( Analyze the output, which includes:
Verify that consistency checks show |
Step 2 | (Optional) Run the Layer 3 troubleshooting command for a specific IPv6 flow. Example:
Use this command to troubleshoot IPv6 packet forwarding issues, replacing the IP addresses and VRF name as needed. If the show troubleshoot l3 command completes successfully and reports "Consistency checker passed" for the route, it indicates that the system has the necessary routing and service acceleration configuration in place to forward the specified packet flow, including the redirection to the DPU. |
Step 3 | Run the consistency checker for FW VLAN to detect any bridged traffic redirection issues. Example:
If the consistency checker command completes successfully for the VLAN, it indicates that the system has the necessary DPU redirection rules configured in place for bridged-traffic in the VLAN. |
Hypershield Agent to NX-OS Interactions
Verify that the Hypershield Agent (HSA) has successfully pushed the necessary configuration objects to NX-OS through the Data Management Engine (DME) for traffic redirection using ePBR. If traffic is arriving at the switch and dropping or bypassing DPUs despite clean consistency checks, verify the redirection information in the switch using the steps in this section.
Verify if the agent has sent the redirect policies and services configuration to NX-OS. Example:
Verify the dynamic redirection configuration pushed by the Hypershield Agent. Ensure that redirection policies are associated with the correct VRFs and VLANs and that redirection services define the correct DPU module for the VRF and VLAN and encapsulation VLAN ID for each VRF. |
By performing these checks, you can confirm that the Hypershield Agent has successfully pushed the necessary redirection configuration objects to NX-OS using DME, defining how traffic for Firewall VRFs and VLANs should be redirected to the appropriate DPUs.
NPU to DPU Redirection Issues
Verify that the redirection policies are correctly programmed in the NPU hardware (TCAM) and that traffic for the VRFs and VLANs is successfully redirected to the appropriate DPU through service port-channel subinterfaces and service port-channel interfaces. These checks help identify the cause if traffic is arriving but not being redirected to the DPU or is sent to the wrong DPU, even if policies are correctly pushed by the agent.
Step 1 | Identify the service port-channel sub-interfaces created for each VRF to redirect the VRF traffic to the DPU and verify their status. Identify the service port-channel in use for each VLAN to redirect the bridged traffic in the VLAN to the DPU and verify the status. Example:
Identify the service port-channel sub-interface ( |
Step 2 | Verify the configuration details of a specific service-ethernet port-channel sub-interface. Example:
Specify the sub-interface ( |
Step 3 | Verify the detailed policy programming information in ePBR for a specific VRF. Example:
Replace |
Step 4 | Verify detailed policy programming information in ePBR for bridged traffic in VLAN. Example:
Replace |
Step 5 | Verify the packet counters matching the internally generated policies for the VRF to redirect to the DPU. Example:
Example:
Identify the rules relevant to the VRF of interest by identifying the table id for the VRF using the show vrf vrf detail command which is used as the VNI in the above rules. Verify that the rules show redirection to the right subinterface for the VRF. Verify the packet hit counters for the VRF rules. |
Step 6 | Verify that the VLAN access-maps are created and attached to the VLAN for bridged IPv4 or IPv6 traffic. Example:
|
Step 7 | Verify the access-lists used inside the access-maps and the VACL counters. Example:
|
Step 8 | (Optional) Verify resource utilization for Service-acceleration redirect policies. Example:
Verify hardware resource usage for ACL policies, including |
By performing these checks, you can verify that the NPU is correctly programmed to redirect traffic for firewall VRFs and VLANs to the appropriate DPU through the service port-channel sub-interfaces or service port-channel interfaces and confirm that traffic is hitting the expected hardware rules.
Post DPU packet flows
Verify that traffic returning from the DPU is correctly processed by the NPU for forwarding towards its destination by checking the Pass-2 ACLs applied to service port-channel interfaces and verifying interface configuration details. If traffic is dropping in the NPU or not routing after returning from the DPU, perform the verification steps in this section.
Step 1 | Verify the packet counters matching the internally generated policies for the VRF to redirect to the DPU for the second pass of inspection, before the flows are learned. Example:
Example:
Identify the rules relevant to the VRF of interest by identifying the table ID for the VRF using show vrf <> detail , which is used as the VNI in the above rules. Verify that the rules include permit entries matching traffic returning from the DPU and redirect it to the correct service port-channel subinterface for the VRF. Confirm the packet hit counters for new flow learns using these VRF rules. |
Step 2 | Verify the packet counters matching the internally generated policies for the VLANs to redirect to the DPU for the second pass of inspection before the flows are learned. Example:
For pinning mode unique ACLs per DPU are created. |
Step 3 | Show statistics for DPU connected service-ethernet ports. Example:
These commands provide various statistics for the service-ethernet interfaces connected to the DPUs, including packet/byte counters, detailed stats, error stats, and average rates. Checking these counters helps determine if traffic is reaching or leaving the DPUs as expected. |
By performing these checks, you can verify that the NPU has correctly programmed the Pass-2 ACLs to process traffic returning from the DPU and that the service-port channel interfaces and subinterfaces are configured with the correct parameters for forwarding.
Packet Tracer Support
Use the Packet Tracer utility to capture and analyze packets at different points in the end-to-end NPU to DPU to NPU packet flow. This allows for detailed troubleshooting of traffic handling at various stages.
These steps outline how to capture packets at specific points:
- NPU ingress
- NPU to DPU TX
- DPU to NPU RX - first packet
- DPU to NPU RX - subsequent packets
- NPU egress
Step 1 | Capture packets received on the ingressing Layer 3 interface (NPU ingress). Example:
This captures packets received by the NPU on the ingressing Layer 3 interface before redirection to the DPU. |
Step 2 | Capture packets transmitted from the NPU to the DPU on the service-ethernet interface (SETH TX). Example:
This captures packets transmitted by the NPU towards the DPU on the service-ethernet interface. The snapshot is taken before the DPU header encapsulation happens, so DPU header details cannot be traced or captured at this point. |
Step 3 | Capture the first packet received by the NPU from the DPU with the pass bit set (SETH RX dpu pass 0). Example:
This captures the initial packets received by the NPU from the DPU on the service-ethernet interface, specifically those marked with DPU pass bit 0 (typically the first packet of a flow punted to the DPU's control plane). The snapshot is taken before DPU decap happens, but you can get insights into the received DPU header. For routed-traffic, the service-vlan to filter on can be obtained from the VLAN information in show service-acceleration redirect-policy vrf name . For bridged-traffic use the forwarding VLAN of the packet as the service VLAN. |
Step 4 | Capture subsequent packets received by the NPU from the DPU with the pass bit set (SETH RX dpu pass 1). Example:
This captures subsequent packets for a flow received by the NPU from the DPU on the service-ethernet interface, specifically those processed directly by the DPU's data plane (P4) and marked with DPU pass bit 1. The snapshot is taken before DCPU decap happens, but you can get insights into the received DPU header. |
Step 5 | Capture packets transmitted from the NPU to the egress L3 interface after routing (NPU egress). Example:
This captures packets transmitted by the NPU towards the egress Layer 3 interface after being processed and routed post-DPU. The snapshot is taken before the DPU header encapsulation happens, so the DPU header details cannot be traced or captured at this point. |
After performing these packet capture steps, you can analyze the captured packets to gain insight into how traffic is being processed at different points in the NPU to DPU to NPU flow.
Inter-VRF Traffic Inspection
When troubleshooting Inter-VRF traffic inspection using route leak details, the route is initially learned in the source VRF, for example, red, and then leaked to the target VRF, for example, green, as shown by route entries with next-hop information referencing the source VRF.
Running the show troubleshoot command provides a comprehensive view of the route leak. The output displays route information from the PI RIB, FIB, and hardware layers. This confirms the route's presence and next-hop details across VRFs and verifies its installation in hardware for proper forwarding. This layered inspection helps identify where the route leak is functioning correctly or where issues can arise in the Inter-VRF traffic path.
Use this procedure to verify the route leak details for Inter-VRF traffic inspection and troubleshoot accordingly.
Step 1 | Use the show ip route ip-address vrf vrf-name command for the source VRF, for example, VRF red. Example:
|
Step 2 | Use the show ip route ip-address vrf vrf-name command for the target VRF, for example, VRF green. Example:
|
Step 3 | Use the troubleshoot command to see the route leak details. The route details are shown from URIB, FIB, and the hardware. Example:
|
High Availability
Use this procedure to verify the details of High Availability (HA) and troubleshoot issues for these scenarios accordingly:
- connectivity issues
-
both peers are in
no in-servicemode - one peer is in-service mode and the other is not
- both peers are in in-service mode
- membership failures
High Availability (HA) connectivity between Hypershield Agents is accomplished only when:
- HA configuration is complete, correct, and symmetric
- HA is enabled, and
- Service Firewall is configured.
Troubleshooting connectivity issues involves these steps:
- Verify the HA configuration.
- Verify the network connectivity between the peers and the containers.
- Verify Hypershield Agent states for High Availability.
Step 1 | Verify the High Availability status. Example:
|
Step 2 | Verify Service-acceleration configuration and network configuration and routes on each peer.
Example:
|
Step 3 | Verify the networking configuration has been set up for the HA connectivity to the configured peer from the local HA IP. In this example, the loopback interface is configured with the correct IP address and OSPF settings, and the routing table shows the route to the destination reachable using OSPF through the Ethernet interface. Example:
|
Step 4 | Run HA consistency checker on each peer to validate the NX-OS configuration with the Hypershield Agent driven states and detect discrepancies. Example:
|
Step 5 | Run the network connectivity troubleshooting tool to detect network connectivity issues to the peer's Hypershield Agent. Example:
|
Both peers are in no in-service mode
When both peers are in no in-service traffic deflection for vPCs and bridged traffic is turned off in both vPC peers, vPCs carrying VLANs that are Layer 3 or Layer 2 firewall VLANs are not suspended.
VRFs remain in isolated state on both peers. For more information, see Network configuration before firewall bringup.
Any traffic arriving for VRFs or VLANs drops. For more information, see Network configuration before firewall bringup.
DPU keepalives and syncs are disabled when firewall is not in-service. Only Hypershield Agents establish HA connectivity at this time.
Step 1 | Both peers must report local state as not-ready and peer service state as not-ready. Peer HA state is unavailable on both sides. Example:
|
Step 2 | Verify Hypershield Agent HA information/configuration. Example:
Example: |
One peer is in in-service mode and the other is not
When one peer is in no in-service and other peer is in in-service, traffic must be deflected towards peer that is in-service.
-
Verify that the FW service in Hypershield Agent and NXOS is ready on the peer that is
in-service. - Verify state visibility between the peers.
- Verify VPC and bridged traffic deflection has been turned off on the peer that is FW ready.
- Verify VRFs are no longer isolated and traffic is able to redirect to DPU on the peer that is FW ready. For more information, see Packet Forwarding.
- Verify vPC and bridged traffic deflection is in effect on the peer that is FW not-ready.
- DPU keepalives and syncing are still disabled on the peer that is not in-service.
Step 1 | Verify the HA status. Example:Example:The peer that is in-service reports the local state as ready and remote state as not-ready. Similarly, the peer that is not in-service reports the local state as not-ready and remote state as ready. The HA state is not-ready on the peer that is onboarded and is unavailable on the other peer. |
Step 2 | Run HA consistency checker on each peer to validate NX configuration, states with Agent driven states and verify traffic deflection expectations. Example: |
Step 3 | Verify that the vPCs are not suspended on the peer that is not in-service and are not suspended on the peer that is in in-service mode. Example: |
Step 4 | Verify that the HSRP groups transition to active on the peer that is now ready, as the VRFs are no longer isolated while the HSRP groups remain in INIT state on the peer that is not-ready. As vPC uses HSRP active-active, traffic deflection to the VIP occurs as a result of vPCs being suspended on the not-ready HA vPC peer. Example:Example: |
Step 5 | Verify Hypershield Agent HA information/configuration. The peer that is ready assumes the role of the leader. Example:
|
Both peers are in in-service mode
When both peers are in in-service mode, traffic deflection must be turned off, when both peers are FW ready:
- Verify the FW readiness on each peer.
- Verify the state visibility between the peers.
- Verify that the vPC and bridged traffic deflection has been turned off on both peers (See previous sections).
- Verify that the VRFs are no longer isolated and traffic is able to redirect to DPU on the peer that is FW ready (Refer to Packet Forwarding).
- Verify flow-sync between peers.
Step 1 | Verify that both peers report local and peer state as ready. HA state is ready on both peers. Example:Example: |
Step 2 | Run HA consistency checker on each peer to validate NX configuration, states with Agent driven states and verify traffic deflection expectations. Example:
|
Step 3 | The encapsulation VLANs used for the FW VRFs must be consistent on both peers. If these are not consistent, these should be flagged by the CC. These IDs must be consistent on the DPUs of the two switches and may be verified by comparing them to the values visible on NXOS using the show service-acceleration redirect-policy brief command. Example:
|
Step 4 | Verify the Hypershield Agent HA information/configuration. Example:Example: |
Step 5 | Verify the Hypershield Agent HA status on the follower that is now ready. Example:
|
Step 6 | Verify the Hypershield Agent HA peer status on peer that is ready. Example:
|
Membership failures
Membership failures can trigger a switchover/takeover scenario and can occur as a result DPU keep-alive failures, version and platform mismatches, load-balance mode mismatches.
In case both switches are FW ready, the leader takes over the traffic and the follower transitions to not-ready.
Traffic deflection mechanisms are enabled when the firewall transitions to not-ready while in in-service.
Step 1 | The peer elected as leader takes over the traffic and the follower transitions local state as not-ready. Example:
|
Step 2 | Use this command to identify reasons for compatibility failures. Example:
|
Step 3 | Verify that the VRFs are moved to redirection Example:
|
Step 4 | Verify Hypershield Agent HA status during switchover or takeover. Example:
|