Hypershield🔗

Architecture🔗

Hypershield separates policy administration, policy distribution, and policy enforcement into three planes. Each plane has a defined responsibility and interacts with the other planes through well-defined interfaces.

The architecture consists of three planes:

• Management plane

  This plane administers security policies and provides inventory management, observability, and management APIs. It consists of Timescape, the local management interface.

  Timescape can optionally be fronted by Cisco Security Cloud Control, which reaches Timescape through the Secure Connector (not shown in the diagram, for simplicity). The choice of management interface determines the operating mode; see Operating modes.

• Control plane

  This plane distributes approved security policies to managed enforcement points and reports deployment status. The on-prem controller performs this role.

• Data plane

  This plane enforces security policy on traffic at supported enforcement points. See Enforcement points for the enforcement points supported today.

The following diagrams show the logical relationships between components; physical deployment may differ. The management plane runs on the leader cluster and the control plane runs on one or more follower clusters; see Leader and follower clusters.

Security policy flow🔗

Each architectural plane corresponds to one stage of the security policy flow:

• Policy administration (management plane)

  Administrators define, review, and approve security policies and policy objects that describe the protected resources. Policies represent administrative intent and do not affect network traffic until they are deployed.

• Policy deployment (control plane)

  After policy is approved, the control plane distributes it to the appropriate enforcement points and reports deployment status back to the management plane, allowing administrators to monitor policy rollout.

• Policy enforcement (data plane)

  Each enforcement point applies its locally installed policy to network traffic. Policy decisions are made locally and continue to be enforced even if connectivity to the management or control plane is temporarily unavailable, until the policy is updated or removed.

Operating modes🔗

Hypershield supports two operating modes: connected and disconnected. The operating mode determines where security policies are administered. It does not change how those policies are deployed or enforced: in both modes, the control plane is local and enforcement is local.

• Connected mode

  This mode uses Cisco Security Cloud Control (SCC) as the management interface, providing centralized policy administration across managed environments. Approved policy is distributed to the local leader (connected by the Secure Connector), and deployment status is reported back to Cisco Security Cloud Control.

• Disconnected mode

  This mode uses Timescape as the local management interface, providing local policy administration. Intended for environments that require local administration or that operate without connectivity to SCC, such as air-gapped or isolated deployments.

Leader and follower clusters🔗

The management plane and the control plane are implemented as separate Kubernetes clusters. A Hypershield deployment includes one leader cluster and one or more follower clusters. Timescape coordinates security policy across the clusters.

• Leader

  This cluster hosts the management plane. Administrators interact with the leader to define, review, and approve security policy. It is the source of truth for policy state and distributes approved policy to the followers.

• Follower

  This cluster hosts the control plane for a set of enforcement points, typically the Smart Switches within a single failure domain such as a datacenter. It receives approved policy from the leader and is responsible for pushing that policy to its enforcement points and for reporting deployment status back.

Choosing a design🔗

In practice, users usually choose between the two designs based on questions such as:

• Which unit is the security concern - a broader network boundary, or specific workload relationships?

• Does the existing network design already provide natural trust boundaries to align policy with?

• Do you need to distinguish access rules between workloads that share the same broader environment?

Structure🔗

A security policy is built from a small set of parts. Each policy contains one or more rules; rules reference network objects, either directly or grouped as network object groups. Separating these parts from policy rules lets multiple policies reuse the same definitions, reducing duplication and simplifying administration.

Lifecycle🔗

A security policy moves through the following stages, from definition to enforcement and later change:

• Define

  An administrator creates the policy that describes the intended communication behavior.

• Review

  Optionally, an administrator stages the change as a draft (Timescape) and reviews it before deployment. See Validate policies with drafts for the draft workflow.

• Validate

  Hypershield checks the policy for structural validity as a deployment-time gate. A failure rejects the deployment.

• Deploy

  Hypershield distributes the validated policy to the relevant enforcement points, where it becomes part of the active policy state.

• Enforce

  Enforcement points evaluate traffic against the active policy state. Enforcement continues locally even if the management or control plane is temporarily unreachable.

• Change

  An administrator updates, replaces, or removes the policy. Each change returns to validation and deployment before it becomes active.

Evaluation and rule order🔗

Order matters when multiple rules could apply to the same traffic. Broad rules, narrow rules, and overlapping rules can affect each other, so the final outcome depends on the active policy set rather than any single rule in isolation.

For example, a broad rule can make a narrower rule ineffective if the broad rule matches the same traffic first. This is sometimes called a shadowed rule. That kind of overlap can hide policy intent and make live behavior differ from what an administrator expects.

As policy sets grow, it is important to consider how rules interact, how changes affect the overall result, and whether a policy behaves as intended before it is enforced.

Kubernetes implementation🔗

Internally, Hypershield represents security policies and network object groups as Kubernetes custom resources, such as SmartSwitchNetworkPolicy and NetworkObjectGroup. Administrators interact with these resources through Timescape and SCC, either directly or through Cisco Security Cloud Control (SCC), which fronts Timescape. Timescape also exposes a public REST API for automation; see Timescape API usage.

Smart Switch🔗

A Smart Switch operates as a managed enforcement point in a Hypershield deployment. It enforces security policies inline as traffic traverses the switch.

Key components🔗

Three hardware and software components work together within the Smart Switch to receive centrally managed policies and enforce them on network traffic.

Data flows🔗

The Smart Switch maintains separate flows for policy deployment and traffic processing. The deployment flow installs security policies on the switch; the enforcement flow applies those policies to network traffic.

DPU traffic steering🔗

DPU traffic steering is the NX-OS mechanism that redirects traffic from the NPU (the switch’s normal forwarding path) to the DPU for inspection.

Policy is evaluated only for traffic that NX-OS steers to a DPU. Traffic that remains on the NPU forwarding path bypasses Hypershield policy evaluation. Steering configuration therefore determines the scope of enforcement, while policy definition determines how traffic within that scope is handled.

High availability🔗

The Smart Switch continues enforcing previously deployed security policies during temporary management connectivity failures and supported network redundancy events.

New or updated policy can be deployed after follower connectivity is restored.

When deployed using supported network redundancy mechanisms, such as Virtual PortChannel (vPC), policy enforcement continues across the remaining forwarding path during a switch failure, provided the underlying network redundancy remains operational.

Deployment topology🔗

Timescape spans the leader and follower clusters. The leader hosts the primary Timescape instance where administrators define and approve policy; each follower runs a coordinating Timescape instance that receives approved policy from the leader and materializes it as security policy objects in the follower’s Kubernetes cluster, which are then applied at the enforcement points.

Connectivity between clusters uses the Secure Connector, a client-server pair in which the client always initiates the connection. Two independent pairs are involved:

• Follower to leader. Each follower runs a Secure Connector client that connects to a Secure Connector server on the leader. This link carries coordination traffic between the follower’s Timescape and the leader’s Timescape.

• Leader to SCC (connected mode only). The leader runs a Secure Connector client that connects to a Secure Connector server in the Cisco cloud, enabling Cisco Security Cloud Control to front the leader’s Timescape.

Access🔗

Timescape is not exposed by default. When users or API clients need access, you, the administrator, expose Timescape by configuring ingress, TLS, and a deployment-specific URL for the web UI and API.

For authentication, Timescape supports OpenID Connect (OIDC), so authentication can be delegated to an existing identity provider. Combined with role-based access control, this determines what different users can view or change. OIDC also authenticates public REST API clients when API RBAC is enabled (see Public REST API).

User interface paths🔗

You reach the user interface through one of two paths, one for each operating mode (see Operating modes):

• SCC (connected mode) - Timescape is embedded within the Cisco Security Cloud Control platform.

• directly (disconnected mode) - Timescape is accessed without going through SCC, with Isovalent Timescape branding.

Web UI layout🔗

The layout of the Timescape sidebar also differs between the two paths. With SCC, Hypershield appears under the Platform menu alongside the other SCC platform services. When accessing directly, Timescape presents its own sidebar grouped by function.

SCC (connected)

Direct (disconnected)

Public REST API🔗

The versioned public REST API supports customer automation for network object and security policy lifecycle operations. See Timescape API usage to configure access, then follow Manage network objects or Manage security policies for common workflows.

The public API records desired configuration. A successful policy mutation does not by itself confirm that every enforcement point has applied the change. Use policy deployment status to inspect the current state reported by enforcement agents, not as proof that they applied the desired revision. For the definitive endpoint and response contract, see the Timescape public API reference.

Management paths🔗

The identity and access model applies across different management paths:

• user access to local management surfaces such as Timescape

• user access to centralized management surfaces such as Cisco Security Cloud Control (SCC)

• API-driven access for automation and integrations

• service-to-service access used by platform components

Each path defines which management boundary a user or system is crossing and which part of the platform is authoritative for that action.

Authorization and roles🔗

For authentication and authorization, the platform needs to know:

• which identity is making the request

• which management surface or client the session is intended for

• which role or permission set applies to that identity

Management operations often span different responsibility levels. The RBAC approach separates those responsibilities and enables least-privilege operation. Tasks that demand varying levels of authorization include:

• manage platform settings

• create or change policy

• review objects, inventory, and status

• observe changes without modifying the environment

API access🔗

The same identity model applies to APIs used for automation. API access still depends on authenticated sessions or tokens and the permissions associated with the calling identity. API-driven workflows need:

• a valid token or authenticated session

• the correct access context for the service being used

• a role that permits the requested operation

Hypershield exposes APIs on several management surfaces:

• the Kubernetes API server on the on-prem controller, which customers can access directly with kubectl for cluster-level operations

• the Public REST API for object and policy lifecycle operations

• the Cisco Security Cloud Control (SCC) API for centralized management

Component and service identities🔗

Platform components also need trusted identities when they communicate with other services. For example, local services, connectors, and enforcement-side components may need authenticated access to APIs, object stores, or management endpoints.

Visibility streams🔗

Hypershield provides three primary visibility streams. They are complementary rather than interchangeable: a given operational question is usually best answered by one of them.

• Syslog

  Per-flow permit and deny records, exported from enforcement points to external collectors. Syslog is the primary input for SIEM and log-analysis platforms.

• IPFIX

  Connection-oriented flow records, exported from enforcement points to external collectors. IPFIX is used for traffic reporting and trend analysis rather than individual permit or deny decisions.

• Timescape event history

  A record of management activity - who created, modified, deployed, or removed a security policy, network object, or other managed resource. Timescape event history is reviewed in the Timescape UI (see Timescape and SCC) rather than exported.

Origins🔗

The streams originate in different parts of the platform, which affects how they are configured and what they can report on:

• Syslog and IPFIX originate on the data plane. Enforcement points generate the records as they process traffic and export them directly to the configured collectors.

• Timescape event history originates on the management plane. It reflects administrative intent and deployment state, not live traffic.

Because syslog and IPFIX are exported to external infrastructure, collector reachability affects whether records can be delivered. Timescape event history is part of the management surface itself and is available wherever Timescape is available.

Related workflows🔗

To configure syslog and IPFIX export, see Manage telemetry destinations and Manage logging groups. For the topology view of enforcement points and traffic, see Use the Topology map.

Prerequisites🔗

• Timescape is deployed and accessible (see Timescape and SCC).

• The leader-path configuration is set in your Timescape deployment.

• You have credentials for an identity provider configured with Timescape OIDC, or direct access to the UI.

Access the Leaders page🔗

When no leader is enrolled, the UI automatically redirects you to the Leaders page on first access. If a leader already exists, navigate to Operations ‣ Leaders in the sidebar.

Register a new leader🔗

1. Click Add leader.

2. Fill in the leader details:

   • Leader name – A human-readable name to identify this leader cluster.

   • Description (optional) – Additional context about the leader’s purpose or environment.

3. Click Save.

4. The UI generates a one-time password (OTP) for the leader cluster to use during its initial connection. Copy this value before leaving the page.

5. The leader cluster uses this OTP to authenticate its first connection to Timescape. Once connected, the leader appears with an active status in the Leaders list.

View leader details🔗

Click a leader name in the list to view its detail page. The detail page shows:

• Leader metadata (name, description, creation time)

• Connection status

Place a user in a group🔗

To place a user in a group follow these steps:

1. Log into the Keycloak admin console: https://keycloak.<your-domain>/admin/master/console/

2. Select the hypershield realm

3. Navigate to Users → <username> → Groups

4. Add the user to the group

Permission changes take effect within approximately 30 seconds - existing sessions honor the change on their next API call. Neither a Pod restart nor a user re-login is required.

View the followers list🔗

Navigate to Operations ‣ Followers to see all registered clusters.

The list supports:

• Status filter – Filter by Active, Disabled, or Pending status.

• Search – Filter clusters by name.

• Actions menu – Enable, disable, or delete clusters from the row action menu.

Cluster statuses:

• Active – The cluster is connected and receiving policy updates.

• Pending – The cluster is registered but has not yet connected.

• Disabled – The cluster is paused and not receiving policy updates.

Register a new follower🔗

1. Click Register follower.

2. Fill in the cluster details:

   • Name – A unique name for the cluster.

   • Labels (optional) – Key-value pairs for organizing clusters (for example, by site, region, or role).

   

3. Click Save.

4. The UI generates connection credentials for the cluster. Provide these to the cluster administrator to complete the registration.

   

The cluster administrator uses these credentials to register the follower with the leader. After the follower is successfully registered, it becomes active and can manage enforcement points.

View follower details🔗

Click a cluster name in the list to open its detail page. The detail page shows:

• Cluster metadata (name, labels, creation time)

• Current connection status

• Policy synchronization state

Edit a follower🔗

From the detail page, click Edit to modify the cluster’s name or labels.

Enable or disable a follower🔗

Use the actions menu on the list page to change a cluster’s state:

• Disable – Pauses policy distribution to the cluster. The cluster retains its last-known policy state but stops receiving updates.

• Enable – Resumes policy distribution. The cluster receives all pending policy updates.

View follower event logs🔗

From the cluster detail page, switch to the Logs tab to review the event history (registration, state changes, connection events).

Delete a follower🔗

Use the actions menu to delete a cluster. Only clusters in Disabled or Pending status can be deleted.

Using the user interface🔗

In the Timescape UI, you manage leaders from the Operations ‣ Leaders page.

View the leaders list🔗

Navigate to Operations ‣ Leaders to see all registered leader clusters in your deployment.

When no leaders are enrolled yet, you’ll see an empty state:

View leader details🔗

Click a leader name in the list to open its detail page. The detail page shows:

• Leader metadata (name, description, creation and modification times)

• Connection status and health

• Configuration details

• Associated clusters or resources managed by this leader

Troubleshooting🔗

View the objects list🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" \
  --get \
  --header "Accept: application/json" \
  --data-urlencode "namespace=hypershield" \
  --data-urlencode "search=app" \
  --data-urlencode "labelSelector=environment=production" \
  --data-urlencode "pageSize=50" \
  "${TIMESCAPE_API}/network-object-groups"

Navigate to Network Security ‣ Objects to see all network object groups in your deployment.

The list supports:

• Search – Filter objects by name, labels or IP addresses.

• Pagination – Navigate through large object sets.

• Actions menu – Edit or delete objects from the row action menu.

Create a network object🔗

{
  "namespace": "hypershield",
  "name": "app-servers",
  "labels": {
    "environment": "production"
  },
  "spec": {
    "description": "Application server subnets",
    "cidrs": [
      "10.10.20.0/24",
      "10.10.21.0/24"
    ]
  }
}

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --request POST \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  --data-binary "@app-servers.json" \
  "${TIMESCAPE_API}/network-object-groups"

1. Click Create object.

2. Select a deployment target in the modal that appears. The deployment target determines which draft or live environment receives the new object (see the draft section).

   

3. Fill in the object details:

   • Name – A unique name for the network object group (must be valid as a Kubernetes resource name).

   • Labels (optional) – Key-value pairs for organizing and filtering objects.

   • CIDRs – One or more IP address ranges that this object represents.

   

4. Click Deploy.

The object is created according to your selected deployment target (live or draft).

View object details🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --header "Accept: application/json" \
  "${TIMESCAPE_API}/network-object-groups/hypershield/app-servers"

Click an object name in the list to open its detail page. The detail page shows:

• Object metadata (name, namespace, labels, creation and modification times)

• CIDR list

• Attached policies that reference this object

Edit a network object🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --header "Accept: application/json" \
  "${TIMESCAPE_API}/network-object-groups/hypershield/app-servers"

{
  "labels": {
    "environment": "production"
  },
  "spec": {
    "description": "Application server subnets",
    "cidrs": [
      "10.10.20.0/24",
      "10.10.21.0/24",
      "10.10.22.0/24"
    ]
  }
}

OBJECT_ETAG='"42"'

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --request PUT \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  --header "If-Match: ${OBJECT_ETAG}" \
  --data-binary "@app-servers-update.json" \
  "${TIMESCAPE_API}/network-object-groups/hypershield/app-servers"

1. From the detail page, click Edit, or use the actions menu in the list page.

2. Select a deployment target in the modal that appears. The deployment target determines which draft or live environment receives the updated object (see the draft section).

3. Modify the object’s labels or CIDRs as needed.

   

4. Click Deploy.

View object event logs🔗

From the object detail page, switch to the Logs tab to review the event history for this object (creation, modifications, deletions).

Delete a network object🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" \
  --request DELETE \
  "${TIMESCAPE_API}/network-object-groups/hypershield/app-servers"

Use the actions menu on the list page or the detail page to delete an object.

View the policies list🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" \
  --get \
  --header "Accept: application/json" \
  --data-urlencode "namespace=hypershield" \
  --data-urlencode "labelSelector=environment=production" \
  --data-urlencode "includeStatus=true" \
  --data-urlencode "pageSize=50" \
  "${TIMESCAPE_API}/policies"

Navigate to Network Security ‣ Policies to see all Hypershield policies.

The page includes summary cards at the top showing:

• Policies in sync with enforcement points

• Policies out of sync (pending deployment or with errors)

Create a security policy🔗

{
  "namespace": "hypershield",
  "name": "allow-load-balancers-to-app-servers",
  "labels": {
    "environment": "production"
  },
  "spec": {
    "rules": [
      {
        "description": "Allow load balancers to reach application servers",
        "action": "allow",
        "source": {
          "networkRefs": [
            {
              "name": "load-balancers"
            }
          ]
        },
        "destination": {
          "networkRefs": [
            {
              "name": "app-servers"
            }
          ],
          "protoPorts": [
            {
              "protocol": "TCP",
              "port": 8080
            },
            {
              "protocol": "TCP",
              "port": 8443
            }
          ]
        }
      }
    ]
  }
}

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --request POST \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  --data-binary "@allow-load-balancers-to-app-servers.json" \
  "${TIMESCAPE_API}/policies"

1. Click Create policy.

2. Select a deployment target (save directly or add to a draft, see the draft section).

   

3. Fill in the policy details:

   • Name – A unique name for the policy.

   • Rules – Define one or more rules, each with:

     • Action – Allow or deny.

     • Source – Select a network object or specify an inline CIDR.

     • Destination – Select a network object or specify an inline CIDR.

     • Protocol and ports – TCP or UDP with specific port numbers.

   

   

4. Click Deploy.

View policy details🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" \
  --get \
  --header "Accept: application/json" \
  --data-urlencode "includeStatus=true" \
  "${TIMESCAPE_API}/policies/hypershield/allow-load-balancers-to-app-servers"

Click a policy name in the list to open its detail page. The detail page shows:

• Policy metadata (name, namespace, creation and modification times)

• Enforcement state across all enforcement points

• The complete rule set

Edit a security policy🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --header "Accept: application/json" \
  "${TIMESCAPE_API}/policies/hypershield/allow-load-balancers-to-app-servers"

{
  "labels": {
    "environment": "production"
  },
  "spec": {
    "rules": [
      {
        "description": "Allow load balancers to reach application servers",
        "action": "allow",
        "source": {
          "networkRefs": [
            {
              "name": "load-balancers"
            }
          ]
        },
        "destination": {
          "networkRefs": [
            {
              "name": "app-servers"
            }
          ],
          "protoPorts": [
            {
              "protocol": "TCP",
              "port": 8443
            }
          ]
        }
      }
    ]
  }
}

POLICY_ETAG='"17"'

curl "${TIMESCAPE_CURL_ARGS[@]}" --include \
  --request PUT \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  --header "If-Match: ${POLICY_ETAG}" \
  --data-binary "@allow-load-balancers-to-app-servers-update.json" \
  "${TIMESCAPE_API}/policies/hypershield/allow-load-balancers-to-app-servers"

1. From the detail page, click Edit, or use the actions menu in the list page.

2. Modify rules, sources, destinations, or ports as needed.

   

3. Click Deploy.

View policy event logs🔗

From the policy detail page, switch to the Logs tab to review the event history (creation, modifications, deployments, deletions).

Delete a security policy🔗

curl "${TIMESCAPE_CURL_ARGS[@]}" \
  --request DELETE \
  "${TIMESCAPE_API}/policies/hypershield/allow-load-balancers-to-app-servers"

Use the actions menu on the list page or the detail page to delete a policy.

View active drafts🔗

The Drafts page opens on the Active tab, which lists all drafts that have not yet been deployed or discarded.

View deployment history🔗

Switch to the Deployment history tab to see previously deployed drafts and their outcomes.

Create a draft🔗

1. Click Create draft.

2. In the modal, enter a name and optional description for the draft.

   

3. Click Create. You are redirected to the draft detail page.

Add changes to a draft🔗

From the draft detail page, you can add policy and object changes:

1. Click Add policy change or Add object change.

2. In the modal that appears, select an existing resource to modify or delete, or create a new one.

   

   

3. Make your changes in the form and save.

The draft detail page shows all pending changes in a table.

Review draft changes🔗

The draft detail page displays all changes grouped by type:

You can filter changes by:

• Kind – Show only policies, only objects, or all changes.

• Operation – Filter by create, edit, or delete operations.

• Conflicts – Show changes that conflict with the current live state (another user modified the same resource since the draft was created).

• Search – Filter by resource name.

Resolve conflicts🔗

When a resource in a draft has been modified in the live environment since the draft was created, the change shows a conflict indicator. Conflicts can be:

• Auto-resolvable – The draft change and live change do not overlap; the system can merge them automatically.

• Manual resolution – The changes overlap and require you to choose which version to keep.

Review conflicting changes and resolve them before deploying the draft.

Deploy a draft🔗

When all changes are reviewed and conflicts are resolved:

1. Click Deploy in the draft header.

2. Confirm the deployment in the modal.

   

3. The draft moves to the deployment history with a success or failure status.

View draft event logs🔗

From the draft detail page, switch to the Logs tab to see the event history for this draft (creation, changes added, deployment attempts).

Delete a draft🔗

Use the delete action in the draft header or from the drafts list. Deleting a draft discards all its pending changes without applying them.

View the logging groups list🔗

Navigate to Operations ‣ Logging Groups to see all configured logging groups.

The list supports:

• Type filter – Filter by Syslog or IPFIX type.

• Search – Filter groups by name.

• Actions menu – Edit or delete groups from the row action menu.

Create a logging group🔗

1. Click Create logging group. You will see the logging group creation form.

   

2. Fill in a unique name for the logging group.

3. Assign the telemetry destination this group sends data to. These are

   the servers that will receive data from clusters in this group. These are the servers that will receive data from clusters in this group and can be of type Syslog or IPFIX.

4. Select which follower clusters belong to this logging group. Those

   clusters will export telemetry to the assigned destinations.

5. Click Save.

View logging group details🔗

Click a group name in the list to open its detail page. The detail page shows:

• Group metadata (name, type, creation time)

• Assigned telemetry destinations

• Member clusters

Edit a logging group🔗

From the detail page, click Edit to modify the group’s destinations or cluster assignments.

Delete a logging group🔗

Use the actions menu on the list page or the detail page to delete a group. Deleting a logging group stops telemetry export for the affected clusters and destinations.

View the telemetry destinations list🔗

Navigate to Operations ‣ Telemetry Destinations to see all configured destinations.

The list supports:

• Type filter – Filter by Syslog or IPFIX type.

• Search – Filter destinations by name.

• Actions menu – Edit or delete destinations from the row action menu.

Create a telemetry destination🔗

1. Click Create telemetry destination.

2. Fill in the destination details:

   • Name – A unique name for the destination.

   • Description (optional) – Additional context about the destination.

   • Type – Select Syslog or IPFIX.

   • Protocol – The transport protocol (TCP or UDP).

   • Hostname / IP – The collector’s address.

   • Port – The collector’s listening port.

   

3. Click Save.

View destination details🔗

Click a destination name in the list to open its detail page. The detail page shows:

• Destination metadata (name, type, protocol, hostname, port)

• Logging groups that reference this destination

Edit a telemetry destination🔗

From the detail page, click Edit to modify the destination’s connection parameters.

Delete a telemetry destination🔗

Use the actions menu on the list page or the detail page to delete a destination.

Overview🔗

The topology map displays:

• Network object groups as cards (vertices) on the map

• Connections between objects as directional arrows (edges)

• Zones or VLANs as container regions that group related objects

• Policy coverage indicated by arrow coloring (governed vs. ungoverned traffic)

Navigate the map🔗

• Pan – Click and drag the map background to move the view.

• Zoom – Use the scroll wheel or pinch gesture to zoom in and out.

• Select – Click a card or arrow to view its details in a side panel.

Filter the view🔗

The filter bar at the top of the page lets you refine what the map shows:

• CEL expression – Write a Common Expression Language filter to show only specific traffic (for example, by source or destination CIDR, VLAN, or protocol).

• Time range – Select the time window for traffic data.

• Refresh rate – Set how frequently the map updates with new data.

Understand policy coverage🔗

Arrows on the map are colored to indicate policy coverage:

• Governed (default color) – Traffic between these objects is covered by at least one Hypershield policy.

• Ungoverned (warning color) – Traffic exists between these objects but no policy governs it. This may indicate a gap in your security posture.

View connection details🔗

Click an arrow to open the arrow widget, which shows:

• Source and destination objects

• Traffic volume and protocol breakdown

• Policy coverage details (which policies apply)

Create resources from the map🔗

The topology map supports inline resource creation for common workflows:

Create a network object from an ungrouped CIDR🔗

When the map shows traffic from an IP range that is not yet assigned to a network object:

1. Click the ungrouped CIDR card on the map.

2. In the side panel, click Create object.

3. Select a deployment target (live or draft).

4. The object creation form opens pre-filled with the CIDR.

Create a policy from a connection🔗

When you identify ungoverned traffic that needs a policy:

1. Click the arrow between two objects.

2. In the arrow widget, click Create policy.

3. Choose a deployment target (live or draft).

4. The policy creation form opens with source and destination pre-filled from the edge endpoints.

View connection logs🔗

Below the map, a resizable panel shows the connection logs table. This table lists individual connections that match the current map filter and time range.

Zone and VLAN grouping🔗

When your environment uses VLANs or zones, the map groups objects into labeled containers. Use the zone selector in the header to show or hide specific zones. You can also click on one zone in the map to restrict data to that zone.

Back up the Leader cluster🔗

Run the following to back up the Leader cluster:

sudo KUBECONFIG="<leader kubeconfig>" \
  hypershield/scripts/hs-backup.sh --appliance-type leader

Example successful output:

[21:10:44] INFO  Checking available disk space...
...
Enter backup passphrase:
...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Backup complete (leader): ./hypershield/hypershield-backup-20260603-211044-leader.tar.gz
SHA256: 8e9290d9493669e8860798cfae16ac1f220cfe94831e35c1e76d5e69d9a7ba81
...

Back up the Follower cluster🔗

Run the following to back up the Follower cluster:

sudo KUBECONFIG="<follower kubeconfig>" \
  hypershield/scripts/hs-backup.sh --appliance-type follower

Example successful output:

[21:10:44] INFO  Checking available disk space...
...
Enter backup passphrase:
...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Backup complete (follower): ./hypershield/hypershield-backup-20260603-211044-follower.tar.gz
SHA256: 8e9290d9493669e8860798cfae16ac1f220cfe94831e35c1e76d5e69d9a7ba81
...

1. Leader-only restore🔗

This section shows you how to restore a lost leader cluster from a previously made backup onto a freshly-deployed replacement cluster that uses the original cluster’s IP addressing and DNS records.

isoctl hypershield install -f /home/iep/hypershield/hypershield-install-new-leader.yaml

sudo KUBECONFIG=<leader kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type leader --force

kubectl get pods -n hubble-timescape

NAME                                            READY   STATUS      RESTARTS      AGE
chi-hubble-timescape-hubble-data-0-0-0          1/1     Running     0             46m
hubble-timescape-analyzer-8488ffcc67-mbjc4      1/1     Running     0             7m13s
hubble-timescape-gateway-5ff76557ff-2h28n       1/1     Running     0             15m
hubble-timescape-ingester-798459d5c9-tcnrx      1/1     Running     0             7m12s
hubble-timescape-migratepg-b9e9a4391b-5n4sr     0/1     Error       0             62m
hubble-timescape-migratepg-b9e9a4391b-dmkdv     0/1     Completed   0             60m
hubble-timescape-pg-1                           1/1     Running     0             60m
hubble-timescape-pg-2                           1/1     Running     0             57m
hubble-timescape-public-api-575b47bcff-gjhcn    1/1     Running     0             12s
hubble-timescape-server-cdb4c4b74-kq4qb         1/1     Running     0             7m13s
hubble-timescape-trimmer-29753340-c2xlv         0/1     Completed   0             40m
hubble-timescape-ui-6ff788668d-zdcs5            1/1     Running     0             15m
sc-server-574669d455-knpsn                      1/1     Running     7 (46m ago)   52m
sc-server-migratepg-833b59182b-7czh5            0/1     Completed   0             52m

kubectl --kubeconfig=<new leader kubeconfig> get secret \
  -n hubble-timescape sc-server-root-ca-tls \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > sc-server-ca.crt

kubectl --kubeconfig=<new leader kubeconfig> get secret \
  -n hubble-timescape hubble-timescape-server-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > leader-ca.crt

isoctl hypershield install -f hypershield-install-follower.yaml
kubectl delete job -n hubble-timescape hubble-timescape-follower-join

level=INFO msg="Follower-join completed successfully" leader-address=hs10-leader.pseudo.co:443 cluster-id=1f5723c3-80a0-426c-ac3c-e463dc01da05

kubectl delete secret -n hubble-timescape sc-client-bootstrap
kubectl rollout restart deployment -n hubble-timescape sc-client

level=INFO msg="tunnel active, serving SOCKS5 connections" subsys=connector leader=198.18.42.158:443

1.5 Validate the results in Timescape🔗

In the Timescape web UI, verify that all restored data is present and correct:

• Objects

• Policies

• Event Logs

• Followers

• Telemetry Destinations

• Logging Groups

The follower cluster should show as Active and any Smart Switches should show as Connected.

Below is a screenshot of the Objects page:

2. Follower-only restore🔗

This section shows you how to restore a lost follower cluster from a previously made backup onto a freshly-deployed replacement cluster that uses the original cluster’s IP addressing and DNS records.

kubectl --kubeconfig=<leader kubeconfig> get secret -n hubble-timescape sc-server-root-ca-tls \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > sc-server-ca.crt

kubectl --kubeconfig=<leader kubeconfig> get secret -n hubble-timescape hubble-timescape-server-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > leader-ca.crt

isoctl hypershield install -f hypershield-install-follower.yaml

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

isoctl hypershield install -f hypershield-install-follower.yaml

kubectl delete job -n hubble-timescape hubble-timescape-follower-join

Condition transition: SecureConnectorReady=True reason=SecureConnectorReady
Hypershield installed successfully

./get-token.sh hypershield smartswitch-token --ca-source secret

switch# service system hypershield register <token>

switch# run bash sudo docker exec -it HypershieldAgent bash
HypershieldAgent:/usr/src/app# rm /iox_data/mtls_certs/mtls_certificates.json

switch# run bash sudo su
bash-5.2# docker restart HypershieldAgent
bash-5.2# exit

3. Leader and follower restore🔗

This section shows you how to restore both a lost leader cluster and a lost follower cluster from previously made backups onto freshly-deployed replacement clusters that uses the original clusters’ IP addressing and DNS records.

Leader-only restore🔗

This section shows you how to restore a lost leader cluster from a previously made backup onto a freshly-deployed replacement cluster that uses new IP addressing and DNS records as compared to the original cluster.

isoctl hypershield install -f hypershield-install-leader.yaml

Condition transition: TimescapeReady=True reason=TimescapeReady
Condition transition: SecureConnectorReady=True reason=SecureConnectorReady
Hypershield installed successfully

sudo KUBECONFIG=<new leader kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type leader --force

[01:20:25] INFO  Checksums OK.
[01:20:26] INFO  Cluster reachable: mgmtcluster.isovalent.com
[01:20:38] INFO  AUDIT restore complete with k8s apply failures: count=1

4.3 Validate the results in Timescape🔗

In the Timescape web UI, verify that all restored data is present and correct:

• Objects

• Policies

• Event Logs

• Followers

• Telemetry Destinations

• Logging Groups

At this time, there are no registered Smart Switches.

Below is a screenshot of the Objects page:

4.4 Reconnect the follower🔗

For the restored leader, export fresh TLS material:

kubectl --kubeconfig=<new leader kubeconfig> get secret \
  -n hubble-timescape sc-server-root-ca-tls \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > sc-server-ca.crt

kubectl --kubeconfig=<new leader kubeconfig> get secret \
  -n hubble-timescape hubble-timescape-server-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > leader-ca.crt

Transfer the TLS material to the follower cluster host:

scp sc-server-ca.crt leader-ca.crt iep@<follower host>:follower-creds/<name>/

In Timescape, generate a new join credential (Operations → Followers).

Edit the follower cluster’s configuration file hypershield-install-follower.yaml by updating leaderAddress to the new leader cluster’s hostname, and secureConnector.serverAddress to the new leader cluster’s SecureConnector IP address.

On the follower, force the follower-join job to re-run by deleting and recreating it from the Helm release hooks (this pair of commands may need to be re-run):

isoctl hypershield install -f hypershield-install-follower.yaml
kubectl delete job -n hubble-timescape hubble-timescape-follower-join

Sample successful output:

Condition transition: SecureConnectorReady=True reason=SecureConnectorReady
Hypershield installed successfully

Still on the follower, delete the sc-client-bootstrap secret and restart sc-client:

kubectl delete secret -n hubble-timescape sc-client-bootstrap
kubectl rollout restart deployment -n hubble-timescape sc-client

Sample successful output:

level=INFO msg="tunnel active, serving SOCKS5 connections" subsys=connector leader=198.18.42.129:443

4.5 Validate the results in Timescape🔗

In the Timescape web UI, verify that all restored data is present and correct:

• Objects

• Policies

• Event Logs

• Followers

• Telemetry Destinations

• Logging Groups

The follower cluster should show as Active and any Smart Switches should show as Connected.

Below is a screenshot of the Objects page:

5. Follower-only restore🔗

This section shows you how to restore a lost follower cluster from a previously made backup onto a freshly-deployed replacement cluster that uses new IP addressing and DNS records as compared to the original cluster.

kubectl --kubeconfig=<leader kubeconfig> get secret -n hubble-timescape sc-server-root-ca-tls \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > sc-server-ca.crt

kubectl --kubeconfig=<leader kubeconfig> get secret -n hubble-timescape hubble-timescape-server-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > leader-ca.crt

isoctl hypershield install -f hypershield-install-follower.yaml

kubectl get pods -n hubble-timescape
kubectl get pods -n hubble-timescape -l app.kubernetes.io/name=sc-client

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

kubectl get cm -n hypershield smartswitch-timescape-config -o jsonpath='{.data.timescape_config}'

./get-token.sh hypershield smartswitch-token --ca-source secret

switch# service system hypershield register <token>

switch# run bash sudo docker exec -it HypershieldAgent bash
HypershieldAgent:/usr/src/app# rm /iox_data/mtls_certs/mtls_certificates.json

switch# run bash sudo su
bash-5.2# docker restart HypershieldAgent
bash-5.2# exit

6. Leader and follower restore🔗

This section shows you how to restore both a lost leader cluster and a lost follower cluster from previously made backups onto freshly-deployed replacement clusters that uses new IP addressing and DNS records as compared to the original clusters.

kubectl get cm -n hypershield smartswitch-timescape-config -o jsonpath='{.data.timescape_config}' > /tmp/ts-config.json

sed -i 's/198\.18\.42\.168/198.18.42.108/' /tmp/ts-config.json   # old follower IP -> new follower IP

kubectl patch cm -n hypershield smartswitch-timescape-config --type=merge \
  -p "{\"data\":{\"timescape_config\": $(python3 -c 'import json; print(json.dumps(open("/tmp/ts-config.json").read()))')}}"

Restore Hypershield to original IP addressing🔗

The corresponding procedural page for this section is here.

isoctl hypershield install -f hypershield-install.yaml –timeout=30m

kubectl patch tlsroute -n hubble-timescape hubble-timescape-server \
  --type=json -p='[{"op":"add","path":"/spec/parentRefs/0/sectionName","value":"grpc"}]'

sudo KUBECONFIG=<leader kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type leader --force

[01:20:31] INFO  Restoring critical secrets...
Error from server (Invalid): Secret "hubble-timescape-credential-encryption-keys" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "hubble-timescape-signing-keys" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "sc-client-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "sc-server-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set
[01:20:33] WARN  Some critical secrets in hubble-timescape may not have applied cleanly.
...
K8s apply failures: 1

kubectl --kubeconfig=<kubeconfig> delete secret -n hubble-timescape \
  hubble-timescape-credential-encryption-keys \
  hubble-timescape-signing-keys \
  sc-client-kek \
  sc-server-kek

sudo KUBECONFIG=<kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type leader --force

level=WARN msg="unable to connect to DB" error="unable to connect to database: code: 516, message: timescape_migrate: Authentication failed: password is incorrect, or there is no user with such name" attempts=2 sleep-duration=4s
level=WARN msg="unable to connect to DB" ... attempts=8 sleep-duration=10s
context deadline exceeded

hubble-timescape-*-creds Secrets, patch chi-hubble-timescape-common-usersd,

level=WARN msg="Initial TLS configuration load failed, certificate files might not yet be available, waiting on fswatcher update to become ready" config=tls-client reloadKeypairError="failed to load keypair: open /var/lib/hubble-timescape-public-api/client-tls/tls.crt: no such file or directory" reloadCAError=<nil>
error creating Timescape upstream clients: getting gRPC client options: cannot get TLS configuration: context canceled

kubectl patch deployment -n hubble-timescape hubble-timescape-public-api --type=json -p='[
  {"op":"add","path":"/spec/template/spec/volumes/-",
   "value":{"name":"public-api-client-tls","projected":{"defaultMode":256,
     "sources":[{"secret":{"name":"hubble-timescape-public-api-cert"}}]}}},
  {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/-",
   "value":{"name":"public-api-client-tls",
     "mountPath":"/var/lib/hubble-timescape-public-api/client-tls","readOnly":true}}
]'

cannot create public API RBAC auth: failed to create public API policy authorizer: policy loader has no current policy in place: multiLoader: error getting current policy from loader "config": open /etc/hubble-timescape/rbac-policy.yaml: no such file or directory

kubectl patch cm -n hubble-timescape hubble-timescape-public-api-config --type=merge \
  -p '{"data":{"config.yaml":"...policy-file: /etc/timescape-public-api-rbac/policy/hubble-timescape-public-api-rbac-policy.yaml..."}}'
kubectl rollout restart deployment -n hubble-timescape hubble-timescape-public-api

kubectl get deployment -n hubble-timescape sc-server -o json | python3 -c "
import sys, json
d = json.load(sys.stdin)
spec = d['spec']['template']['spec']; c = spec['containers'][0]
job = {'apiVersion': 'batch/v1', 'kind': 'Job',
  'metadata': {'name': 'sc-server-migratepg-manual', 'namespace': 'hubble-timescape'},
  'spec': {'template': {'spec': {'restartPolicy': 'Never',
    'containers': [{'name': 'migratepg', 'image': c['image'],
      'command': ['/usr/local/bin/sc-server'], 'args': ['migratepg'],
      'env': c.get('env', []), 'volumeMounts': c.get('volumeMounts', [])}],
    'volumes': spec.get('volumes', []), 'serviceAccountName': spec.get('serviceAccountName', '')}}}
print(json.dumps(job, indent=2))" | kubectl apply -f -

kubectl wait job/sc-server-migratepg-manual -n hubble-timescape --for=condition=complete --timeout=60s

kubectl rollout restart deployment -n hubble-timescape sc-server

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

The Secret "sc-client-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set

kubectl --kubeconfig=<follower kubeconfig> delete secret -n hubble-timescape sc-client-kek

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

Restore Hypershield to new IP addressing🔗

The corresponding procedural page for this section is here.

sudo KUBECONFIG=<leader kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type leader --force

Error from server (Invalid): Secret "hubble-timescape-credential-encryption-keys" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "hubble-timescape-signing-keys" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "sc-client-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set
Error from server (Invalid): Secret "sc-server-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set

kubectl --kubeconfig=<kubeconfig> delete secret -n hubble-timescape \
  hubble-timescape-credential-encryption-keys \
  hubble-timescape-signing-keys \
  sc-client-kek \
  sc-server-kek

timescape_migrate: Authentication failed: password is incorrect, or there is no user with such name
open /var/lib/hubble-timescape-public-api/client-tls/tls.crt: no such file or directory
policy loader has no current policy in place ... open /etc/hubble-timescape/rbac-policy.yaml: no such file or directory

CURRENT=$(kubectl get cm -n hubble-timescape hubble-timescape-ui-config -o jsonpath='{.data.config\.yaml}')

FIXED=$(echo "$CURRENT" | sed 's|redirect-url: https://hs10-leader-ui.pseudo.co/oauth2/callback|redirect-url: https://test-leader-ui.pseudo.co/oauth2/callback|')

kubectl patch cm -n hubble-timescape hubble-timescape-ui-config --type=merge \
  -p "{\"data\":{\"config.yaml\": $(echo \"$FIXED\" | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')}}"

kubectl rollout restart deployment -n hubble-timescape hubble-timescape-ui

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

The Secret "sc-client-kek" is invalid: data: Forbidden: field is immutable when `immutable` is set

kubectl --kubeconfig=<follower kubeconfig> delete secret -n hubble-timescape sc-client-kek

sudo KUBECONFIG=<follower kubeconfig> ./hs-restore.sh --file <backup.tar.gz> \
  --mode rollback --appliance-type follower --force

General health check🔗

Invoke these commands on the management VM:

kubectl get nodes                          # all Nodes 'Ready'
kubectl get pods -n hypershield            # all Hypershield Pods 'Running', no restarts accumulating
kubectl get pods -n hubble-timescape       # all Timescape Pods 'Running'
kubectl get smartswitches -n hypershield   # all switches 'Connected'
etcdctl endpoint health --cluster          # all etcd members healthy

Timescape UI is not reachable🔗

1. Confirm that the Timescape Pod is running:

   kubectl get pods -n hubble-timescape
   

2. Confirm that the Load Balancer IP address is assigned:

   kubectl get svc -n hubble-timescape
   

3. Test connectivity to the Load Balancer IP address:

   curl -sk https://<timescape-lb-ip>/healthz
   

4. If the Timescape login page loops back to Keycloak without granting access, verify the Keycloak realm configuration:

   • realm name must be exactly: hypershield

   • client ID must be exactly: timescape

   • redirect URI must match your Timescape FQDN

API returns 401 or 403🔗

401 - the token is missing or was issued with the wrong audience

• re-acquire token with: client_id=timescape

• decode your JWT at jwt.io and confirm: aud contains “timescape”

403 - the assigned Keycloak group provides insufficient privileges

• check Keycloak group membership for the account

• group policy_manager or admin is required for CRUD operations

• a read-only role gets 403 on any mutation

Compare policy intent to DPU state🔗

There is an important distinction between:

• policy intent - what you have declared in SmartSwitchNetworkPolicy objects

• DPU state - what is actually programmed into the enforcement engine

Ideally, these are identical. However, when troubleshooting unexpected permit or deny behavior, it is worth verifying:

To view policy intent, run these commands on the management VM:

kubectl get smartswitchnetworkpolicies -n hypershield
kubectl describe smartswitchnetworkpolicy <policy-name> -n hypershield

To view DPU state, run this command on the switch console:

HypershieldAgent:/usr/src/app# ./agwctl policies show

If the two views diverge, a policy has not been pushed to the DPU. Check the switch’s connection status and controller logs:

HypershieldAgent:/usr/src/app# ./agwctl show_status
kubectl logs -n hypershield deployment/hypershield-controller --tail=100 | grep <switch-name>

Policy not applied🔗

Possible symptoms: SmartSwitchNetworkPolicy shows an error status

Run the below commands.

On the management VM:

kubectl describe smartswitchnetworkpolicy -n hypershield <policy-name>

On the switch console:

HypershieldAgent:/usr/src/app# ./agwctl show_log

Common causes:

Traffic is blocked that should be allowed🔗

1. Confirm the policy exists on the controller. On the management VM:

   kubectl get smartswitchnetworkpolicies -n hypershield
   

2. Confirm the rule reached the DPU. Within the AGW shell:

   HypershieldAgent:/usr/src/app# ./agwctl policies show
   

   If the rule is absent, check switch connectivity:

   HypershieldAgent:/usr/src/app# ./agwctl show_status
   

   The output must show Connected=true.

3. Confirm the traffic matches the policy rules exactly:

   • the source IP address must fall within the policy’s source CIDR

   • the destination IP address must fall within the policy’s destination CIDR

   • the protocol and port must match exactly (no wildcards)

   • VRF of the traffic must match the VRF steered to the DPU

4. Confirm the correct VRFs are steered to the DPU:

   switch# show service-acceleration status details
   

   VRF count should include the VRF the traffic traverses

Analyze dropped network packets🔗

Traffic may be blocked or allowed unexpectedly. That is, traffic does not appear to match policy. In these cases, one strategy is to analyze dropped network packets.

1. Check that the policy is on the DPU:

   dpu1# dpctl hs policies show
   

2. Enable drop logging:

   dpu1# dpctl hs logging drop start
   

3. Generate some test traffic:

4. Check the capture:

   dpu1# dpctl hs logging drop show
   

   Ensure Total Unique Packets has incremented.

5. Stop the capture and inspect it:

   dpu1# dpctl hs logging drop stop
   dpu1# dpctl hs logging drop write
   less /var/log/pensando/pkt.log
   

VRF resolution errors🔗

Possible symptoms: policy shows the VRF not found error

Run the below commands on the switch console.

Check which VRFs are available on the AGW:

HypershieldAgent:/usr/src/app# ./agwctl vrf list

See other VRFs that are not “in service” of the AGW:

HypershieldAgent:/usr/src/app# ./agwctl vrf show

Check VRFs on a switch console:

switch# show vrf

Switch shows status Pending or Disconnected🔗

1. If registration of the switch was very recent, wait at least 60 seconds - the initial TLS handshake can take some time.

2. Check network reachability from the controller to the switch’s source-interface address:

   kubectl exec -n hypershield <controller-pod> -- \
      curl -sk https://<switch-source-interface-ip>/
   

3. Confirm the source-interface address is configured on the switch:

   switch# show interface loopback <N>
   

4. If the switch was previously Connected and is now Disconnected, check for interface events on the switch:

   switch# show logging last 100 | include eth1/48
   

General health check🔗

Invoke the below commands on the management VM.

Check the status of all switches:

kubectl get smartswitches -n hypershield

Check the security policy status:

kubectl get smartswitchnetworkpolicy -n hypershield

Invoke these commands from the AGW shell:

HypershieldAgent:/usr/src/app# ./agwctl show_status     # Connected=true, Phase: fw-ready|redir-done
HypershieldAgent:/usr/src/app# ./agwctl show_ha         # Cluster State: active/active, all criteria OK
HypershieldAgent:/usr/src/app# ./agwctl policies show   # Rules present (at minimum, the default-deny rule)

AGW health check🔗

Inspect the agent container (AGW) by invoking commands on the switch console.

switch# show service-acceleration status detail | inc "Agent Health"

switch# run bash sudo docker ps | grep agw

d902a5237ee7   isovalent/agw:v1.19.0-pre.8-151   "/usr/src/app/init.sh"   2 weeks ago   Up 2 weeks (healthy)             HypershieldAgent

switch# run bash sudo docker exec -it HypershieldAgent ./agwctl show_dpu

lastPing   healthy   dpu   uid             hardware   agent                         datapath   dpuReboot   policySync
1s         true      1     169.254.151.1   giglio     v1.19.0-pre.8-150-ge2ba52cc   1.150.2    13          true
1s         true      2     169.254.159.1   giglio     v1.19.0-pre.8-150-ge2ba52cc   1.150.2    13          true

DPU health checks🔗

Various checks to perform on a DPU:

dpu1# dpctl hs policies show               # loaded policies
dpu1# dpctl show flow                      # detailed flow table dump
dpu1# pdsctl show port statistics          # per-port counters and errors
dpu1# dpctl show pipeline statistics drop  # pipeline drop statistics (check for errors)

Switch not discovered🔗

Possible symptom: a SmartSwitch CR remains in a pending state

Run these commands on the management VM.

kubectl describe smartswitch -n hypershield <switch-name>
kubectl logs -n hypershield deployments/hubble-cpc | grep <switch-name>

AGW not connected to control plane🔗

Possible symptoms:

• a Smart Switch CR is not created or remains in a pending state

• a Smart Switch does not receive updated policies or configurations

• policy changes on the control plane is not reflected on DPUs

An ordered series of diagnostic steps are recommended.

Perform all commands on a switch console.

switch# service system hypershield test controller connection

==============================
Running config checks
==============================
Using source-interface loopback0 for controller connection
Controller connection status: [success]
Controller reason: [connected ok with Hypershield controller]
Controller endpoint: 10.62.156.185:6443

=================================================
Starting network connectivity checks on the switch
=================================================

Not using proxy for controller connection

Using curl to check connection to 10.62.156.185 port 6443
Curl successfully connected to 10.62.156.185:6443.

switch# run bash sudo docker exec -it HypershieldAgent ./agwctl show_tokens

k8s_controller_url=https://10.62.156.185:6443
k8s_service_account=smartswitch
k8s_namespace=hypershield
k8s_token=<token>

switch# run bash sudo docker exec -it HypershieldAgent netstat -an | grep ESTABLISHED | grep 6443

tcp        0      0 127.254.254.2:46018     10.62.156.185:6443      ESTABLISHED

switch# run bash sudo docker logs HypershieldAgent --tail 100

DPU issues🔗

Diagnostic commands to perform on a switch console:

switch# show dpu internal info
switch# show dpu internal msgs
switch# show dpu internal event-history errors
switch# show dpu internal event-history msgs

Capacity issues🔗

switch# run bash sudo docker exec -it HypershieldAgent ./agwctl show_logs | grep -i "exceeded"

dpu1# dpctl show pipeline statistics drop

dpu1# dpctl hs policies show --summary

Performance issues🔗

Invoke the below commands from within a DPU shell or the switch console directly.

switch# top

dpu1# pdsctl show port statistics

dpu1# dpctl show pipeline statistics drop

dpu1# dpctl hs logging drop start

dpu1# dpctl hs logging drop status
dpu1# dpctl hs logging drop summary

dpu1# dpctl hs logging drop stop

dpu1# dpctl hs logging drop write --format json --file /tmp/drops.json
switch# cat /tmp/drops.json | jq '.entries[] | {src: .source_ip, dst: .dest_ip, policy: .policy_id}'

Upgrade issues🔗

switch# install add bootflash:agent-1.19.0.pre.8.273-10.6.3q.x86_64.rpm activate upgrade

Adding the patch (/agent-1.19.0.pre.8.273-10.6.3q.x86_64.rpm)
[####################] 100%
Install operation 16 failed because this package already exists. at Thu Jun  4 23:04:34 2026

switch# install activate agent-1.19.0.pre.8.273-10.6.3q.x86_64.rpm

Groups-RPM Database                              58 MB/s |  60 kB     00:00
Local RPM Database                              1.5 MB/s | 1.5 kB     00:00
Local RPM Database                              1.4 MB/s | 1.4 kB     00:00
Thirdparty RPM Database                         1.5 MB/s | 1.5 kB     00:00
Groups-RPM Database                             2.2 MB/s | 2.3 kB     00:00
[####################] 100%
Install operation 17 completed successfully at Thu Jun  4 23:07:43 2026

Verify Cilium configuration🔗

kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config --all | grep EnableL2Announcements

EnableL2Announcements             : true

kubectl delete pod -l app.kubernetes.io/part-of=cilium -n kube-system

kubectl get gatewayclasses.gateway.networking.k8s.io cilium

NAME     CONTROLLER                     ACCEPTED   AGE
cilium   io.cilium/gateway-controller   True       4m

Collect control plane data🔗

Access the management VM and run the following:

isoctl sysdump \
  --kubeconfig /home/iep/hypershield/control-plane.yaml \
  --hypershield \
  --output-filename isoctl-sysdump-hypershield-control-plane-2026-10-31

The above captures the following:

• Pod logs

• CRD state

• CNPG diagnostics

• Hypershield-specific telemetry

The command places all this information into a single file named isoctl-sysdump-hypershield-control-plane-2026-10-31 .

Collect switch data🔗

Access an affected switch and run the given commands on its console. There are two data bundles to collect from a switch.

First bundle:

switch# show tech-support service-acceleration > bootflash:show-tech-svc-accel.txt

The above captures:

• AGW logs

• DPU logs

• core dumps

• end-to-end diagnostic state

The command places all this information into a single file named bootflash:show-tech-svc-accel-2026-10-31.txt .

Second bundle:

switch# show tech-support dpu module all > bootflash:show-tech-dpu-all.txt

The above captures:

• DPU firmware versions

• hardware health

• memory utilization

• per-module operational state

The command places all this information into a single file named bootflash:show-tech-dpu-all-2026-10-31.txt .

Provide the output files🔗

You should now have three files:

• isoctl-sysdump-hypershield-control-plane-2026-10-31

• bootflash:show-tech-svc-accel-2026-10-31.txt

• bootflash:show-tech-dpu-all-2026-10-31.txt

Provide these files to the Cisco TAC team.