Security Patches for Crosswork Network Controller

This chapter contains the following topics:

Security patches for OS and core components

A security patch or operating system patches are essential updates designed to protect Crosswork Network Controller from vulnerabilities, improve system performance, and maintain compatibility with the latest platform requirements. These updates are applied with minimal disruption, targeting one node at a time to ensure system availability.

Security patches for the Crosswork Network Controller are offered as tar bundles. Each tar bundle contains packages in a unique format specific to Crosswork, known as SPACK (System Package). The naming convention for these tar bundles is as follows:cw-na-spack-7.1.<version>-<release>-<date>.tar.gz

Supported patch types within the SPACK bundle

  • OS security updates

  • Kubernetes security updates

  • Container Runtime Interface security updates

  • Calico security updates

  • Robot Orchestrator pod upgrade

  • Custom binaries such as vmexec

The patching process supports base VM patches and includes inline patching support for OS, Kubernetes, Calico, and Container Runtime Interface security patches.

Security patch update workflow

The patching process facilitates the update of packages on the base VM and other core components. This is a series of steps in the security patch update workflow:

  1. Backup:It is advised to take a backup before starting the patch update procedure to ensure data safety in case of unforeseen issues.

  2. Maintenance mode:Maintenance mode will be activated automatically before the patching procedure begins and deactivated after the patching is completed. Alarms will be raised to indicate the activation and deactivation of maintenance mode.

  3. Patch application order:

    • Each SPACK bundle contains one or more updates (e.g., OS, Kubernetes, Calico, etc.), and these patches are applied in the order defined in the bundle manifest.

    • Patches are applied to all nodes sequentially, starting with the worker nodes and then proceeding to the hybrid nodes, targeting one node at a time to ensure system availability.

  4. Job tracking:

    • After applying security patches, individual jobs are created in the System Management > Job History window for each node in the cluster.

    • Operations performed on each node can be reviewed in the Job Details page to understand the actions being executed.

  5. Handling failed jobs: If a job fails on a particular node, you can retry the job. The retry process will start from the failed node while ignoring nodes that were successfully patched.

  6. Adding new worker nodes: If a new worker node is added after the patching process, patches must be manually applied to that node. The system does not automatically apply patches to newly added nodes.

  7. Estimated time for completion: It typically takes 3 to 4 hours to apply patches on a 5-node cluster. It is recommended to schedule all upgrades during a maintenance window to minimize disruptions.

  8. Verification: Once all jobs on all nodes are completed successfully, you can verify the updated package list by navigating to Administration > Crosswork Manager > System Summary > <Node-Name> > View details > Package details tab.


Important

  • All packages are bundled and do not require any external connectivity to fetch packages or configurations.

  • The security patch update workflow applies to both clusters and single VM setups. For Geo HA (High Availability) setups, patches must be applied to both the active and standby sites.

  • There may be node restarts during the patch operation.

  • Rollback is not supported. If an issue occurs, you should retry the patch process or use an updated system patch bundle with fixes.

Add a security OS patch

This section explains how to add and install an OS patch.

Before you begin

Before installing the patch, verify that all inventory details are visible in the System summary page (Administration > Crosswork Manager > System summary). Patch installation will fail if this information is missing. page.

Procedure

Step 1

From the main menu, choose Administration > Crosswork Manager. The Crosswork Summary page is displayed with Crosswork Cluster and Crosswork Platform Infrastructure tiles.

Step 2

Click the System management tab.

Step 3

Click Add OS patch, and the Add OS Patch popup windowis displayed. Choose either URL or SCP as your preferred protocol. Based on your selection, fill in the additional fields with the required information. Click Add to proceed.

Individual jobs are created to add the patch to repository, and to apply the package to each VM in your cluster.

Step 4

Click System Management > Job History window to monitor each job created. You can review operations performed on each node in the Job Details page to understand the actions being executed.

The newly added OS patch will be displayed under the System updates tab.

Step 5

You can upgrade an already installed OS patch by using the Upgrade button displayed under the System updates tab.

Step 6

(Optional) To view details of an OS patch, select the patch file and click Package details.

The Package details drawer panel will be displayed with details such as package name, version and description.

Events are generated for system patch success or failure:

  • Success: 
    System package <version> update completed for <node name>

  • Failure:

    System package <package name> update failed for <node name>
    System update failed due to <reason>, please fix the system and retry the operation again

Step 7

(Optional) Once all jobs on all nodes are completed successfully, you can verify the updated package list by navigating to Administration > Crosswork Manager > System Summary > <Node-Name> > View details > Package details tab.

The Package details tab displays information such as package name, current version, and description.

Crosswork Data Gateway patches and patch management

Patch management in Crosswork Data Gateway refers to the process of applying minor updates or fixes to the Data Gateway base virtual machine to ensure optimal performance, reliability, and security.

While patching focuses on maintaining the integrity and functionality of the base VM, the upgrade and management of other components, such as collectors, are handled separately by Crosswork Network Controller. These updates are applied with minimal disruption to data collection activities and are managed through a rolling upgrade process to ensure system availability.

Supported patch types

The patching process facilitates base VM upgrades and applies security patches. It also ensures that newly enrolled Data Gateway VMs automatically receive the applicable patches.

How Data Gateway upgrade works

The patching process facilitates base VM upgrades and applies security patches. It also ensures that newly enrolled Data Gateway VMs automatically receive the applicable patches.

Data Gateway upgrade scenarios

Rolling upgrades support various scenarios to ensure minimal service disruption and prevent multiple Data Gateway failovers:

  • Geo redundancy-enabled: The upgrade process considers geo redundancy setups, ensuring that Data Gateways in active sites are upgraded in a controlled order, while Data Gateway in standby sites can be upgraded in parallel.

  • HA pools: The upgrade process can handle different HA pool configurations, including those with no spares, partial protection (for example, 2 Active, 1 Spare), and full protection.

  • Spare Data Gateway patch failure: Crosswork Network Controller can handle scenarios where a spare Data Gateway fails during the patch process, pausing the rolling upgrade until the issue is resolved or a new spare is added.

  • Assigned Data Gateway patch failure: Crosswork Network Controller provides recovery options when an assigned Data Gateway fails during the patch process, including recovering the Data Gateway or deploying a new Data Gateway as a spare.

Precautions during Data Gateway upgrade

Important: Parameter mismatch or stack name reuse not allowed

  • Use a unique stack name for each deployment.

  • Ensure that updated parameters such as VM Node IP addresses, ENI ID, Subnet ID, and Security Group, in the deployment template match those used previously.

Apply patch upgrades to Data Gateway

With the patch upgrade process, minor updates or patch releases are applied to the Crosswork Data Gateway base VM with the less disruptive upgrade process.

Before you begin

Complete these requirements before you start the Data Gateway base VM upgrade:

  • Review the available patches and obtain the applicable Data Gateway patch file from cisco.com.

  • Schedule the patching during a maintenance window to minimize impact on network operations.

  • Understand that the upgrade procedure applies only to the Data Gateway base VM. Crosswork Network Controller automatically upgrades other components, such as collectors.

  • Read the precautions that you must take when performing minor upgrades. For more information, see Precautions during Data Gateway upgrade.

  • Ensure that you have the Crosscluster Infra API permission to access the Data Gateway Management page. For details on user roles and permissions, see the User Roles, Functional Categories, and Permissions section in the Cisco Crosswork Network Controller 7.1 Administration Guide.

Procedure

Step 1

From the main menu, choose the Administration > Crosswork Manager > Application Management tab.

Step 2

Click Add new file > Upload application bundle (.tar.gz).

The Add Application Bundle (.tar.gz) (.tar.gz) dialog box appears.

Step 3

In the Add Application Bundle (.tar.gz) (.tar.gz) dialog box

  • select the protocol as URL or SCP to access the installation file

  • enter the location (path or address) where the installation file is stored

  • select the Automatically clean all repository files before adding new one check box to remove any existing repository files before adding new ones

  • select the Basic Auth check box if the file source requires authentication, and

  • select the Automatically clean all repository files before adding new one check box.

Step 4

Click Add.

A message appears confirming that the patch installation has started.

After the upgrade is complete:

  • The Data Gateway VMs are enrolled with Crosswork Network Controller.

  • All the destinations, pools, device-mapping information is available on Crosswork Network Controller UI.

  • The upgrade Data Gateway VMs start collecting data from the devices.

What to do next

  • If you want to move Cisco NSO out of the maintenance mode, use the ncs_cmd -c maapi_read_write command on the Data Gateway VM.

  • If you delete a Data Gateway from the Cisco Crosswork Network Controller after a successful upgrade and re-enroll it later (for example, during maintenance), the patch upgrade status will not be displayed. To re-enroll the Data Gateway, see the Re-enroll Crosswork Data Gateway sections in the Cisco Crosswork Network Controller 7.1 Administration Guide.

  • Monitor the progress of the installation process. See Monitor the Data Gateway patch status.

  • Review the events for issues.

Monitor the Data Gateway patch status

Monitoring the upgrade process is essential to ensure a smooth, reliable, and recoverable patch deployment.

Follow one of these methods to monitor the progress of the security patch installed on Data Gateway VM.

Procedure

Step 1

Using the Data Gateway Management page.

  1. Go to the Data Gateway Management page. The Data Gateway currently being upgraded will appear with the Admin state status as in progress.

  2. In the Admin state column, hover over the info icon next to the Data Gateway to view details about the package being applied.

    A banner also appears on the Data Gateway Management page indicating that the Data Gateway is being upgraded to a specific version.

  3. Go to the Data Gateway Management page, click the Data Gateway that is being upgraded. The Data Gateway details page opens.

Step 2

Using the Events page.

  1. Go to the Data Gateway Management page, click the Data Gateway that is being upgraded. The Data Gateway details page opens.

  2. Go to the Events pane and review the details, including operational state changes, role changes, a message indicating the reason for the status change, timestamp, and duration.

Patch progress and status tracking

During the patching process, the Crosswork Network Controller UI displays a banner that helps you monitor the upgrade status of each Data Gateway.

The banner provides information about:

  • Patch status at Data Gateway level: Track the progress of each individual Data Gateway as the patch is applied.

  • Overall patch status: View the complete upgrade timeline, from when the DG-manager starts processing the patch to when all Data Gateway VMs are successfully upgraded.

  • Container image version details: View the previous and current image versions for each container updated by the patch on each Data Gateway.

Figure 1. Patch status

Verify the patch upgrade status

Follow one of these methods to confirm that the security patches were applied to the Data Gateway VM.

Procedure

Step 1

Using the Data Gateway Management status.

  1. Go to the Data Gateway Management page.

  2. Confirm that the operational state of the upgraded Data Gateway status is in the UP state.

  3. Verify that the upgrade banner is no longer visible. Its disappearance indicates that the patch process is completed.

Step 2

Using the patch upgrade details.

  1. Go to the Data Gateway Management page and hover over the upgraded node.

    The Enrollment Details drawer opens.

  2. In the drawer, go to the Patch Upgrade Details pane.

  3. Review these values to verify the patch application:

    • Status

    • Installation version, and

    • Build date.