Node removals | Manage the Crosswork Network Controller VMs | Cisco Crosswork Network Controller 7.2 Administration Guide

Understand the behaviors, limits, and availability impact associated with removing hybrid and worker nodes from a Crosswork cluster.

Node removals are cluster maintenance operations that

allows administrators to delete failed or healthy nodes from a Cisco Crosswork cluster,
eliminates the node reference from the Crosswork Network Controller cluster, and
deletes the node from the host VM.

Node removal behaviors and limits

This topic lists the supported limits, expected effects, and actions associated with removing hybrid and worker nodes in the system.

Supported node roles and limits

Hybrid nodes: The system must maintain three operational hybrid nodes at all times to ensure high availability (HA) and system protection. If one of the hybrid nodes stops functioning, Crosswork will attempt to compensate, but performance and resilience against further failures will be severely impacted. In such cases, the faulty node must be erased, and a new hybrid node should be deployed to replace it.
Worker nodes: You can have up to two worker nodes. Both worker nodes can be erased without immediate consequences, but it is recommended to erase and replace them one at a time.

Effects of hybrid node removal

When a hybrid node is removed (either through an erase operation or directly from the backend), the following effects are observed:

Remaining hybrid nodes display a "degraded" status, indicating high availability (HA) is lost.
A further node failure could cause operational issues.
Alarms are generated, and you are expected to restore the down node. Three functioning hybrid nodes should always be present.
Several pods may enter the "Pending" state. This is expected because some critical infrastructure services, which run as three instances for maximum HA, are pinned to specific hybrid nodes.
- Examples of services in the "Pending" state: cw-ftp, cw-sftp, nats, robot-etc, robot-kafka, and tyk.
Some pods may remain pending due to being configured as DaemonSet.
Once the down hybrid node is restored, the system returns to normal and pending issues are resolved.

Effects of worker node removal

Up to two worker nodes are supported.
Both can be erased without immediate system impact.
It is recommended to erase and replace worker nodes one at a time.

Note
When a Worker node is removed while a vCenter alarm on that VM requires user acknowledgement, the node is deleted from the Crosswork Network Controller UI but not from the backend, causing the total count in the UI to remain incorrect and leaving the VM in vCenter. This stale backend entry can also cause new Worker node additions to fail with a duplicate-IP error. To clean up the stale entry, run this command:
robotctl remove-node-from-inventory <node-ip>
.

Manual cluster installation requirements

For manual cluster installations, you must erase the VM from the Crosswork UI and then delete it from the data center (for example, from vCenter).

Troubleshooting and escalation

If you continue to experience issues after performing these steps, contact the Cisco Customer Experience team for assistance.

Remove a node

Remove a node from Crosswork Network Controller.

Use this task to permanently erase a VM node in Crosswork Network Controller. This operation is disruptive and should be performed during a maintenance window.

Before you begin

Erasing a node can disrupt services and block certain processes until the action completes. Perform this operation during a scheduled maintenance window.
Removing worker or hybrid nodes increases the load on remaining nodes and may impact system performance. Contact Cisco Customer Experience before removing nodes.

Follow these steps to erase a node:

Procedure

	1.	From the main menu, choose Administration > Crosswork Manager > System summary.
	2.	On the VM node you want to remove, click and select View details.
	3.	Click and select Erase VM node.
	4.	On the dialog prompt, click Erase to confirm the action. Note During the removal of a hybrid or worker node, the Crosswork Network Controller UI may become temporarily unreachable for a short duration due to the relocation of the `robot-ui` pod to another node. A removed node will continue to be visible in the Grafana dashboard as an entry with only historical data.

The selected node is erased and removed from active management in Crosswork, but remains in Grafana as a historical entry.

What to do next

Review cluster performance and update operational procedures to account for the removed node.