Increasing CPU Cycle Reservation for Orchestrator VMs
Cisco ACI Multi-Site Orchestrator VMs require a certain amount of dedicated CPU cycles. While new deployments apply CPU cycle reservation automatically, if you upgrade the Orchestrator from a release prior to Release 2.1(1), you will need to update each Orchestrator VM's settings manually.
Configuring appropriate CPU cycle reservation can resolve or help prevent a number of seemingly random issues, such as:
Orchestrator GUI items requiring one or more retries to load.
One or more nodes changing to
Unknownstatus and then resolving back to
Readysome time later on its own:
# docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION t8wl1zoke0vpxdl9fysqu9otb node1 Ready Active Reachable 18.03.0-ce kyriihdfhy1k1tlggan6e1ahs * node2 Unknown Active Reachable 18.03.0-ce yburwactxd86dorindmx8b4y1 node3 Ready Active Leader 18.03.0-ce
Transient heartbeat misses in the Orchestrator logs (located in
/var/log/messages) with the following example log entries:
node2 dockerd: [...] level=error msg="agent: session failed" backoff=100ms error="rpc error: code = Canceled desc = context canceled" module=node/agent [...] node2 dockerd: [...] level=error msg="heartbeat to manager [...] failed" error="rpc error: code = Canceled desc = context canceled" [...]
To update the CPU cycle reservation setting, repeat the following steps for each Orchestrator VM:
Log in to the vSphere client.
Navigate to the ESX host where your Orchestrator VM is located.
Shut down the VM.
Right click the VM and choose Edit Settings
In the Virtual Hardware tab, expand the CPU category.
In the Reservation field, enter
Click OK to save the changes.
Power on the VM and wait for the Orchestrator cluster to stabilize with all nodes healthy.