System

This section contains the following topics:

Architecture overview

The Crosswork Workflow Manager architecture is a microservice-based solution that operates on top of the Kubernetes container orchestration system. This section shows a diagram presenting its core architectural components along with short descriptions of each.


Architecture diagram

  • User Interface (UI): allows operators to add and instantiate workflows, enter workflow data, list running workflows, monitor job progress. The Admin section of the UI enables adding workers, managing worker processes and assigning activities from adapters to workers.

  • REST API: includes all interaction with the CWM application: deploying adapters, publishing and instantiating workflows, managing workers, resources and secrets.

  • Control Server: dispatches API requests to relevant microservices.

  • Workflow Engine: it is the core component that conducts how workflows are handled; it interprets and manages the execution of workflow definitions.

  • Execution Engine (Workflow Worker): it is responsible for executing the workflow tasks. It receives the workflow tasks from the Workflow Engine, executes them in the correct order, and sends the results back to the Workflow Engine.

  • Adapter Workers: they are processes responsible for executing the tasks defined in workflow definitions and adapter code. They receive the tasks from the Workflow Worker, execute them, and send the results back to the Workflow Worker. The Execution Workers are capable to load additional adapters as plugins, which allows them to work with different systems and technologies.

  • Adapters: they interface and integrate with external systems, applications and technologies. Inside them, activities that can be consumed in a workflow are defined.

  • Adapter SDK: a Software Development Kit that helps developers create new adapters to integrate with external systems.

  • Workflow Definitions: workflow code written in the JSON format based on the Serverless Workflow specification.

  • K8s Infrastructure: runtime platform for the CWM application. It is a collection of services that provide the necessary infrastructure to support the deployment and management of the application within a Kubernetes cluster.

  • PostgreSQL: it is the database used by the system to store and manage its data.

Upgrade CWM

You can upgrade the CWM platform on your virtual machine to a newer version using a CWM service pack with the tar.gz extension. Here's how to do it.

Prerequisites

Procedure

CWM running on vSphere vCenter 6.7+ VM with an ESXi 6.7+ host.

Get CWM service pack

To get a service pack:

Procedure

Step 1

Go to the Cisco CWM SharePoint page and in the search bar, type in 'cwm', then select a version of the service pack from the list.

Step 2

Click the Download button on the command bar.

Copy service pack file to remote host

To be able to install the service pack:

Procedure

Copy the file into the /tmp catalog of your remote host. Open a command-line terminal on your local machine and run the following command: 

scp -P <your_port_number> -i ~/.ssh/id_rsa ~/Downloads/<servicepackname>.tar.gz nxf@<your_resource_pool_address>:/tmp

Remember to replace placeholders with your actual port number, location of ssh key and service pack file, and IP address of your CWM instance.

Install service pack

Install the service pack using command-line access to the VM:

Procedure

Step 1

Using a command-line terminal, log in to the NxF on your guest OS with SSH:

ssh -o UserKnownHostsFile=/dev/null  -p <your_port_number>  nxf@<your_resource_pool_address>
Optional: If you are logging in for the first time, provide the path name for your private key:
ssh -i <your_ssh_private_key_name_and_location> nxf@<your_resource_pool_address>

Step 2

To install the service pack, go to the /tmp catalog and from there, run the following command: 

sedo service install <servicepackname>.tar.gz
Remember to replace the placeholder with your actual service pack file name.

After successful installation, your system is upgraded to a newer version.

Upgrade adapters

With each new version of CWM, you need to upgrade cwm-sdk and your adapter to a higher version of Golang. For CWM 1.2, the go version is 1.22.0 or higher. Follow the steps below to upgrade your adapter:

Before you begin

To perform the cwm-sdk and adapter upgrade, you need to have your Golang version upgraded to at least 1.22.0. To check your current version of go, in a terminal, run go version. See the go.dev downloads section for latest packages.

For cwm-sdk

Procedure

Step 1

In a terminal, go to your local cwm-sdk directory.

Step 2

Check out on the main branch and pull the latest changes: run git checkout main, then git pull.

Step 3

Run make clean build to build the application using go 1.22.

For adapter

Procedure

Step 1

In a terminal, go to your main adapter directory.

Step 2

Run cwm-sdk upgrade-adapter -cwm-version 1.2.

Step 3

Then, run cwm-sdk create-installable.

If you have an adapter and its dedicated worker already existing in the system, remember to delete the worker first, and then delete the adapter. Now you're all set to upload and install the freshly generated adapter tar.gz using CWM UI or API.

Backup and restore

Backup and restore allow you to keep your CWM data (workflow definitions, job executions, adapters, workers, resources and secrets, and events) in case of system failure or in case you need to move your data to another instance of CWM.

Prerequisites

Procedure

CWM running on vSphere vCenter 7.0 VM with an ESXi 7.0 host.

Create backup

Create the database backup using command-line access to the VM:

Procedure

Step 1

Using a command-line terminal, log in to the NxF on your guest OS with SSH:

ssh -o UserKnownHostsFile=/dev/null  -p <your_port_number>  nxf@<your_resource_pool_address>
Optional: If you are logging in for the first time, provide the path name for your private key:
ssh -i <your_ssh_private_key_name_and_location> nxf@<your_resource_pool_address>

Step 2

To create a new backup, run the following command:

sedo backup create full backup

Step 3

Check if your backup has been created successfully by running:

sedo backup list

Example result (note that the newly created backup appears at the bottom of the list):

│ NAME                                                     │ TIME                                    │ SIZE                        │ TYPE  │ HOSTNAME   │ POSTGRES VERSION │
├───────────────────-──────────────────────────────────────┼─────────────────────────────────────────┼─────────────────────────────┼───────┼────────────┼──────────────────┤
│ base_000000010000000200000087                            │ 2024-07-12 02:00:00.490634918 +0000 UTC │ 7.5 kB (47 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_0000000100000002000000B0                            │ 2024-07-13 02:00:00.265086889 +0000 UTC │ 7.5 kB (48 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_0000000100000002000000D9                            │ 2024-07-14 02:00:00.140319108 +0000 UTC │ 7.5 kB (48 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_000000010000000300000002                            │ 2024-07-15 02:00:00.239989995 +0000 UTC │ 7.5 kB (48 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_00000001000000030000002B                            │ 2024-07-16 02:00:00.239447116 +0000 UTC │ 7.5 kB (48 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_000000010000000300000054                            │ 2024-07-17 02:00:00.230182886 +0000 UTC │ 7.6 kB (48 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_00000001000000030000007D                            │ 2024-07-18 02:00:00.270256042 +0000 UTC │ 7.6 kB (49 kB Uncompressed) │ full  │ postgres-0 │ 150006           │
│ base_000000010000000300000084_D_00000001000000030000007D │ 2024-07-18 06:00:00.185023069 +0000 UTC │ 411 B (1.1 kB Uncompressed) │ delta │ postgres-0 │ 150006           │
│ base_00000001000000030000008E_D_000000010000000300000084 │ 2024-07-18 12:00:00.242624998 +0000 UTC │ 668 B (2.4 kB Uncompressed) │ delta │ postgres-0 │ 150006           │
│ base_000000010000000300000090                            │ 2024-07-18 12:13:47.503748893 +0000 UTC │ 7.5 kB (49 kB Uncompressed) │ full  │ postgres-0 │ 150006

Step 4

Download the backup file to /tmp catalog on your VM. 

cd /tmp
sedo backup download <backup_name>

Restore database

To export the database to a different CWM instance using the restore feature, follow the steps below.

Procedure

Step 1

Using a command-line terminal, copy a backup file from your CWM instance to your local machine:

scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -P <ssh_port> -i ~/.ssh/id_ed25519 <nxf@ip_address_of_deployment>:/tmp/nxf-backup-2.xxxxxxxxxxxx.tar.gz .
 
    Remember to replace the port, address of deployment and backup file name placeholders with your values.

Step 2

Using another command-line terminal window, upload the backup file from your local machine to another of your CWM instances to which you want to export the data: 

 scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -P <ssh_port> -i ~/.ssh/id_ed25519 ~/Downloads/nxf-backup-2.xxxxxxxxxxxx.tar.gz <nxf@ip_address_of_deployment>:/tmp

Step 3

Log in using ssh to your other CWM instance and run the upload and restore commands:

  1. Before you upload the backup and restore it, you need to delete all the preexisting backup files from the system:

    sedo backup list

  2. Copy the names of backup files one by one and delete each of them using the delete command: 

    sedo backup delete nxf-backup-2.xxxxxxxxxxxx.tar.gz

  3. Then, upload the backup file to the database (replace the placeholder name with your actual file name):

    sedo backup upload nxf-backup-2.xxxxxxxxxxxx.tar.gz

  4. Use the restore command to restore the database 

    sedo backup restore backup_name

The system will be restarted, which may take a few minutes.

Application health and logs

CWM is a microservice-based application that leverages Kubernetes cluster architecture as its runtime environment. The health of the CWM application can thus be checked using Kubernetes commands.

To see all the supported kubectl commands, log in to the OS on your VM and use kubectl --help.

Pod status

Procedure

Step 1

Using a command-line terminal, log in to the OS on your virtual machine with SSH:

ssh -o UserKnownHostsFile=/dev/null  -p 22  nxf@<virtual_IP_address>

Step 2

To check status of pods for namespace zone-a (this is the default namespace for pods contaning CWM microservices), run the following command: 

kubectl get pods -n zone-a

Step 3

A list of pods will appear:

Note

 

Step 4

If a pod has a status different from Running, you can 'restart' it using the following command: 

kubectl delete pod <pod_name> -n zone-a
The pod will be deleted, but as Kubernetes configuration is declarative, it will effectively recreate the deleted pod and rerun it.

Logs

Procedure

Step 1

CWM provides comprehensive logging for pods within the Kubernetes cluster, essential for monitoring service status, debugging the application, and viewing logs of custom workflow execution messages. Among these, the most valuable for debugging purposes are microservice logs for minIO, Postgres, and Loki. They offer insights to help identify any arising issues.

Step 2

Using the kubectl logs -n <namespace> <pod_name> command, you can retrieve logs for the most of available microservices. To see the list of pods, use: kubectl get pods -A Example result: [nxf@wf-nat-08 ~]$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-flannel kube-flannel-ds-trr95 1/1 Running 0 103m kube-system coredns-htg9j 1/1 Running 0 103m kube-system etcd-wf-nat-08 1/1 Running 0 103m kube-system kube-apiserver-wf-nat-08 1/1 Running 0 103m kube-system kube-controller-manager-wf-nat-08 1/1 Running 0 103m kube-system kube-proxy-c25f5 1/1 Running 0 103m kube-system kube-scheduler-wf-nat-08 1/1 Running 0 103m local-path-storage local-path-provisioner-6fb6f599c7-ckcjc 1/1 Running 0 103m nxf-system authenticator-5db8885675-qlrmg 2/2 Running 0 102m nxf-system controller-cbd87f8c5-6tg6f 2/2 Running 1 (102m ago) 102m nxf-system ingress-proxy-56f7c9899d-6st6j 1/1 Running 0 102m nxf-system kafka-0 1/1 Running 0 102m nxf-system loki-7c994678f8-fnrs9 3/3 Running 0 102m nxf-system minio-0 2/2 Running 0 103m nxf-system postgres-0 2/2 Running 0 102m nxf-system promtail-v6tb4 1/1 Running 0 102m nxf-system registry-7dd84db44f-n5q7h 2/2 Running 0 102m nxf-system vip-wf-nat-08-28131000-772k5 0/1 Completed 0 3m42s zone-a api-service-745759bffc-v6r25 2/2 Running 2 (100m ago) 100m zone-a dsl-service-77d5fc96cc-5nv42 2/2 Running 3 (100m ago) 100m zone-a logcli-5c7ddbc95d-mkpcc 2/2 Running 0 100m zone-a plugin-manager-665b7bbd4d-jvqdk 2/2 Running 1 (100m ago) 100m zone-a ui-service-57cf6d6bcc-smmvt 2/2 Running 0 100m zone-a worker-manager-6d6b445d46-r6nzk 2/2 Running 1 (99m ago) 100m zone-a workflow-frontend-77bc897549-kcz5k 2/2 Running 1 (99m ago) 100m zone-a workflow-history-58bdb85b8d-88t25 2/2 Running 1 (99m ago) 100m zone-a workflow-history-58bdb85b8d-h22bd 2/2 Running 1 (99m ago) 100m zone-a workflow-history-58bdb85b8d-ph5fh 2/2 Running 1 (99m ago) 100m zone-a workflow-matching-86cfc5577c-4mxhb 2/2 Running 1 (99m ago) 100m zone-a workflow-ui-68f857645-9mq9v 2/2 Running 0 100m zone-a workflow-worker-8496898f7b-wcrqs 2/2 Running 1 (99m ago) 100m

Loki logCLI

Application logs can be checked with Loki logCLI command-line interface. To gather logs from the CWM platform, follow these steps:

Procedure

Step 1

Use the command below to list all running pods:

kubectl get pods -A

Example result:

[nxf@wf-nat-08 ~]$ kubectl get pods -A
NAMESPACE            NAME                                      READY   STATUS      RESTARTS       AGE
...
nxf-system           vip-wf-nat-08-28131000-772k5              0/1     Completed   0              3m42s
zone-a               api-service-745759bffc-v6r25              2/2     Running     2 (100m ago)   100m
zone-a               dsl-service-77d5fc96cc-5nv42              2/2     Running     3 (100m ago)   100m
zone-a               logcli-5c7ddbc95d-mkpcc                   2/2     Running     0              100m
zone-a               plugin-manager-665b7bbd4d-jvqdk           2/2     Running     1 (100m ago)   100m
...

Step 2

Identify the logcli tool available in the zone-a namespace. In this example, it is the pod named logcli-5c7ddbc95d-mkpcc.

Step 3

Connect to the correct pod and list the available log labels for filtering:

kubectl exec --namespace=zone-a -ti logcli-5c7ddbc95d-mkpcc -- logcli labels
app
container
filename
level
namespace
node_name
pod
stream

Step 4

Gather logs from all applications running in the "zone-a" namespace and save them to a single file. Make sure to adjust the --since option to collect logs from the relevant time period when the troubleshooting event occurred: 

kubectl exec --namespace=zone-a -ti logcli-5c7ddbc95d-mkpcc -- logcli query '{namespace="zone-a"}' --since 60m > zone-a.log

Step 5

Use the SCP tool to copy the log files from the system to your desktop:

scp -P SSH_PORT nxf@ip_address_of_deployment:"*.log"

Step 6

You can send the logs to a support team to provide a detailed description of the issue you are experiencing.

For more details on the logCLI commands and usage, refer to logCLI Grafana documentation.

minIO logs

MinIO is an object storage system for adapter files and plugins (adapter installation files). To access minIO logs, follow these steps:

Procedure

Step 1

Locate the minIO pod. Use the following command to list all pods in the specified namespace:

kubectl get pods -n <namespace>
In the list, identify the pod running the minIO service.

Step 2

Access logs. Once you have identified the minIO pod, use the kubectl logs command to view its logs:

Step 3

kubectl logs -n <namespace> <minIO_pod_name>

Replace <namespace> with the actual namespace where minIO is deployed, and <minIO_pod_name> with the name of the minIO pod.

Step 4

 

Formatting 1st pool, 1 set(s), 4 drives per set. WARNING: Host minio-0:8080 has more than 2 drives of set. A host failure will result in data becoming unavailable.API: SYSTEM()
Time: 07:38:08 UTC 05/28/2024
Error: Write failed. Insufficient number of drives online (offline-disks=4/4) (*fmt.wrapError)
    14: internal/logger/logger.go:259:logger.LogIf()
    13: cmd/erasure-encode.go:115:cmd.(*Erasure).Encode()
    12: cmd/erasure-object.go:1462:cmd.erasureObjects.putObject()
    11: cmd/erasure-object.go:1227:cmd.erasureObjects.PutObject()
    10: cmd/erasure-sets.go:758:cmd.(*erasureSets).PutObject()
    9: cmd/config-common.go:79:cmd.saveConfigWithOpts()
    8: cmd/config-common.go:84:cmd.saveConfig()
    7: cmd/erasure-server-pool-decom.go:471:cmd.poolMeta.save()
    6: cmd/erasure-server-pool-decom.go:517:cmd.(*erasureServerPools).Init()
    5: cmd/erasure-server-pool.go:179:cmd.newErasureServerPools()
    4: cmd/server-main.go:1050:cmd.newObjectLayer()
    3: cmd/server-main.go:790:cmd.serverMain.func10()
    2: cmd/server-main.go:489:cmd.bootstrapTrace()
    1: cmd/server-main.go:788:cmd.serverMain()

API: SYSTEM()
Time: 07:38:08 UTC 05/28/2024
Error: saving pool.bin for pool index 0 failed with: Storage resources are insufficient for the write operation .minio.sys/tmp/75e1a4c5-aecd-4b48-9dc5-a30f515f6423/0ed670f0-d99c-4822-9198-f92868450e16/part.1 (*errors.errorString)
    8: internal/logger/logger.go:259:logger.LogIf()
    7: cmd/erasure-server-pool-decom.go:473:cmd.poolMeta.save()
    6: cmd/erasure-server-pool-decom.go:517:cmd.(*erasureServerPools).Init()
    5: cmd/erasure-server-pool.go:179:cmd.newErasureServerPools()
    4: cmd/server-main.go:1050:cmd.newObjectLayer()
    3: cmd/server-main.go:790:cmd.serverMain.func10()
    2: cmd/server-main.go:489:cmd.bootstrapTrace()
    1: cmd/server-main.go:788:cmd.serverMain()

API: SYSTEM()
Time: 07:38:08 UTC 05/28/2024
Error: Unable to initialize backend: Storage resources are insufficient for the write operation .minio.sys/tmp/75e1a4c5-aecd-4b48-9dc5-a30f515f6423/0ed670f0-d99c-4822-9198-f92868450e16/part.1, retrying in 4.463030339s (*fmt.wrapError)
    6: internal/logger/logger.go:259:logger.LogIf()
    5: cmd/erasure-server-pool.go:185:cmd.newErasureServerPools()
    4: cmd/server-main.go:1050:cmd.newObjectLayer()
    3: cmd/server-main.go:790:cmd.serverMain.func10()
    2: cmd/server-main.go:489:cmd.bootstrapTrace()
    1: cmd/server-main.go:788:cmd.serverMain()
Waiting for all MinIO sub-systems to be initialize...
Automatically configured API requests per node based on available memory on the system: 110
All MinIO sub-systems initialized successfully in 10.44495ms
MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2024-01-01T16-36-33Z (go1.21.5 linux/amd64)

Status:         4 Online, 0 Offline.

Postgres logs

Postgres is serves as a database system for CWM. To access Postgres logs, follow these steps:

Procedure

Step 1

Locate the Postgres pod. Use the following command to list all pods in the specified namespace:

kubectl get pods -n <namespace>
Identify the pod running the Postgres service.

Step 2

Access logs. Once you have identified the Postgres pod, use the kubectl logs command to view its logs: 

kubectl logs -n <namespace> <postgres_pod_name>
Replace <namespace> with the actual namespace where Postgres is deployed, and <postgres_pod_name> with the name of the Postgres pod.