Cisco Cloud Object Storage Release 3.22.1 User Guide

Accessing the V2PC GUI

Step 1 Open a web browser and access the V2PC GUI at https://<v2pc-ip>:8443/.

Step 2 Log in to the V2PC GUI using the following credentials:

• User name: admin
• Password: default

The V2PC GUI opens to the Dashboard page, System Statistics tab.

Figure A-2 V2PC GUI Dashboard, System Statistics Tab

Dashboard Page

The Dashboard section contains the following pages:

• System Overview – provides general system information.
• Node Statistics – graphically displays CPU, memory, disk, and network utilization for each node and server on 24-hour timelines by region, provider, and zone.
• Alarms & Events – lists alarm and event notifications for each node, showing source, status, category, severity, and relevant details.

Cisco Cloud Object Store (COS) Page

This Cisco Cloud Object Store (COS) section contains the following pages:

• COS IP Pools – Lets you add, edit, or delete IP pools, and assign or update IP address ranges,
• COS Clusters – Lets you define asset redundancy policy and configure resiliency for the cluster.
• COS Node Profiles – Lets you add or delete COS node profiles and assign profiles to a COS cluster.
• COS Nodes – Lets you add COS nodes to a COS cluster, remove already decommissioned nodes from a cluster, and change the node Admin State (Inservice or Maintenance).
• COS Service Status – Lists the nodes in the cluster; each node list item can be expanded to show the current status of the associated disks, interfaces, and services.
• COS Service Statistics – Lets you view statistics associated with all constituent COS services.
• COS Status Thresholds – Lets you view and adjust levels for Warning and Critical alarm triggering.

Command Line Reboot

To reboot a COS node, execute the reboot command from a terminal console or remote shell. The system will begin a shutdown phase to reboot.

• Any active HTTP and TCP sessions with the COS node data network interfaces will be reset by the node. The client will have the responsibility of retrying operations with the remaining COS nodes in the cluster.
• The COS management system will automatically update the DNS registry to remove listings for the COS node.
• The COS services will automatically be restarted when the system is back online after reboot.
• If the COS cluster has been configured to replica data, copies of any object data residing on the COS node will be accessible from the remaining COS nodes.

Switching Node Admin State from the GUI

To switch a COS node from Inservice to Maintenance mode or vice versa from the V2PC GUI:

Step 1 Open the GUI as described in Accessing the V2PC GUI and navigate to Cisco Cloud Object Store (COS) > COS Nodes.

Step 2 Locate the node in the COS Nodes list and click its Edit icon to open the Edit dialog for the node.

Step 3 In the Edit dialog, select Maintenance or Inservice as the new Admin State for the node.

Step 4 Click Save to save your changes and return to the COS Nodes page.

Node Decommissioning and Removal

COS lets you decommission a node at the CServer level. Decommissioning tells CServer to copy the data objects of the node to other nodes in the cluster until the target number of mirror copies is reached. After the node is decommissioned, it can be removed from the cluster using either the V2PC GUI or the API.

Node decommissioning itself is currently a CLI-only operation. To decommission a node, run the script cserver-control.pl decommission, installed on the node at /opt/cisco/cos-aic-client/cserver-control.pl.

As decommissioning can take several hours, the CLI does not monitor the decommissioning process for completion. To check for completion, enter the command cserver-control.pl decommission --stats periodically until the response confirms that the operation is complete.

After decommissioning is complete, you can safely remove the node using the GUI or the API. For instructions on removing a node from a cluster using the GUI, see Node Decommissioning and Removal. For API information, see the Cisco Cloud Object Storage Release 3.18.1 API Guide.

Note ● A node cannot be decommissioned after it has been removed from a cluster using the GUI or API. So, you must decommission a node before removing it.

• If a node is in the process of being decommissioned, decommissioning pauses if the node or any node in its cluster is placed in Maintenance mode. Decommissioning resumes when all nodes in the cluster are returned to In Service mode.
• Decommissioning will not start if you try to decommission a node when it or any node in its cluster is already in Maintenance mode. Decommissioning can only start when every node in the cluster is returned to In Service mode.

Verifying Node Removal from a Cluster

When you remove a node from a multi-node cluster through the GUI, the node is first decommissioned from the Cassandra database cluster, and then the Cassandra service and CServer are shut down. If you shut down the node before the Cassandra-level decommissioning completes, the node may continue to be considered part of the Cassandra cluster and listed in the nodetool status output of the remaining nodes, but now in down (DN) state, which prevents you from adding new nodes to the cluster.

To avoid this issue, we recommend opening the COS AIC Client log before removing the node through the GUI. Inspect the log periodically to confirm that Cassandra decommissioning is completed before shutting down the node.

To inspect the log for node decommissioning from the Cassandra cluster:

Step 1 Use the Linux tail command to print new lines being added to the COS AIC Client log, followed by the Linux grep command to search for db-remove:

Step 2 Remove the node using the GUI and inspect the log for db-remove:

Step 3 Inspect the log for Completed db-remove, which shows that the node has been removed from Cassandra cluster:

Step 4 To verify that CServer has also been shut down, inspect the log using tail (or cat) followed by grep for cserverControl-shutdown:

Step 5 To confirm completion of the removal process, inspect the log to ensure that no new messages are printed:

[root@Colusa-4T-72 ~]# tail -f /arroyo/log/cos-aic-client.log.20160506

Step 6 Run the command nodetool status cos on one of the remaining nodes in the cluster to confirm that the removed node is no longer listed as part of the cluster.

Reinstalling a COS Node in a Cluster

It may become necessary to reinstall a COS node in a cluster certain situations, such as:

• A change in the Linux partition size in a new COS version to allow for a larger database or log size.
• Linux file system corruption caused by a reboot (observed at least once on a CDE 465).

If necessary, reinstall a COS node as follows:

Step 1 Log into the V2PC GUI as described in Accessing the V2PC GUI.

Step 2 From the V2PC GUI navigation menu, choose Cisco Cloud Object Store (COS) > COS Nodes.

Step 3 Locate the node to be removed and place the node in Maintenance mode as described in Switching Node Admin State from the GUI.

Step 4 Decommission the node and remove it from the cluster as described in Node Decommissioning and Removal.

Step 5 Is Linux still running on the node just removed?

• If yes, perform steps 1-6 of Verifying Node Removal from a Cluster to verify removal.
• If no, perform only step 6 of Verifying Node Removal from a Cluster to verify removal.

Step 6 Perform a fresh installation of the node using the full COS ISO image as described in the appropriate section of Deploying COS.

Step 7 On the Cisco Cloud Object Store (COS) > COS Nodes page of the V2PC GUI, locate the node and add it to the desired cluster.

Note ● If you must reinstall a COS node immediately after removing it from a cluster, first verify that the node removal has completed in the associated Cassandra database cluster. To check the status of the Cassandra database cluster, execute nodetool status cos on the console of a remaining COS node. When removal has completed, the list will no longer contain the management IP address of the removed node.

• When adding more than one node to a COS cluster, wait at least 5 minutes between adding one COS node before adding the next. This ensures that the first addition to the associated Cassandra database cluster has completed. To check the status of the Cassandra database cluster, executing nodetool status cos on the console of a previously added COS node. When the node addition has completed, the list will include the management IP address of the new node, and its status will be UN, indicating that the node is up and operating normally.

Behavior of COS Services on COS Node Boot

There are four primary system services that provide COS functionality on a COS node: cassandra, cosd, cos_aicc, and cserver. These services can be manipulated using standard Linux system service tools.

To prevent these services from starting automatically on a COS node boot, execute the following commands as the root user from a shell on that node:

To enable automatic service loading on node boot, execute the following commands:

To view the current state of service loading, execute the following command:

To manually start the services, execute the following commands:

COS Node Disks

When the COS AIC server receives a status from the COS AIC client indicating the failure of one or more disks on a COS node, it will process that status as follows:

• If more than 50% or the disks are down, the node interfaces are removed from the DNS server. The Disk Status for that node is then set to Critical and an appropriate alarm is raised.
• If less than 50% of the disks are down, the Disk Status for that node is set to Warning and an appropriate alarm is raised.

COS Services

If the COS AIC server receives a status from the COS AIC client indicating that one or more of the following COS services is down, the Service Status for the node is set to Critical, the node is marked as down, an appropriate alarm is raised, and its service interfaces are removed from the DNS.

• Cisco Cache Server (CServer)
• Cisco Cloud Object Storage Daemon (cosd)
• Cassandra Server

If the COS AIC server receives a status from the COS AIC client indicating that one or more of the following COS services is down, the Service Status for the node is set to Warning and an appropriate alarm is raised.

• NTP Daemon
• Cisco SNMP Agent
• Consul Agent
• Monit
• Sensu Client

COS Node Interfaces

When the COS AIC server receives a status from the COS AIC client indicating that one or more service interfaces are not functional, those interfaces are removed from the DNS.

Additionally, if more than 50% of the nodes interfaces are reported as down, the node is considered non-operational and all interfaces for it are removed from DNS. An appropriate alarm is also raised, and the Interface Status is set to either Critical (for 50% or more down) or Warning (for less than 50% interfaces down).

Server Reachability

The COS AIC server expects to receive a heartbeat message from the COS AIC client running on each node every 2 seconds. Abnormalities are processed as follows:

• If the heartbeat message ceases to arrive, the node is considered as down, an alarm is raised, and all of its interfaces are removed from the DNS.
• If heartbeats still arrive but are slower than expected, the delay is noted in the COS AIC server log and the node is kept in service.

Rotating Logs

Rotate, moves and compresses log files located within the /arroyo/log to /arroyo/log/archive, based on the configuration parameters.

Trim

Trim deletes (trims).tgz files located within the /arroyo/log/archive, based on the configuration parameters. These files are deleted for the following reasons:

• .tgz files located within the /arroyo/log/archive subdirectory are deleted when they are older than the designated rotate$trim_max age.
• They are also deleted based on the percent of disk space used in /arroyo/log/archive. This percentage is configured as the rotate$trim_max_size or rotate$trim_hourly_max_size. Where, the rotate$trim_hourly_max_size parameter is used if the trim occurs at rotate$trim_hour. Otherwise, rotate$trim_max_size is used. Trim maximum size is the maximum percent of /arroyo/log/archive which can be used for the.tgz files.

Examples:

1. The /arroyo/log partition is 20G. Because this partition is less than 30G and rotate$trim_max_size is 50%, the archive files are deleted when the used % is greater than 15G (.50 * 30G).

2. The /arroyo/log partition is 98G, the.tgz files are deleted until the % used is <= 86.24G (.88 * 98G).

The following are important configuration settings, which can be changed to decrease the size of the accumulated log files.

• rotate$log_age_days = 1, where:

– Unit: day

– Default value: 1

– Number of days that the log files stay in the /arroyo/log subdirectory before being moved to /arroyo/log/archive.

• rotate$trim_max_age = 365, where:

– Unit: day

– Default value: 365

– Number of days the *.tgz files stay in the /arroyo/log/archive subdirectory. This time is also affected by the rotate$trim_max_size and the rotate$trim_hourly_max_size configuration settings.

• rotate$trim_max_size = 50, where:

– Unit: percent

– Default value: The default value is 50 if the /arroyo/log size is < 30GB, but 88 if it is larger. Archived log files are deleted if this value is exceeded.

• rotate$trim_hourly_max_size = 75, where:

– Unit: percent

– Default value: The default value is 75 if the /arroyo/log size is < 30GB, but 90 if it is larger. Archived log files are deleted if this value is exceeded.