Cisco Cloud Object Storage Release 3.16.1 User Guide

About Mirroring

Mirroring configures the disks in a node, or the nodes in a cluster, to hold a specified number of exact copies of the object data. The number of copies can be set from 1 to 4, with 1 representing the original object data only, and 4 representing the object data plus three copies. Thus, for example, a value of 2 specifies one copy plus the original object data.

To configure mirroring in the GUI, you specify the number of desired copies differently for local and remote mirroring:

• For local mirroring, the number you specify includes the original plus the number of local copies. For example, setting the number to 3 in the GUI specifies the original and two local copies.
• For remote mirroring, the number you specify includes only the number of remote copies. For example, setting the number to 3 in the GUI specifies three remote copies in addition to the (local) original.

Note For clusters of two or more nodes, always use DEC instead of remote mirroring.

About Erasure Coding

Erasure coding, or software RAID, is a method of data protection in which cluster data is redundantly encoded, divided into blocks, and distributed or striped across different locations or storage media. The goal is to enable any data lost due to a drive or node failure in the cluster to be reconstructed using data stored in a part of the cluster that was not affected by the failure.

COS 3.16.1 supports both local erasure coding (LEC), in which data is striped across the disks in a node, and distributed erasure coding (DEC), in which data is striped across nodes in a cluster.

For both LEC and DEC, data recovery is performed as a low-priority background task to avoid possible impact on performance. Even so, erasure coding provides faster data reconstruction than hardware RAID. Speed is important in maintaining resiliency, because a failed node cannot help to recover other failed nodes until it has fully recovered.

Defining Resiliency

Two key factors define the degree of resiliency of a given cluster configuration:

• Resiliency ratio (RR) functionally divides the disks or nodes in a cluster into data blocks and parity blocks. The RR of a cluster indicates how much of its total storage is devoted to protection against data loss. Conventionally, the RR is expressed as an N:M value, where N represents the number of data blocks and M is the number of parity blocks.
• Resiliency factor (RF) is the number of data (N) blocks that can fail at one time before losing the ability to fully recover any lost data. The RF is equivalent to the parity (M) value in the RR for the cluster, and can be selected in the GUI as described in About Mirroring.

For LEC, COS assigns a default RR of 12:2, but you can choose another RR up to 18:4 maximum. For DEC, the cos-aic-client calculates the RR for a chosen cluster size and RF, but you can choose another RR up to 18:18 maximum. As you consider whether to use the defaults or assign new values, you must weigh a number of factors specific to your deployment, such as:

• The amount of storage that can be devoted to resiliency
• The degree of redundancy (RF) that must be achieved
• How quickly failed data blocks can be recovered

The following examples illustrate how these factors can be used to determine the best resiliency scheme for a particular COS node or cluster.

Example 1: LEC at Node Level

To see how the LEC resiliency ratio affects individual node behavior, consider a cluster in which each node has 4 TB disks, runs at 80% full, and can maintain a data read rate of 20 Gbps (2.5 GB/s).

If each node in the cluster has an assigned LEC RR of 10:1:

• Each node can recover from a single disk failure on its own, but if more than one concurrent disk failure occurs, DEC must be used to rebuild the data.
• 1 MB of parity data is created for every 10 MB of data written to each node, representing a storage overhead of 1/10 or 10%. Also, in the event of a lost disk, the rebuild logic must read in 10 MB of data for every 1 MB recovered, representing a rebuild overhead of 10:1.
• If a node loses a single disk when running at 80% full, then (4 TB x 0.8 =) 3.2 TB of data must be rebuilt. At a 20 Gbps data read rate and a 10:1 rebuild overhead, the data rebuild rate is (20 Gbps / 10 =) 2 Gb (256 MB) per second, or roughly 0.9 TB per hour. This means that 3.2 TB of data can be rebuilt in under 4 hours.

By comparison, if the same cluster were assigned the default LEC RR of 12:2 instead of 10:1:

• Each node can recover from two concurrent disk failures on its own, before using DEC.
• Each node stores 2 MB of parity for every 12 MB of data, representing a 17% a storage overhead and a 12:1 rebuild overhead.
• At a data rebuild rate of (20 Gbps / 12 =) 0.75 TB per hour, recovery takes about 4.3 hours for one lost disk (3.2 TB), or 8.6 hours for two lost disks.

Note • Actual data rebuild times are highly dependent on the capacity of the hardware components in the server. Servers that can deliver significantly more throughput reduce the rebuild time, while servers with less capacity increase the rebuild time. LEC uses local disk reads when recovering data, so the speed of the disk channel directly impacts recovery time.

• Rebuild activity runs at a lower priority than regular object reads and writes, in an effort to maintain normal performance during rebuild periods. However, because normal reads and writes take priority over repair activity, a busy server takes longer to rebuild data from a lost disk than an idle server, which can devote more bandwidth to repair.
• An exception to the lower priority for rebuild activity is made for on-demand repair work. This occurs when a client requests data that is waiting to be rebuilt due to a disk failure. In this case, the requested data is rebuilt immediately so that the repair work is transparent to the clients.

Example 2: DEC at Cluster Level

To see how the DEC resiliency ratio affects cluster configuration and behavior, consider a cluster of 11 nodes, with each node running 80% full, maintaining a data read rate of 16 Gbps (2 GB/s), and having a total storage capacity of 100 TB.

If the cluster has an assigned DEC RR of 8:2:

• The cluster requires a minimum of 10 active nodes (8 data plus 2 parity) to continue writing data at the assigned 8:2 resiliency ratio. With 11 nodes, this cluster can lose one node and still continue to provide the same resiliency for newly added objects.
• The cluster can recover from two concurrent node failures, but to do so, must have enough free disk space to hold the recovered data for two nodes. For an 11-node cluster, this means that 2/11, or just under 20%, of the total disk space must be kept free. This can be reduced to 10% if it is safe to assume that at least one replacement node can be brought online.
• 2 MB of DEC parity data is created for every 8 MB of data written to the cluster, representing a storage overhead of 2/8 or 25%. This compares favorably to mirroring, whose storage overhead is four times greater (100%) but only enables recovery from one node failure, rather than two. Also, in the event of a lost node, the rebuild logic must read in 8 MB of data for every 1 MB recovered, representing a rebuild overhead of 8:1.
• If one node is lost when running at 80% full, then (100 x 0.8 =) 80 TB of data must be rebuilt by the 10 remaining nodes in the cluster. Dividing the task equally among the remaining 10 servers, each must rebuild 8 TB of data. At a 16 Gbps data read rate and an 8:1 rebuild overhead, the data rebuild rate is (16 Gbps / 8 =) 2 Gb (256 MB) per second, or 0.9 TB per hour. At this rate, each node can rebuild its 8 TB share of the total lost data in 8.9 hours.

By comparison, if the same cluster were assigned a DEC RR of 8:1 instead of 8:2:

• The cluster can fully recover data from only a single failed node, but requires only half the free disk space (10%) of the 8:2 RR case to support this recovery.
• Each node stores 1 MB of parity for every 8 MB of data, representing a 12.5% a storage overhead and an 8:1 rebuild overhead.
• Because the rebuild rate is determined by the N value (8), it is the same for 8:1 as for 8:2. At a data rebuild rate of (16 Gbps / 8 =) 0.9 TB per hour, recovery of one lost node (8 TB) takes 8.9 hours.

Note • Actual data rebuild times are directly affected by the number of nodes in the cluster. If there are fewer nodes in the cluster, the work is divided among fewer servers, resulting in longer rebuild times. DEC uses remote data reads across the network when recovering data, so the speed of the network channel directly impacts the recovery time.

• Actual data rebuild times are highly dependent on the capacity of the hardware components in the server. Servers that can deliver significantly more throughput reduce the rebuild time, while servers with less capacity increase the rebuild time.
• Rebuild activity runs at a lower priority than regular object reads and writes, in an effort to maintain normal performance during rebuild periods. However, because normal reads and writes take priority over repair activity, a busy cluster takes longer to rebuild data from a lost node than an idle cluster, which can devote more bandwidth to repair.
• An exception to the lower priority for rebuild activity is made for on-demand repair work. This occurs when a client requests data that is waiting to be rebuilt due to a disk failure. In this case, the requested data is rebuilt immediately so that the repair work is transparent to the clients.

Example 3: Using LEC and DEC Together

To see how LEC and DEC work together, consider the cluster configuration described in the previous examples, but with a LEC RR of 10:1 for each node and a DEC RR of 8:2 for the cluster as a whole.

With this configuration:

• The cluster can recover from two concurrent node failures as well as a single disk failure per node at the same time. A LEC RR of 10:1 means that a node must lose at least two disks concurrently before DEC is required to recover the lost data.
• Because the DEC parity data is distributed across the nodes in the cluster, it is further protected by LEC at the node level. This allows recovery from disk failure to be performed locally, including the DEC parity data stored on the node.
• The DEC RR of 8:2 means that every 10 MB block of data includes 8 MB for content storage and 2 MB for DEC parity. In addition, the LEC RR of 10:1 means that every 10 MB block of (content + DEC parity) data requires another 1 MB of parity data. So in total, every 8 MB of content storage requires (2 DEC + 1 LEC =) 3 MB of parity data, giving a total parity overhead of (3 / 8 =) 37.5%.

For any cluster configuration using both DEC and LEC, the total parity overhead is found by first applying the DEC overhead and then applying the LEC overhead, as follows:

Total Parity Overhead = (M1/N1) + ((1 + M1/N1) X (M2/N2))

where N1:M1 represents DEC parity and N2:M2 represents LEC parity.

Using the values from this example to illustrate:

Total Parity Overhead = (2/8) + ((1 + 2/8) X (1/10) = 37.5%

Note The parity calculations just described do not account for the additional free disk space needed for storage of recovered data. Be sure to include this requirement in your total overhead calculations. For example, a DEC RR of 8:2 requires free disk space for up to two failed nodes.

Configuring Resiliency Using the V2PC GUI

To configure resiliency through the V2PC GUI, open the GUI as described in Accessing the V2PC GUI and navigate to Cisco Cloud Object Store (COS) > COS Clusters.

Figure D-1 V2PC GUI, COS Clusters Page

Locate the cluster to be updated and click its Edit icon to enable it for editing. Choose the desired resiliency policy from the Asset Redundancy Policy drop-down list for the endpoint.

Note • If the desired policy does not appear, choose Service Domain Objects > Asset Redundancy Policies to confirm the policy exists. If not, click Add Row and create a new policy. Enter a profile name and expected cluster size, select a model (appropriate interfaces and names auto-populate), assign the profile to a cluster, configure a DNS server, and assign IP pools to each interface.

• The GUI lets you enter an expected cluster size when creating the COS node initialization profile. For DEC, the cos-aic-client then uses this cluster size to calculate a suitable N value given the expected cluster size and the selected resiliency factor M. Both values can still be overridden as described in Finding N:M Values by editing or creating the /arroyo/test/aftersetupfile COS file to hold them.

Configuring Local Mirroring Manually

To configure local mirroring on a node manually:

Step 1 Open (or if not present, create) the COS file /arroyo/test/aftersetupfile for editing.

Step 2 Include the line vault local copy count in the file and set the value to 2, 3, or 4 as appropriate.

Note Setting the value to 1 simply maintains the original data and creates no additional copies.

Step 3 Disable local erasure coding by setting allow vault raid to 0 (or simply omit or remove this line).

Example

Configuring Local Erasure Coding Manually

To enable local erasure coding manually:

Step 1 Open (or if not present, create) the COS file /arroyo/test/aftersetupfile for editing.

Step 2 Set allow vault raid to 1 to enable LEC.

Step 3 Disable local mirroring by setting vault local copy count to 1 (or simply omit or remove this line).

Example

Migrating from LM to LEC Manually

To migrate a service endpoint from local mirroring to local erasure coding:

Step 1 Temporarily leave local mirroring enabled for the service endpoint.

Step 2 Enable LEC for the service endpoint and let it establish the parity needed for each data object.

Step 3 When parity is established, disable local mirroring.

Configuring Remote Mirroring Manually

Note For clusters of two or more nodes, always use DEC instead of remote mirroring.

To enable and configure remote mirroring manually:

Step 1 Open (or if not present, create) the COS file /arroyo/test/aftersetupfile for editing.

Step 2 Set vault mirror copies to the value 2, 3, or 4 as appropriate to enable remote mirroring. The value you enter specifies the object data plus the number of exact copies desired.

Note Setting the value to 1 simply maintains the original data and creates no additional copies.

Step 3 Disable distributed erasure coding by setting allow server raid to 0 (or simply omit or remove this line).

Example

Configuring Distributed Erasure Coding Manually

To enable and configure distributed erasure coding manually:

Step 1 Open (or if not present, create) the COS file /arroyo/test/aftersetupfile for editing.

Step 2 Set allow server raid to 1 and add the following lines immediately below:

• target server raid data blocks <value>

This controls the number of data blocks used. The default <value> is 8, and the valid range is 1-18.

• target server raid parity blocks <value>

This controls the number of parity blocks used. The default <value> is 1, and the valid range is 1-18.

Note See Finding N:M Values to determine appropriate data block and parity block values.

Step 3 Disable remote mirroring by setting vault mirror copies to 0 (or simply omit or remove this line).

Example

Finding N:M Values

To configure DEC, you must specify the number of data blocks (N) and parity blocks (M) used for data encoding. Table D-1 shows the corresponding data-to-parity block (N:M) values for a given number of nodes in a cluster and for a given degree of resiliency desired for the cluster. For details, see Defining Resiliency.

Note COS does not currently support configuration of new N:M (data:parity) block values through the V2PC GUI. If you need to configure new N:M values, you must do so in the aftersetup file.

In this table:

• Nodes is the number of nodes in the cluster.
• RF is the desired resiliency factor, or number of nodes that can fail without data loss.
• Min is the minimum number of nodes required to achieve a given resiliency factor.

The ratios appearing in the cells of the table are N:M values, where N is the number of data blocks and M is the number of parity blocks needed to achieve the desired resiliency factor for a given node count.

To use the table to find the N:M values for a cluster:

Step 1 In the Nodes column, locate the row corresponding to the number of nodes in the cluster.

Note For COS 3.16.1, you must select the N:M configuration based upon the initial nodes in the cluster. COS does not currently support adding nodes to a cluster after DEC is configured for the cluster.

Step 2 Locate the column in the table whose header represents the desired RF value for the cluster.

Step 3 Find the corresponding N:M value at the intersection of the row and column just located.

Step 4 Configure DEC using N as the number of data blocks and M as the number of parity blocks.

Table D-1 lists the possible N:M values for DEC for 1-20 nodes and a resiliency factor (RF) of 0-4.

Table D-2 shows the total parity overhead (as described in Defining Resiliency) for a given number of nodes and resiliency factor for each of two LEC values, 12:1 and 12:2.

Replicating Objects During Swift Write Operations

While an object is being created or modified using Swift write operations, copies of the object data can be stored in real time on the local COS node and its peer nodes in the COS cluster. This functionality works only if the RAID feature on the node is disabled.

To disable the RAID feature on the node, open /arroyo/test/setupfile and set allow vault raid to 0.

To replicate object data on the node, open /arroyo/test/setupfile and set vault local copy count to a value greater than 1. This value specifies the how many copies of the object data are to be stored on the node.

To replicate object data on the peer nodes, open /arroyo/test/setupfile and set vault mirror copies to a value greater than 1. This value specifies how many remote copies of the object data are maintained.

Related Terminology

To briefly review the components of a COS installation:

• A node is a single COS server. Every node has a server ID that is unique within its cluster.
• A site consists of one or more COS servers that have the same array ID. A site is a user-defined subdivision of a cluster, and may (but need not) correspond to a geographic location.
• A cluster contains one or more COS servers with the same namespace and group ID. A cluster must have at least one COS server and exactly one associated Cassandra database cluster, and may also contain multiple sites. A cluster has a target resiliency that can be met using DEC or mirroring.

Design Considerations

• Within a specific resiliency group, all server hardware should be of the same type.
• COS Release 3.16.1 does not support downgrade if more than one resiliency group is used.
• A resiliency group can contain a maximum of 20 COS nodes.
• When expanding, recommended practice is to fill one resiliency group before adding nodes to another resiliency group.
• The number of servers per resiliency group must support the DEC pattern. For example, if the DEC pattern is 2:1, there must be at least three servers in the resiliency group (and four would be ideal with DEC 2:1).

Defining Resiliency Groups

Defining a new resiliency group simply requires assigning an available resiliency group ID to each of the COS nodes to be included in the group. COS supports defining resiliency groups either via the V2PC GUI or via the setup file (use setupfile with MOS or VMP, and aftersetupfile with V2PC).

Note While it is possible to fully configure COS from the CLI, deployments typically use V2PC COS node profiles, in which case V2PC populates the data for the setup file through the COS node profile.

Using the V2PC GUI

To define resiliency groups using the V2PC GUI:

Step 1 Log in to the V2PC GUI and, from the navigation menu, choose Cisco Cloud Object Storage (COS) > Node Profiles.

Step 2 Locate a COS node to be included in the resiliency group and open its node profile for editing.

Step 3 Assign an available Resiliency Group ID to the node (valid options are 1-200) and save the changes.

Step 4 Repeat Steps 2-3 for all nodes to be assigned to the same resiliency group ID.

Using the Setup File

To define resiliency groups using the setup file:

Step 1 Open (or if not present, create) one of the following files:

• /arroyo/test/aftersetupfile if using V2PC
• /arroyo/test/setupfile if using MOS or VMP

Step 2 Add the following line to the file:

resilience group ID <value>

where <value> is a number between 1 and 200.

Example

Note The line allow server raid 0 must be set or cserver will not load.

Step 3 Save the changes to the file.

Resiliency Group Assignment

At boot-up, the local COS node goes through all the servers in the RemoteServers file and finds their Resilience Group ID. This request also tells the remote server the local server Resilience Group ID.

Read Requests

When a read request is received by a server, it will look up the object in Cassandra and locate the corresponding Resilience Group ID(s). If the local Resilience Group is in the list of IDs, the read can continue the same as it does in the current version of COS.

If the first Resilience Group in the list doesn’t actually have the desired data available, check the other resilience Groups one at a time until the desired data is located.

For the case where a different Resilience Group contains the desired object, check if the client supports being redirected to the host Resilience Group, and do so if it is supported.

For the case where read redirection isn’t supported:

1. Proxy read the information for the client.

2. Send a locate request to the servers in the Resilience Group we are connected to

3. Get back the location of the data stripes and their host server IP addresses.

4. Issue HTTP read requests to those servers to read the corresponding stripe data.

5. Once the stripe data is received, forward it to the client.

Write Requests

When a write request is received by a server, it will check if the current Resilience Group’s capacity is above average or below average. If the group is above average for capacity (more than 10%), it will choose another Resilience Group better suited to handle the write and forward the write request to a server in that Resilience Group.

If the client supports redirect, then the client write request can be redirected to the other Resilience Group.

If redirect isn’t supported, then that node will act as a proxy and forward the write request to the other Resilience Group.

Once a Resilience Group accepts a write request, it proceeds the same as in COS 3.12. When the goid value is written to Cassandra, the Resilience Group ID value will also be written so that the goid can be found in the future.

Troubleshooting

A new proc file /proc/calypso/status/resiliencegroupjsoninfo is available on every server in the cluster. It shows the server's view of the resiliency groups defined for the cluster. These groups are displayed in JSON format for ease of parsing.

Each server’s json info file should be similar – containing the same resilience groups and associated servers – but the order of the entries may be different on different servers.

Example JSON File

{

"Resilience Groups":

[

{

"Resilience Group ID": "2",

"Status": "Normal",

"Partner Resilience Group ID": "none",

"Member Count": 3,

"Members":

[

{

"Server ID": 14,

"State": "Normal",

"Affinity Hint": 0,

"Anti-affinity Hint": 2

},

{

"Server ID": 13,

"State": "Normal",

"Affinity Hint": 0,

"Anti-affinity Hint": 1

. . . .