Contents

Introduction

Identify

Cluster Types

Performance Chart Explanation

IOPS

Throughput

Latency

Frequency

External Factors

Identify Performance Issues at the Guest VM Level

Windows

ESXi

PVSCSI Check

Network Adapter Check

RSS Check

Introduction

This document describes the performance impact in a hyperflex environment, from the perspective of a Guest Virtual Machine (VM), ESXi host, and (SCVM)

Identify

In order to troubleshoot the performance in a Hyperflex environment it is important to identify the type of cluster, the operation where the performance is degraded, the frequency of the performance degradation, and the level of performance impact that causes performance degradation.

There are multiple levels of impact in a hyperflex cluster, at the guest VM, the ESXI host level, and the Storage controller VM level.

Cluster Types

●      Hybrid nodes: Uses Solid State Drives (SSD) drives for caching and HDDs for the capacity layer.

●      All-flash nodes: Uses SSD drives or Non-Volatile Memory Express (NVMe) storage for caching, and SSD drives for the capacity layer.

●      All-NVMe nodes: Uses NVMe storage for both caching and the capacity layer all-NVMe nodes deliver the highest performance for the most demanding workloads with caching

Performance Chart Explanation

The hyperflex systems have a feature to monitor performance, the charts display the read and write performance of the storage cluster.

IOPS

Input/output operations per second (IOPS)  is a common performance metric used to measure computer storage devices, including HDDs. This metric is used to evaluate performance for random I/O workloads.

IOPS performance chart.

Throughput

The image shows the rate of data transfer in the storage cluster measured in Mbps.

Throughput performance chart.

Latency

Latency is a measure of how long it takes for a single I/O request to complete. It is the duration between issuing a request and receiving a response, is measured in milliseconds.

Latency performance chart.

Frequency

It is important to define the frequency and duration of the performance impact to review the possible impact on the environment.

If the performance is impacted all the time, is necessary to check where it started to degrade the performance and check for any configuration changes or issues between the cluster.

If the performance is impacting intermittently, is necessary to check if there is an operation or service running at that time.

External Factors

The performance of the cluster can be affected by external factors such as snapshots and backup operations.

Review these links for further information on external factors:

VMware vSphere Snapshots: Performance and Best Practices.

Cisco HyperFlex Systems and Veeam Backup and Replication White Paper.

Identify Performance Issues at the Guest VM Level

This is the most visible level of impact in the hyperflex environment, it affects directly the services that the VMs are providing and it is more evident with the users that are directly affected.

Here are common tests to identify performance on common Operating systems.

Windows

Review the available tools to identify performance issues in Windows Guest VMs:

Performance Monitor

Resource Monitor

ESXi

After identifying the performance impact and reviewing the possible causes of the performance degradation, there are some performance checks to improve the performance.

- Check for overprovision (the total number of vCPUs that are assigned to all the VMs must not exceed the total number of physical cores available on the ESXi host machine).

- Even if the guest OS does not use some of its vCPUs, configuring VMs with those vCPUs still imposes some small resource requirements on ESXi that translate to real CPU consumption on the host.

- Over-allocating memory also unnecessarily increases the VM memory overhead and can lead to memory contention, especially if reservations are used.

- Verify that the Balloon driver is not retaining a hold on the memory, for more information, refer to this link.

Review to Troubleshooting ESX/ESXi virtual machine performance issues.

PVSCSI Check

Paravirtual SCSI (PVSCSI) adapters are high-performance storage adapters that can result in greater throughput and lower CPU utilization for virtual machines with high disk IO requirements, it is recommended to use PVSCSI adapters. PVSCSI controller is a virtualization-aware, high-performance SCSI adapter that allows the lowest possible latency and highest throughput with the lowest CPU overhead.

PVSCSI adapter.

Network Adapter Check

VMXNET 3 is a paravirtualized NIC designed for performance and provides high-performance features commonly used on modern networks, such as jumbo frames, multi-queue support (also known as Receive Side Scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery and hardware offloads.

Ensure the adapter type is VMXNET3.

Network adapter.

RSS Check

Note: This check only applies to the guest virtual machines that are running a Windows operating system.

Receive side scaling (RSS) is a network driver technology that enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems.

Windows servers have a driver configuration that enables the distribution of the kernel-mode network processing load across multiple CPUs.

Verify if it is enabled run this command on the Windows PowerShell:

netsh interface tcp set global rss=enabled

In order to enable RSS review this link

CPU Hot-Plug Check

CPU hotplug is a feature that enables the VM administrator to add CPUs to the VM without having to power it off. This allows adding CPU resources on the fly with no disruption to service. When CPU hotplug is enabled on a VM, the vNUMA capability is disabled.

CPU hot plug disabled.

Review the best practices for common Operating systems and applications:

Windows.

Performance Tuning Guidelines for Windows Server 2022.

Red Hat.

3 tips for Linux process performance improvement with priority and affinity.

SQL Server.

Architecting Microsoft SQL Server on VMware.

RedHat.

Performance Tuning Guide.

Identify Performance Issues at the Host Level

To identify the performance impact at the host level you can review the performance charts that the ESXI host has built-in in the ESXI hypervisor, and check how many hosts are impacted.

You can view the performance charts in vCenter in the monitor tab, click the performance tab.

vCenter performance charts.

In these charts, you can view the performance charts related to CPU, memory, and Disk. Refer to this link to understand the charts.

Note: CRC errors and MTU mismatch especially in the storage network generate latency issues. Storage Traffic must use Jumbo Frames.

Storage I/O Control and Queue Depth Check

Storage I/O Control (SIOC) is used to control the I/O usage of a virtual machine and to gradually enforce the predefined I/O share levels, is necessary to have this feature disabled in Hyperflex Clusters.

Queue depth is the number of pending input/output (I/O) requests that a storage resource can handle at any one time.

You can use these steps to verify SIOC is disabled and that the Queue Depth configuration.

Confirm that SIOC is running on ESXi and the Queue Depth Configuration

Step 1. SSH to an HX ESXi host and issue the command to list the datastores.

[root@] vsish -e ls /vmkModules/nfsclient/mnt

encrypted_app/

Prod/                                        <----- Datastore name

Dev/

App/

Step 2. Use the Data Store name and issue the command.

vsish -e get /vmkModules/nfsclient/mnt/<datastore name>/properties

[root@] vsish -e get /vmkModules/nfsclient/mnt/Prod/properties

mount point information {

volume name:Prod

server name:7938514614702552636-8713662604223381594

server IP:127.0.0.1

server volume:172.16.3.2:Prod

UUID:63dee313-dfecdf62

client src port:641

busy:0

socketSendSize:1048576

socketReceiveSize:1048576

maxReadTransferSize:65536

maxWriteTransferSize:65536

reads:0

readsFailed:0

writes:285

writesFailed:0

readBytes:0

writeBytes:10705

readTime:0

writeTime:4778777

readSplitsIssued:0

writeSplitsIssued:285

readIssueTime:0

writeIssueTime:4766494

cancels:0

totalReqsQueued:0

metadataReqsQueued(non IO):0

reqsInFlight:0

readOnly:0

hidden:0

isPE:0

isMounted:1

isAccessible:1

unstableWrites:0

unstableNoCommit:0

maxQDepth:1024       <-------- Max Qdepth configuration

iormState:0          <-------- I/O control disabled

latencyThreshold:30

shares:52000

podID:0

iormInfo:0

NFS operational state: 0 -> Up

enableDnlc:1

closeToOpenCache:0

highToAvgLatRatio:10

latMovingAvgSmoothingLevel:2

activeWorlds:55

inPreUnmount:0

}

Step 3. In the output look for the line

iormState:0 0= disabled 2= enabled

The line maxQDepth must be 1024

Step 4. The same steps must be repeated for the rest of the Datastores

Disable SIOC

In order to disable the SIOC run these steps.

Step 1. Login to vsphere using the HTML client.

Step 2.  From the drop-down menu, select Storage and then select the applicable HX Datastore in the left pane.

Select datastore.

Step 3. In the right pane top section of the Datastore, select the configure tab.

Configure tab.

Step 4. In the right pane middle section Under More, select General, and on the right side scroll down to DataStore Capabilities and click Edit

Edit datastore capabilities.

If the Disable Storage I/O Control and Statistics collection radio button is unchecked, check it.

Disable storage I/O control.

If the Disable Storage I/O Control and statistics collection radio button are checked, toggle between Enable Storage I/O Control and statistics collection and Disable Storage I/O Control and statistics collection.

Storage I/O control disabled.

Step 5. Repeat the Step 1 to 4 as necessary for all other datastores.

Modify MaxQDepth

In order to modify the maxQDepth issue the next command for each datastore.

vsish -e set /vmkModules/nfsclient/mnt/<yourdatastorename>/properties maxQDepth 1024

Check for Rx_no_Buff

Hyperflex servers with heavy network traffic or network traffic with microbursts can lead to packet loss seen in the form of rx_no_bufs.

To identify this Issue run these commands in the ESXi host to check the rx_no_buf  counters.

/usr/lib/vmware/vm-support/bin/nicinfo.sh | egrep "^NIC:|rx_no_buf"

NIC: vmnic0

rx_no_bufs: 1

NIC: vmnic1

rx_no_bufs: 2

NIC: vmnic2

rx_no_bufs: 2

NIC: vmnic3

rx_no_bufs: 71128211 <---------Very high rx_no_bufs counter

NIC: vmnic4

rx_no_bufs: 1730

NIC: vmnic5

rx_no_bufs: 897

NIC: vmnic6

rx_no_bufs: 24952

NIC: vmnic7

rx_no_bufs: 2

Wait a few minutes and run the command again and check if the rx_no_bufs counters are not increasing.

- If these counters are low (< 1,000) then with little packet loss due to the default queue configuration and likely no tuning needs to be done.

- If these counters are high (> 10,000) then there is some impact due to this queue configuration and tuning could help a little.

- If these counters are very high (> 1,000,000) then there is a more significant impact, increasing queues is highly recommended.

- If rx_no_bufs are actively incrementing, this means the packet made it all the way across the network and arrives at the virtualized layer then the packet is dropped.

If you see the counter on these values please contact Cisco TAC to tune the vNIC configuration for better performance.

Review the best practices and additional checks at ESXI Level.

Performance Best Practices for VMware vSphere 7.0.

Identify Performance Issues Storage Controller Virtual Machine (SCVM) Level

Cluster Health

Verify if the cluster is healthy.

hxshell:~$ sysmtool --ns cluster --cmd healthdetail

Cluster Health Detail:

---------------------:

State: ONLINE                       <---------- State of the cluster

HealthState: HEALTHY                <---------- Health of the cluster

Policy Compliance: COMPLIANT

Creation Time: Tue May 30 04:48:45 2023

Uptime: 7 weeks, 19 hours, 45 mins, 51 secs

Cluster Resiliency Detail:

-------------------------:

Health State Reason: Storage cluster is healthy.

# of nodes failure tolerable for cluster to be fully available: 1

# of node failures before cluster goes into readonly: NA

# of node failures before cluster goes to be crticial and partially available: 3

# of node failures before cluster goes to enospace warn trying to move the existing data: NA

# of persistent devices failures tolerable for cluster to be fully available: 2

# of persistent devices failures before cluster goes into readonly: NA

# of persistent devices failures before cluster goes to be critical and partially available: 3

# of caching devices failures tolerable for cluster to be fully available: 2

# of caching failures before cluster goes into readonly: NA

# of caching failures before cluster goes to be critical and partially available: 3

Current ensemble size: 3

Minimum data copies available for some user data: 3

Minimum cache copies remaining: 3

Minimum metadata copies available for cluster metadata: 3

Current healing status:

Time remaining before current healing operation finishes:

# of unavailable nodes: 0

hxshell:~$

This output shows an unhealthy cluster due to an unavailable node.

hxshell:~$ sysmtool --ns cluster --cmd healthdetail

Cluster Health Detail:

---------------------:

State: ONLINE                   <-------State of the cluster

HealthState: UNHEALTHY          <-------Health of the cluster

Policy Compliance: NON-COMPLIANT

Creation Time: Tue May 30 04:48:45 2023

Uptime: 7 weeks, 19 hours, 55 mins, 9 secs

Cluster Resiliency Detail:

-------------------------:

Health State Reason: Storage cluster is unhealthy.Storage node 172.16.3.9 is unavailable.                  <----------- Health state reason

# of nodes failure tolerable for cluster to be fully available: 0

# of node failures before cluster goes into readonly: NA

# of node failures before cluster goes to be crticial and partially available: 2

# of node failures before cluster goes to enospace warn trying to move the existing data: NA

# of persistent devices failures tolerable for cluster to be fully available: 1

# of persistent devices failures before cluster goes into readonly: NA

# of persistent devices failures before cluster goes to be critical and partially available: 2

# of caching devices failures tolerable for cluster to be fully available: 1

# of caching failures before cluster goes into readonly: NA

# of caching failures before cluster goes to be critical and partially available: 2

Current ensemble size: 3

Minimum data copies available for some user data: 2

Minimum cache copies remaining: 2

Minimum metadata copies available for cluster metadata: 2

Current healing status: Rebuilding/Healing is needed, but not in progress yet. Warning: Insufficient node or space resources may prevent healing. Storage Node 172.16.3.9 is either down or initializing disks.

Time remaining before current healing operation finishes:

# of unavailable nodes: 1

hxshell:~$

This output shows an unhealthy cluster due to the rebuilding.

Cluster Health Detail:

---------------------:

State: ONLINE

HealthState: UNHEALTHY

Policy Compliance: NON-COMPLIANT

Creation Time: Tue May 30 04:48:45 2023

Uptime: 7 weeks, 20 hours, 2 mins, 4 secs

Cluster Resiliency Detail:

-------------------------:

Health State Reason: Storage cluster is unhealthy.

# of nodes failure tolerable for cluster to be fully available: 1

# of node failures before cluster goes into readonly: NA

# of node failures before cluster goes to be crticial and partially available: 2

# of node failures before cluster goes to enospace warn trying to move the existing data: NA

# of persistent devices failures tolerable for cluster to be fully available: 1

# of persistent devices failures before cluster goes into readonly: NA

# of persistent devices failures before cluster goes to be critical and partially available: 2

# of caching devices failures tolerable for cluster to be fully available: 1

# of caching failures before cluster goes into readonly: NA

# of caching failures before cluster goes to be critical and partially available: 2

Current ensemble size: 3

Minimum data copies available for some user data: 3

Minimum cache copies remaining: 2

Minimum metadata copies available for cluster metadata: 2

Current healing status: Rebuilding is in progress, 58% completed.

Time remaining before current healing operation finishes: 18 hr(s), 10 min(s), and 53 sec(s)

# of unavailable nodes: 0

These commands show an overall summary of the health of the cluster and let you know if there is something is affecting the operation of the cluster, for instance, if there is a blacklisted disk, an offline node, or if the cluster is healing.

Nodes Participating In I/O

The performance can be impacted by a node not participating in the input and output operations, to check the nodes that are participating in I/O, issue these commands.

Tip: From the 5.0(2a) version, diag user is available to allow users to have more privileges to troubleshoot with access to restricted folders and commands that are not accessible via priv command line which was introduced in Hyperflex version 4.5.x.

Step 1. Enter into the diag shell on a storage controller VM.

hxshell:~$ su diag

Password:

_   _ _                      _  _             _____ _                      ___

| \ | (_)_ __   ___          | || |           |  ___(_)_   _____           / _ \ _ __   ___

|  \| | | '_ \ / _ \  _____  | || |_   _____  | |_  | \ \ / / _ \  _____  | | | | '_ \ / _ \

| |\  | | | | |  __/ |_____| |__   _| |_____| |  _| | |\ V /  __/ |_____| | |_| | | | |  __/

|_| \_|_|_| |_|\___|            |_|           |_|   |_| \_/ \___|          \___/|_| |_|\___|

Enter the output of above expression: -1

Valid captcha

Step 2.  Issue this command to verify the nodes that are participating in I/O operations, the number of IPs must be equal to the number of converged nodes on the cluster.

diag# nfstool -- -m | cut -f2 | sort | uniq

172.16.3.7

172.16.3.8

172.16.3.9

Internal Services Check

Cleaner

One of the main objectives of Cleaner is to identify dead and live storage blocks in the system and remove the dead ones, freeing the storage space occupied by them It is a background job, and its aggressiveness is set based on a policy.

You can check the cleaner service by issuing the next command.

bash-4.2# stcli cleaner info

{ 'name': '172.16.3.7', 'id': '1f82077d-6702-214d-8814-e776ffc0f53c', 'type': 'node' }: OFFLINE                <----------- Cleaner shows as offline

{ 'name': '172.16.3.8', 'id': 'c4a24480-e935-6942-93ee-987dc8e9b5d9', 'type': 'node' }: OFFLINE

{ 'name': '172.16.3.9', 'id': '50a5dc5d-c419-9c48-8914-d91a98d43fe7', 'type': 'node' }: OFFLINE

In order to start the cleaner process, issue this command.

bash-4.2# stcli cleaner start

WARNING: This command should be executed ONLY by Cisco TAC support as it may have very severe consequences. Do you want to proceed ? (y/n): y

bash-4.2# stcli cleaner info

{ 'type': 'node', 'id': '1f82077d-6702-214d-8814-e776ffc0f53c', 'name': '172.16.3.7' }: ONLINE

{ 'type': 'node', 'id': 'c4a24480-e935-6942-93ee-987dc8e9b5d9', 'name': '172.16.3.8' }: ONLINE

{ 'type': 'node', 'id': '50a5dc5d-c419-9c48-8914-d91a98d43fe7', 'name': '172.16.3.9' }: ONLINE           <---------All nodes need to be online

bash-4.2#

Caution: This command must be executed with Cisco TAC approval.

Rebalance

The storage cluster is rebalanced on a regular schedule. It is used to realign the distribution of stored data across changes in available storage and to restore storage cluster health.

Rebalance runs in clusters for different reasons:

- A physical resource (node/disk) is down and HX is relocating those Vnodes to a different physical resource in the cluster.

- The individual drives across the cluster are not all comparably utilized and therefore some hotspots have been created in terms of data availability within the HX cluster (data collocation).

- Rebalance can also run if Zone compliance is not there even if the cluster is healthy.

- When a new node is added to the existing cluster, the added node(s) take on new writes as soon as it joins the existing cluster.

Verify that the cluster has rebalance enabled.

hxshell:~$ stcli rebalance status

rebalanceStatus:

percentComplete: 0

rebalanceState: cluster_rebalance_not_running

rebalanceEnabled: True     <---------Rebalance should be enabled

hxshell:~$

Caution: Any operation related to Rebalance must be done with Cisco TAC approval.

Disk Failure

For proper operation, the cluster must not have any blacklisted disks or offline resources.

You need to check if there is any blacklisted disk on the cluster in the HX Connect interface.

Blacklisted disk.

Check on the CLI for any offline resources on each Converge Node.

sysmtool --ns cluster --cmd offlineresources

UUID                                Type         State      InUse      Last modified

----                                ----         -----      -----      -------------

000cca0b019b4a80:0000000000000000   DISK         DELETED    YES          <------- Offline disk

5002538c405e0bd1:0000000000000000   DISK         BLOCKLISTED NO          <------- Blacklisted disk

5002538c405e299e:0000000000000000   DISK         DELETED    NO

Total offline resources: 3, Nodes: 0, Disks: 3

Verify if there are any blacklisted resources.

hxshell:~$ sysmtool --ns disk --cmd list | grep -i blacklist

Blacklist Count: 0

Blacklist Count: 0

Blacklist Count: 0

Blacklist Count: 0

State: BLACKLISTED

Blacklist Count: 5

Blacklist Count: 0

Blacklist Count: 0

You need to check if there is any failed disk in each Converge Node with this command.

admin:~$ cat /var/log/springpath/diskslotmap-v2.txt

0.0.1:5002538e000d59a3:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302248:HXT76F3Q:SATA:SSD:3662830:Inactive:/dev/sdj    <---------Inactive disk

1.0.2:5002538c40be79ac:Samsung:SAMSUNG_MZ7LM240HMHQ-00003:S4EGNX0KC04551:GXT51F3Q:SATA:SSD:228936:Active:/dev/sdb

1.0.3:5002538e000d599e:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302243:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdc

1.0.4:5002538e000d59a0:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302245:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdd

1.0.5:5002538e000eb00b:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302480:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdi

1.0.6:5002538e000d599b:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302240:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdf

1.0.7:5002538e000d57f6:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M301819:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdh

1.0.8:5002538e000d59ab:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302256:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sde

1.0.9:5002538e000d59a1:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M302246:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdg

1.0.10:5002538e0008c68f:Samsung:SAMSUNG_MZ7LH3T8HMLT-00003:S4F3NY0M200500:HXT76F3Q:SATA:SSD:3662830:Active:/dev/sdj

0.1.192:000cca0b01c83180:HGST:UCSC-NVMEHW-H1600:SDM000026904:KNCCD111:NVMe:SSD:1526185:Active:/dev/nvme0n1

admin:~$

Example of a Node without any disk failure.

hxshell:~$ sysmtool --ns cluster --cmd offlineresources

No offline resources found              <-------- No offline resources

hxshell:~$ sysmtool --ns disk --cmd list | grep -i blacklist

hxshell:~$                              <-------- No blacklisted disks

hxshell:~$ cat /var/log/springpath/diskslotmap-v2.txt

1.14.1:55cd2e404c234bf9:Intel:INTEL_SSDSC2BX016T4K:BTHC618505B51P6PGN:G201CS01:SATA:SSD:1526185:Active:/dev/sdc

1.14.2:5000c5008547c543:SEAGATE:ST1200MM0088:Z4009D7Y0000R637KMU7:N0A4:SAS:10500:1144641:Active:/dev/sdd

1.14.3:5000c5008547be1b:SEAGATE:ST1200MM0088:Z4009G0B0000R635L4D3:N0A4:SAS:10500:1144641:Active:/dev/sde

1.14.4:5000c5008547ca6b:SEAGATE:ST1200MM0088:Z4009F9N0000R637JZRF:N0A4:SAS:10500:1144641:Active:/dev/sdf

1.14.5:5000c5008547b373:SEAGATE:ST1200MM0088:Z4009GPM0000R634ZJHB:N0A4:SAS:10500:1144641:Active:/dev/sdg

1.14.6:5000c500854310fb:SEAGATE:ST1200MM0088:Z4008XFJ0000R6374ZE8:N0A4:SAS:10500:1144641:Active:/dev/sdh

1.14.7:5000c50085424b53:SEAGATE:ST1200MM0088:Z4008D2S0000R635M4VF:N0A4:SAS:10500:1144641:Active:/dev/sdi

1.14.8:5000c5008547bcfb:SEAGATE:ST1200MM0088:Z4009G3W0000R637K1R8:N0A4:SAS:10500:1144641:Active:/dev/sdj

1.14.9:5000c50085479abf:SEAGATE:ST1200MM0088:Z4009J510000R637KL1V:N0A4:SAS:10500:1144641:Active:/dev/sdk

1.14.11:5000c5008547c2c7:SEAGATE:ST1200MM0088:Z4009FR00000R637JPEQ:N0A4:SAS:10500:1144641:Active:/dev/sdl

1.14.13:5000c5008547ba93:SEAGATE:ST1200MM0088:Z4009G8V0000R634ZKLX:N0A4:SAS:10500:1144641:Active:/dev/sdm

1.14.14:5000c5008547b69f:SEAGATE:ST1200MM0088:Z4009GG80000R637KM30:N0A4:SAS:10500:1144641:Active:/dev/sdn

1.14.15:5000c5008547b753:SEAGATE:ST1200MM0088:Z4009GH90000R635L5F6:N0A4:SAS:10500:1144641:Active:/dev/sdo

1.14.16:5000c5008547ab7b:SEAGATE:ST1200MM0088:Z4009H3P0000R634ZK8T:N0A4:SAS:10500:1144641:Active:/dev/sdp  <------All disks are active

hxshell:~$

Free Memory

Check the free memory with this command, the free memory must be more than 2048 MB (free +cache).

hxshell:~$ free –m

total        used        free      shared  buff/cache   available

Mem:       74225624    32194300    38893712        1672     3137612    41304336

Swap:             0           0           0

hxshell:~$

if the free + cache memory is less than 2048, is necessary to identify the process that is generating the Out Of Memory condition.

Note: You can use the top command to identify processes that consume a lot of memory, however, any changes have to be done with TAC approval, contact Cisco TAC to troubleshoot OOM conditions.

End Of Space Condition

The best practice of storage cluster space utilization is to not go beyond 76 percent at the HX Connect capacity view. Beyond 76 percent, usage at the HX Connect capacity view results in performance degradation.

If the storage cluster is experiencing an ENOSPC condition, the cleaner automatically runs at high priority, which can create performance issues in the cluster, the priority is determined by cluster space usage.

If the storage cluster reaches an ENOSPC WARN condition, the cleaner increases its intensity by increasing the number of I/O to collect garbage with an ENOSPC set condition, it runs at the highest priority.

You can check the ENOSPCINFO status on the cluster with this command.

hxshell:~$ sysmtool --ns cluster --cmd enospcinfo

Cluster Space Details:

---------------------:

Cluster state: ONLINE

Health state: HEALTHY

Raw capacity: 42.57T

Usable capacity: 13.06T

Used capacity: 163.08G

Free capacity: 12.90T

Enospc state: ENOSPACE_CLEAR    <--------End of space status

Space reclaimable: 0.00

Minimum free capacity

required to resume operation: 687.12G

Space required to clear

ENOSPC warning: 2.80T           <--------Free space until the end of space warning appears

Rebalance In Progress: NO

Flusher in progress: NO

Cleaner in progress: YES

Disk Enospace: NO

hxshell:~$

Review the Capacity Management in Cisco HyperFlex White Paper to identify the best practice to manage the space on your Hyperflex Cluster.

Troubleshooting Performance Charts

Sometimes the hyperflex performance charts are not displaying information.

Hyperflex performance charts.

If you face this behavior you need to review if the stats services are running in the cluster.

hxshell:~$ priv service carbon-cache status

carbon-cache stop/waiting

hxshell:~$ priv service carbon-aggregator status

carbon-aggregator stop/waiting

hxshell:~$ priv service statsd status

statsd stop/waiting

If the processes are not running, manually start the services.

hxshell:~$ priv service carbon-cache start

carbon-cache start/running, process 15750

hxshell:~$ priv service carbon-aggregator start

carbon-aggregator start/running, process 15799

hxshell:~$ priv service statsd start

statsd start/running, process 15855

Related Information

- Capacity Management in Cisco HyperFlex White Paper.

- Mission-critical Workload Performance Testing of Different Hyperconverged Approaches on the Cisco Unified Computing System Platform (UCS)

- Technical Support & Documentation - Cisco Systems.