Introdução

Este documento descreve o processo para executar a ferramenta da saúde & da PRE-elevação de Hypercheck. Esta ferramenta é uma utilidade para executar auoes-avaliação pro-ativos em sistemas do hyperflex para assegurar suas estabilidade e elasticidade. Ajuda a automatizar uma lista de verificações da saúde e da PRE-elevação em sistemas do hyperflex para ganhar o tempo durante a elevação e as operações de manutenção do hyperflex.

NOTA: Sempre a versão a mais atrasada da transferência da ferramenta antes que você a usar. Desde que a ferramenta é aumentada frequentemente, usar uma versão mais velha pode conduzir a faltar verificações importantes.

Sistemas apoiados HX 

• Versões 1.8 de Hyperflex, 2.0, 2.1, 2.5, 2.6, 3.0, 3.5, 4.0
• Conjunto do padrão de Hyperflex
• Conjunto da borda de Hyperflex (nó 2 nó, 3 nó & 4)
• Apoiado somente no conjunto de Hyperflex em VMware ESXi

Quando usar

• Antes das elevações de Hyperflex.
• Exame médico completo de Hyperflex antes e depois das janelas de manutenção
• Para identificar falhas na unidade/discos.
• Ao trabalhar com tac Cisco
• Exame médico completo pro-ativo a qualquer momento.

Como se usar

Etapa 1. A transferência Hyperflex-Hypercheck.zip do devnet do github de Cisco explica aqui. Obtenha por favor a cópia a mais atrasada que terá as melhorias e as atualizações as mais atrasadas. 

NOTA: Use somente o script transferido da conta do devnet do github de Cisco.

Etapa 2. Transfira-a arquivos pela rede ao controlador VM do armazenamento (SCVM) com o IP do gerenciamento de grânulos (o CMIP)

Use seu método preferido - scp/sftp/ftp/tftp - para copiar o Hyperflex-Hypercheck.zip ao diretório de /tmp

Para o MAC:

Execute o SCP do CLI (confirme por favor que o Hyperflex-Hypercheck.zip está no mesmo dobrador de onde você está executando o scp)

# scp Hyperflex-Hypercheck.zip root@<scvm-eth0:mgmtip>:/tmp/

Use o seguinte para identificar o IP do gerenciamento de grânulos em seu manual de Hyperflex do ambiente HX

Para Windows:

Nós podemos usar WINSCP para transferir como mostrado os arquivos abaixo

Etapa 3. Extraia os índices de Hyperflex-Hypercheck.zip

Datilografe o CD /tmp para mudar a /tmp o diretório 

root@SpringpathController7PVQWP6WV1:~# cd /tmp/

O tipo abre o zíper Hyperflex-Hypercheck.zip para extrair os arquivos 

root@SpringpathController7PVQWP6WV1:/tmp# unzip Hyperflex-Hypercheck.zip 
Archive:  Hyperflex-Hypercheck.zip
b61c59f7962b72902692ce70548ba3d760efdf06
   creating: Hyperflex-Hypercheck/
  inflating: Hyperflex-Hypercheck/HXTool.py 
  inflating: Hyperflex-Hypercheck/LICENSE.txt 
  inflating: Hyperflex-Hypercheck/ReadMe.txt 
  inflating: Hyperflex-Hypercheck/TestInfo.txt 
  inflating: Hyperflex-Hypercheck/prettytable.py 
  inflating: Hyperflex-Hypercheck/progressbar.py 
root@SpringpathController7PVQWP6WV1:/tmp# 

Etapa 4. Execute o script do pitão de HXTool

Datilografe o CD Hyperflex-Hypercheck para ir ao diretório de Hyperflex-Hypercheck 

root@SpringpathControllerABCDE01234:/tmp# cd Hyperflex-Hypercheck

Datilografe o pitão HXTool.py para executar o script 

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# python HXTool.py

Etapa 5. Incorpore a senha root SCVM quando alertado 

Please enter below info of HX-Cluster:
Enter the HX-Cluster Root Password:
Enter the ESX Root Password:

NOTA: Para parar a execução do script use o [CTRL+Z] chave e parará imediatamente

Etapa 6. A ferramenta de Hyperflex-Hypercheck começa suas verificações. Tomará aproximadamente 3-10 minutos para que a execução termine segundo o número de Nós convirgidos no conjunto

Etapa 7. Compreendendo as saídas/verificações executadas

As seguintes verificações são executadas por Hyperflex-Hyperchecktool

Hyperflex Checks: (Below checks are performed on all the storage controller VMs)
    Cluster services check - Verifies the status of storfs, stMgr and stNodeMgr services. 
    Enospc state check - Checks if the cluster space usage is above the warning threshold or no.     
    Zookeeper check - Checks whether the Zookeeper is running or no.
    Exhibitor check - Verfies the status of the Exhibitor service which manages the ZK.          
    HDD health check - Reports if you have any blacklisted disk in your cluster.        
    DNS check - Checks whether DNS is configured and reachable.                 
    vCenter reachability check - Checks whether the vCenter is reachable on the required ports.
    Timestamp check - Checks if all the controller VMs have the exact same time.           
    NTP sync check - Checks whether NTP is reachable from the storage controller VMs and synced.
    Check package & versions - Checks for packages and versions on Storage Controller VMs.
    Check Iptables count - Checks for Iptables count on and ensure it is same on all Storage Controller VMs.          
    Cluster upgrade check - Checks if there are any previous stale upgrade entries.     
    Extra pnodes check - Looks for any extra/duplicate pnode entries in the cluster.       
    Disk usage(/var/stv) check - Checks if the utilization of /var/stv is more that 80% or no.
    Out of memory check - Checks through the log files if the cluster had any oom event.
    Supported vSphere versions - Shows all the vSphere Versions supported with your current HXDP version.
    Network Checks - Checks the connectivity between Mgmt and Storage network.

ESXi Checks: (Below checks are performed on each ESXI node)
    HX User Account check - Verifies if the HXUser is created on all the esxi hosts and has admin rights.
    vMotion enabled check - Checks if the vMotion network is confirgued.
    vMotion reachability check - Checks the connectivity between vmotion network.
    Check for ESXI Failback timer - Check for ESXi Failback timer on ESXi host.
    Check ping to vmk0, eth0, eth1 - Checks the connectivity between the Mgmt and Storage network.
    No extra controller vm folders check - Checks if there are any extra controller VM folders in the Springpath datastore.
    Check the dumpfile in springpathDS - Check to confirm if SpringpathDS is configured for coredump.

*vMotion reachability check is only performed if vMotion enabled check is successful/PASS*

Etapa 8. Obtenha o relatório das saídas do script. Você pode obtê-lo como mostrado abaixo,

O arquivo TAR do relatório de Hypercheck salvar sob /var/log/springpath e /tmp/Hyperflex-Hypercheck. Assim, você pode transferir o pacote do alcatrão de debaixo de /var/log/springpath or/tmp/Hyperflex-Hypercheck. Alternativamente, você pode apenas gerar e transferir arquivos pela rede um pacote do storfs-apoio que igualmente contenha o alcatrão do relatório do hypercheck.

Relate o exemplo do arquivo TAR - HX_Report_23_09_2019_15_32_22.tar é copiado ao trajeto: /var/log/springpath

Tipo ls - l | grep HX_Report para rever os arquivos criados pela ferramenta de Hyperflex-Hypercheck

Under /var/log/springpath,

root@SpringpathControllerABCDE01234:/var/log/springpath# ls -l | grep HX_Report
-rw-r--r-- 1 root root 380K Sep 23 15:41 HX_Report_23_09_2019_15_32_22.tar
root@SpringpathControllerABCDE01234:/var/log/springpath#

Under /tmp/Hyperflex-Hypercheck,

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# ls
HX_Report_23_09_2019_15_32_22.tar prettytable.py HX_Report_23_09_2019_15_32_22 TestInfo.txt progressbar.py
HXTool.py prettytable.pyc ReadMe.txt progressbar.pyc LICENSE.txt
root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# 

Os arquivos e entram o pacote do log de Hypercheck

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# ls HX_Report_13_02_2019_16_15/
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.181.txt
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.179.txt
HX_Report_13_02_2019_16_15/HX_Tool_13-02-2019_04-15-49 PM.log
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.180.txt
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.182.txt
HX_Report_13_02_2019_16_15/HX_Tool_Main_Report.txt

Etapa 9. A exportação HX_Report_DD_MM_YYYY_HH_MM_SS.tar e compartilha d com o TAC.

Use seu método preferido para exportar os logs de Hypercheck usando o scp/sftp/ftp/tftp do SCVM ou você pode simplesmente transferir o pacote do apoio dos storfs que conterá o pacote do alcatrão de HX_Report.

Etapa 10. Amostra HXTool Output de um conjunto 3-node.

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# python HXTool.py

                 HX Health Check 3.5

Please enter below info of HX-Cluster:
Enter the HX-Cluster Root Password:
Enter the ESX Root Password:

SSH connection established to HX Node: 10.X.X.79
SSH connection established to HX Node: 10.X.X.80
SSH connection established to HX Node: 10.X.X.81

HX Cluster Nodes:
+-------+---------------+--------------------------------+
| Nodes | IP Address    | HostName                       |
+-------+---------------+--------------------------------+
| 1     | 10.X.X.79     | SpringpathControllerABCD123456 |
+-------+---------------+--------------------------------+
| 2     | 10.X.X.80     | SpringpathControllerABCD123457 |
+-------+---------------+--------------------------------+
| 3     | 10.X.X.81     | SpringpathControllerABCD123458 |
+-------+---------------+--------------------------------+

SSH connection established to ESX Host: 10.X.X.76
SSH connection established to ESX Host: 10.X.X.75
SSH connection established to ESX Host: 10.X.X.77

HX Controller: 10.X.X.79
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS

HX Controller: 10.X.X.80
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS

HX Controller: 10.X.X.81
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS


HX Controller: 10.X.X.79
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

HX Controller: 10.X.X.80
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

HX Controller: 10.X.X.81
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

################################################################################
			Network check:
################################################################################

ESX vmk0: 10.X.X.75, 10.X.X.76, 10.X.X.77

ESX vmk1: 10.X.X.102, 10.X.X.103, 10.X.X.104

SCVM eth0: 10.X.X.79, 10.X.X.80, 10.X.X.81

SCVM eth1: 10.X.X.98, 10.X.X.99, 10.X.X.100

ESX Host: 10.X.X.76
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+ 

ESX Host: 10.X.X.75
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+

ESX Host: 10.X.X.77
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+


Main Report File: HX_Tool_Main_Report_23-09-2019_03-41-03_hx-01.txt
Report tar file: HX_Report_23_09_2019_15_32_22.tar
Report file copied to path: /var/log/springpath

Release Notes:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html

Upgrade Guides:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

Note:
1) Please check the status of Compute nodes manually, script only verifies the config on the converged nodes.
2) Hypercheck doesnot perform FAILOVER TEST, so please ensure that the upstream is configured for network connectivity for JUMBO or NORMAL MTU size as needed.

*vMotion reachability check is only performed if vMotion enabled check is successful/PASS*

Analise a ferramenta Output - Próximas etapas

• A ferramenta automatiza o processo de executar comandos manuais em sistemas de Hyperflex.
• Se a ferramenta é executado ESTÁ BEM e dá PASSE sobre todos os testes. O sistema HX é bom para todas as verificações que o script executou.
• Nas situações onde, a ferramenta FALHA em algum verificam ou não são executado com sucesso, você podem usar os comandos CLI (alistados abaixo) executar as mesmas verificações no sistema de Hyperflex que feito pelo script manualmente.
• A ferramenta não verifica para ver se há velho/novo/abre/interrupção de processo resolvida e daqui é altamente recomendado rever Release Note de Hyperflex e guias da elevação antes de alguns elevação ou atividade de manutenção.

NOTA: Não abra um caso de TAC porque o script não foi executado. Execute por favor os comandos manualmente, identifique a edição e abra um SÊNIOR para o problema identificado.

Comandos CLI

Em Hyperflex SCVM-

SSH to All Hyperflex SCVMs-

# service_status.sh

# sysmtool --ns cluster --cmd enospcinfo

# echo srvr | nc 0 2181

# pidof exhibitor

# stcli disk list --ip <Corresponding ESXi Mgmt IP Address> |grep -B 2 -A 8 blacklisted

# stcli services dns show (and ping the IPs listed)

# ping <vCenter IP Address>

# date ; compare the time on all SCVMs. They should ideally be identical

# stcli services ntp show

# stcli cleaner info

# ntpq -p -4 

# dpkg -l | grep -i springpath 

# iptables -L -n | wc -l 

# stcli cluster upgrade-status ; only applicable for HX version 2.5 and above

# stcli cluster info

# df -h ; check that /var/stv should not be more than 10-15% used

# zgrep -i “out of memory” /var/log/springpath/debug-storfs.*

# ping -I eth0 <eth0> of all SCVMs

# ping -I eth1 <eth1> of all SCVMs

Em sistemas de ESXi

SSH to all ESXi hosts

# esxcli system account list

# esxcli network firewall ruleset list | grep -i vMotion

# esxcli software vib list| grep -i spring 

# esxcfg-vmknic -l ; confirm that vMotion VMK2 is created

# vmkping -I vmk0 <eth0> of all SCVMs

# vmkping -I vmk1 <eth1> of all SCVMs

# cd /vmfs/volumes/Springpath-XXXXXXXXXX ; Ensure that it has only one Folder that has the Storage Controller VM