简介

本文描述进程运行Hypercheck健康&升级前工具。此工具是执行在hyperflex系统的积极的自检的工具保证其稳定性和弹性。在hyperflex升级和维护操作期间，它帮助自动化健康和升级前检查列表在hyperflex系统节省时间。

NOTE: 工具的总是下载新版本，在您使用它前。因为工具频繁地被提高，使用早版本可以导致未命中重要检查。

支持的HX系统 

• Hyperflex版本1.8， 2.0， 2.1， 2.5， 2.6， 3.0， 3.5， 4.0
• Hyperflex标准的集群
• Hyperflex边缘集群(2节点， 3节点& 4节点)
• 只支持在VMWare ESXi的Hyperflex集群

什么时候使用

• 在Hyperflex升级前。
• 在维护窗口前后的Hyperflex健康检查
• 识别发生故障的驱动器/磁盘。
• 当工作与Cisco TAC时
• 任何时候积极的健康检查。

如何使用

步骤1.从Cisco github devnet的下载Hyperflex-Hypercheck.zip认为得此处。 请获得将有最新的改进和更新的新副本。 

注意：请使用从Cisco github devnet帐户下载的仅脚本。

步骤2.上传它对存储设备控制器VM (SCVM)与集群管理IP (CMIP)

请使用您的首选方法- scp/sftp/ftp/tftp -复制Hyperflex-Hypercheck.zip到/tmp目录

MAC ：

执行从CLI的SCP (请确认Hyperflex-Hypercheck.zip在同一个文件夹从您运行scp)

# scp Hyperflex-Hypercheck.zip root@<scvm-eth0:mgmtip>:/tmp/

请使用以下识别在您的HX环境Hyperflex剧本的集群管理IP

Windows：

我们能使用WINSCP转接文件如下所示

步骤3.解压缩Hyperflex-Hypercheck.zip内容

键入cd /tmp变成/tmp目录 

root@SpringpathController7PVQWP6WV1:~# cd /tmp/

类型解Hyperflex-Hypercheck.zip压缩抽出文件 

root@SpringpathController7PVQWP6WV1:/tmp# unzip Hyperflex-Hypercheck.zip 
Archive:  Hyperflex-Hypercheck.zip
b61c59f7962b72902692ce70548ba3d760efdf06
   creating: Hyperflex-Hypercheck/
  inflating: Hyperflex-Hypercheck/HXTool.py 
  inflating: Hyperflex-Hypercheck/LICENSE.txt 
  inflating: Hyperflex-Hypercheck/ReadMe.txt 
  inflating: Hyperflex-Hypercheck/TestInfo.txt 
  inflating: Hyperflex-Hypercheck/prettytable.py 
  inflating: Hyperflex-Hypercheck/progressbar.py 
root@SpringpathController7PVQWP6WV1:/tmp# 

步骤4.执行HXTool Python脚本

键入cd Hyperflex-Hypercheck去Hyperflex-Hypercheck目录 

root@SpringpathControllerABCDE01234:/tmp# cd Hyperflex-Hypercheck

键入Python HXTool.py执行脚本 

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# python HXTool.py

步骤5.输入SCVM根密码，当提示 

Please enter below info of HX-Cluster:
Enter the HX-Cluster Root Password:
Enter the ESX Root Password:

注意：要终止脚本执行请使用关键[CTRL+Z]，并且将立即终止

步骤6. Hyperflex-Hypercheck工具开始其检查。将需要大约执行的3-10分钟能根据聚合的节点编号在集群的完成

步骤7.了解被执行的输出/检查

以下检查由Hyperflex-Hyperchecktool执行

Hyperflex Checks: (Below checks are performed on all the storage controller VMs)
    Cluster services check - Verifies the status of storfs, stMgr and stNodeMgr services. 
    Enospc state check - Checks if the cluster space usage is above the warning threshold or no.     
    Zookeeper check - Checks whether the Zookeeper is running or no.
    Exhibitor check - Verfies the status of the Exhibitor service which manages the ZK.          
    HDD health check - Reports if you have any blacklisted disk in your cluster.        
    DNS check - Checks whether DNS is configured and reachable.                 
    vCenter reachability check - Checks whether the vCenter is reachable on the required ports.
    Timestamp check - Checks if all the controller VMs have the exact same time.           
    NTP sync check - Checks whether NTP is reachable from the storage controller VMs and synced.
    Check package & versions - Checks for packages and versions on Storage Controller VMs.
    Check Iptables count - Checks for Iptables count on and ensure it is same on all Storage Controller VMs.          
    Cluster upgrade check - Checks if there are any previous stale upgrade entries.     
    Extra pnodes check - Looks for any extra/duplicate pnode entries in the cluster.       
    Disk usage(/var/stv) check - Checks if the utilization of /var/stv is more that 80% or no.
    Out of memory check - Checks through the log files if the cluster had any oom event.
    Supported vSphere versions - Shows all the vSphere Versions supported with your current HXDP version.
    Network Checks - Checks the connectivity between Mgmt and Storage network.

ESXi Checks: (Below checks are performed on each ESXI node)
    HX User Account check - Verifies if the HXUser is created on all the esxi hosts and has admin rights.
    vMotion enabled check - Checks if the vMotion network is confirgued.
    vMotion reachability check - Checks the connectivity between vmotion network.
    Check for ESXI Failback timer - Check for ESXi Failback timer on ESXi host.
    Check ping to vmk0, eth0, eth1 - Checks the connectivity between the Mgmt and Storage network.
    No extra controller vm folders check - Checks if there are any extra controller VM folders in the Springpath datastore.
    Check the dumpfile in springpathDS - Check to confirm if SpringpathDS is configured for coredump.

*vMotion reachability check is only performed if vMotion enabled check is successful/PASS*

步骤8.得到脚本输出的报告。您能获得它如下所示，

Hypercheck报告TAR文件保存在/var/log/springpath和/tmp/Hyperflex-Hypercheck下。因此，您能从/var/log/springpath or/tmp/Hyperflex-Hypercheck下面下载tar套件。 或者，您能生成和上传也将包含hypercheck报告tar的storfs支持套件。

报告TAR文件示例- HX_Report_23_09_2019_15_32_22.tar复制到路径：/var/log/springpath

类型ls - l|grep查看文件的HX_Report创建由Hyperflex-Hypercheck工具

Under /var/log/springpath,

root@SpringpathControllerABCDE01234:/var/log/springpath# ls -l | grep HX_Report
-rw-r--r-- 1 root root 380K Sep 23 15:41 HX_Report_23_09_2019_15_32_22.tar
root@SpringpathControllerABCDE01234:/var/log/springpath#

Under /tmp/Hyperflex-Hypercheck,

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# ls
HX_Report_23_09_2019_15_32_22.tar prettytable.py HX_Report_23_09_2019_15_32_22 TestInfo.txt progressbar.py
HXTool.py prettytable.pyc ReadMe.txt progressbar.pyc LICENSE.txt
root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# 

文件和登陆Hypercheck日志套件

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# ls HX_Report_13_02_2019_16_15/
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.181.txt
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.179.txt
HX_Report_13_02_2019_16_15/HX_Tool_13-02-2019_04-15-49 PM.log
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.180.txt
HX_Report_13_02_2019_16_15/HX_Report_10.197.163.182.txt
HX_Report_13_02_2019_16_15/HX_Tool_Main_Report.txt

步骤9.导出HX_Report_DD_MM_YYYY_HH_MM_SS.tar并且共享它与TAC。

请使用您的首选方法导出Hypercheck日志使用scp/sftp/ftp/tftp从SCVM或您能下载将包含HX_Report tar套件的storfs支持套件。

步骤10.示例从3 NODE集群的HXTool输出。

root@SpringpathControllerABCDE01234:/tmp/Hyperflex-Hypercheck# python HXTool.py

                 HX Health Check 3.5

Please enter below info of HX-Cluster:
Enter the HX-Cluster Root Password:
Enter the ESX Root Password:

SSH connection established to HX Node: 10.X.X.79
SSH connection established to HX Node: 10.X.X.80
SSH connection established to HX Node: 10.X.X.81

HX Cluster Nodes:
+-------+---------------+--------------------------------+
| Nodes | IP Address    | HostName                       |
+-------+---------------+--------------------------------+
| 1     | 10.X.X.79     | SpringpathControllerABCD123456 |
+-------+---------------+--------------------------------+
| 2     | 10.X.X.80     | SpringpathControllerABCD123457 |
+-------+---------------+--------------------------------+
| 3     | 10.X.X.81     | SpringpathControllerABCD123458 |
+-------+---------------+--------------------------------+

SSH connection established to ESX Host: 10.X.X.76
SSH connection established to ESX Host: 10.X.X.75
SSH connection established to ESX Host: 10.X.X.77

HX Controller: 10.X.X.79
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS

HX Controller: 10.X.X.80
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS

HX Controller: 10.X.X.81
Cluster services check      [########################] PASS
ZooKeeper & Exhibitor check [########################] PASS
HDD health check            [########################] PASS
Pre-Upgrade Check           [########################] PASS
Network check               [########################] PASS


HX Controller: 10.X.X.79
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

HX Controller: 10.X.X.80
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

HX Controller: 10.X.X.81
Test Summary:
+----------------------------+-----------+----------+
| Name                       | Result    | Comments |
+----------------------------+-----------+----------+
| Cluster services check     | PASS      |          |
+----------------------------+-----------+----------+
| Enospc state check         | PASS      |          |
+----------------------------+-----------+----------+
| Zookeeper check            | PASS      |          |
+----------------------------+-----------+----------+
| Exhibitor check            | PASS      |          |
+----------------------------+-----------+----------+
| HDD health check           | PASS      |          |
+----------------------------+-----------+----------+
| DNS check                  | PASS      |          |
+----------------------------+-----------+----------+
| vCenter reachability check | PASS      |          |
+----------------------------+-----------+----------+
| Timestamp check            | PASS      |          |
+----------------------------+-----------+----------+
| NTP sync check             | PASS      |          |
+----------------------------+-----------+----------+
| Check package & versions   | PASS      |          |
+----------------------------+-----------+----------+
| Check Iptables count       | PASS      |          |
+----------------------------+-----------+----------+
| Cluster upgrade status     | PASS      |          |
+----------------------------+-----------+----------+
| Extra pnodes check         | PASS      |          |
+----------------------------+-----------+----------+
| Disk usage(/var/stv) check | PASS      |          |
+----------------------------+-----------+----------+
| Memory usage check         | PASS      |          |
+----------------------------+-----------+----------+
| Incidence of OOM           | PASS      |          |
+----------------------------+-----------+----------+
| Supported vSphere versions | 6.0.0-U3  |          |
|                            | 6.5.0-U1  |          |
|                            | 6.5.0-U2  |          |
|                            | 6.7.0-UGA |          |
|                            | 6.7.0-U1  |          |
+----------------------------+-----------+----------+

################################################################################
			Network check:
################################################################################

ESX vmk0: 10.X.X.75, 10.X.X.76, 10.X.X.77

ESX vmk1: 10.X.X.102, 10.X.X.103, 10.X.X.104

SCVM eth0: 10.X.X.79, 10.X.X.80, 10.X.X.81

SCVM eth1: 10.X.X.98, 10.X.X.99, 10.X.X.100

ESX Host: 10.X.X.76
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+ 

ESX Host: 10.X.X.75
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+

ESX Host: 10.X.X.77
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Name                                 | Result | Comments                                                                        |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| HX User Account check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion enabled check                | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| vMotion reachability check           | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check for ESXI Failback timer        | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check ping to vmk0, eth0, eth1       | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| Check the dump in springpathDS       | PASS   |                                                                                 | 
+--------------------------------------+--------+---------------------------------------------------------------------------------+
| No extra controller vm folders check | PASS   |                                                                                 |
+--------------------------------------+--------+---------------------------------------------------------------------------------+


Main Report File: HX_Tool_Main_Report_23-09-2019_03-41-03_hx-01.txt
Report tar file: HX_Report_23_09_2019_15_32_22.tar
Report file copied to path: /var/log/springpath

Release Notes:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html

Upgrade Guides:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

Note:
1) Please check the status of Compute nodes manually, script only verifies the config on the converged nodes.
2) Hypercheck doesnot perform FAILOVER TEST, so please ensure that the upstream is configured for network connectivity for JUMBO or NORMAL MTU size as needed.

*vMotion reachability check is only performed if vMotion enabled check is successful/PASS*

分析输出的工具-以下步骤

• 工具自动化运行手工的on命令Hyperflex系统进程。
• 如果工具运行好并且给在所有测验的PASS。HX系统是有效对于脚本执行了的所有检查。
• 在，在一些检查的工具FAIL或不成功运行，您能使用CLI命令的情况下(如下所示)执行在Hyperflex系统的同样检查象手工完成由脚本。
• 工具不检查其中任一旧有/新建/打开/解决的警告并且是高度推荐的在其中任一前查看Hyperflex版本注释和升级指南升级或维护活动。

注意： 因为脚本失败运行，请勿开TAC案例。请手工请运行命令，识别问题并且打开识别的问题的SR。

CLI命令

在Hyperflex SCVM-

SSH to All Hyperflex SCVMs-

# service_status.sh

# sysmtool --ns cluster --cmd enospcinfo

# echo srvr | nc 0 2181

# pidof exhibitor

# stcli disk list --ip <Corresponding ESXi Mgmt IP Address> |grep -B 2 -A 8 blacklisted

# stcli services dns show (and ping the IPs listed)

# ping <vCenter IP Address>

# date ; compare the time on all SCVMs. They should ideally be identical

# stcli services ntp show

# stcli cleaner info

# ntpq -p -4 

# dpkg -l | grep -i springpath 

# iptables -L -n | wc -l 

# stcli cluster upgrade-status ; only applicable for HX version 2.5 and above

# stcli cluster info

# df -h ; check that /var/stv should not be more than 10-15% used

# zgrep -i “out of memory” /var/log/debug-storfs.*

# ping -I eth0 <eth0> of all SCVMs

# ping -I eth1 <eth1> of all SCVMs

在ESXi系统

SSH to all ESXi hosts

# esxcli system account list

# esxcli network firewall ruleset list | grep -i vMotion

# esxcli software vib list| grep -i spring 

# esxcfg-vmknic -l ; confirm that vMotion VMK2 is created

# vmkping -I vmk0 <eth0> of all SCVMs

# vmkping -I vmk1 <eth1> of all SCVMs

# cd /vmfs/volumes/Springpath-XXXXXXXXXX ; Ensure that it has only one Folder that has the Storage Controller VM