Hyperflex Playbook For Easier Administration

Introduction

When you are troubleshooting Hyperflex with Cisco Support, they would often request you to login to all the SCVM and/or the ESXi's if need be.

This document introduces one of the tools Cisco Support would generally ask you to install so that adminstation of the SCVM's becomes easier.

This document also introduces you to commands that are helpful in day to day Hyperflex administration.

SCVM = Storage Controller Virtual Machine

Q. How do I view all the SCVM status without having to putty into each one separately ?

A.

When you are troubleshooting a problem with Cisco Support, they would want to have a look at the status of all the SCVM's in your environment. 

    To make the adminstration easier:

Fow Windows:

    MobaXtermis recommended to assist with multiexec on all SCVMs and ESXi hosts

For Mac:

   iTerm2 is recommended to assist with multiexec on all SCVMs and ESXi hosts

SSH client software

• MobaXterm software download URL: MobaXterm Software Download
• iTerm2 software download URL iTerm2 Software Download

MobaXTerm

1. Select Session

2. Select SSH, enter crendtials. Repeat for the number of sessions you want to launch

3. Merge the sessions by selecting MultiExec

iTerm2

1. Click on Shell

2. Selct "Split Vertically/Horizontally with Current Profile" for the session you want

3. Merge the sessions by selecting Broadcast Input

Q. I can't seem to recollect a command and tab won't auto complete ?

A. 

Suffix the command with -h or --help

Example:

stcli -h

usage: stcli [-h]

             {about,services,vm,dp,snapshot-schedule,cluster,appliance,node,disk,cleaner,datastore,file,security,license,rebalance}

        

You could also go to the web CLI and run the commands from there. (There are a few handy commands in here, but its not full blown like the CLI is)

As you type the commands in the Web CLI wil pop with a handful suggestions

    

Q. How do I check the serial number, model and version of HXDP all from a single command ?

A.   

root@SpringpathController4ZH     SN:~# stcli about

serviceType: stMgr

instanceUuid: 71   904-4  9-   -8810-    ea4f513

name: HyperFlex StorageController

locale: English (United States)

serialNumber: WZJ       PN

apiVersion: 0.1

modelNumber: HX220C-M5SX

build: 3.0.1i-29888 (internal)

displayVersion: 3.0(1i)

fullName:  HyperFlex StorageController 3.0.1i

productVersion: 3.0.1i-29888

Q. I wonder what my ESXi and SCVM IPs are ?

A.

root@SpringpathController4ZH     SN:~# stcli cluster info

<++snip++>

nodeIPSettings:

        10.xx.xx.xx:  <<<<<<< ESXi IP address 

            ----------------------------------------

            addr: 10.xx.xx.xx

            stService: hypervisor

            vlanId: 421

            gateway:10.xx.xx.xx

            subnetMask: 255.255.255.224

            method: static

            ----------------------------------------

            addr: 192.xx.xx.xx

            stService: vmotion

            vlanId: 424

            gateway:

            subnetMask: 255.255.255.224

            method: static

            ----------------------------------------

            addr: 10..xx.xx.xx <<<<<<<<< SCVM IP address

            stService: stctl_mgmt

            vlanId: 421

            gateway: 10..xx.xx.xx

            subnetMask: 255.255.255.224

            method: static

            ----------------------------------------

            addr: 192..xx.xx.xx

            stService: stctl

            vlanId: 424

            gateway:

            subnetMask: 255.255.255.224

            method: static

            ----------------------------------------

            stService: ipmi

            vlanId: 421

            gateway: 10..xx.xx.xx

            subnetMask: 255.255.255.224

            method: dhcp

Output of the other esxi and scvm have been truncated

<++snip++>

Q. Which ESXi server does the SCVM I've logged into live on ? 

A.

root@SpringpathController4ZH   SN:~#  /opt/springpath/storfs-mgmt-cli/getLocalNode.sh

nodeId=7e    c4-b a-b e-b  6-6dc     865

serialNumber=WZJ    PN

chassisSerialNumber=WZJ      PN

esxiIP=10.xx.xx.xx

Q. What is my cluster management IP (HX Connect IP) ?

A. 

Runifconfig -a on all the SCVM's the one with eth0:mgmtip is the cluster management IP

eth0:mgmtip Link encap:Ethernet  HWaddr 00:0c:29:0f:61:81 

          inet addr:10..xx.xx.xx  Bcast:10..xx.xx.xxMask:255.255.255.224

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Info can also be obtained from 

stcli cluster info

<++snip++>

    mgmtIp:

        addr: 10..xx.xx.xx

        stService: mgmt

        vlanId: -1

        gateway:

        subnetMask: 255.255.255.224

        method: static

<++snip++>

Q. What are the IP addresses of all my SCVM's, UCSM and vCenter

A.

root@SpringpathControllerMAN 3Z:~# stcli cluster info | grep -i stctl_mgmt -n5
gateway: 192.168.52.30
subnetMask: 255.255.255.224
method: static
----------------------------------------
addr: 10.xx.xx.xx
stService: stctl_mgmt
vlanId: 361
gateway: 10.xx.xx.xx
subnetMask: 255.255.255.224
method: static
----------------------------------------

addr: 10.xx.xx.xx
stService: stctl_mgmt
vlanId: 361
gateway: 14.xx.xx.xx
subnetMask: 255.255.255.224
method: static
----------------------------------------

addr: 10.xx.xx.xx
stService: stctl_mgmt
vlanId: 361
gateway: 10.xx.xx.xx
subnetMask: 255.255.255.224
method: static
----------------------------------------

root@SpringpathControllerMAN 3Z:~# stcli cluster info | grep -i ucsm
ucsmOrg: VaultTech
ucsmHostName: 10.xx.xx.xx


root@SpringpathControllerMAN 3Z:~# stcli cluster info | grep -i vcenter
vCenterUrl: https://10.xx.xx.xx
vCenterClusterId: domain-c55
vCenterDatacenter: DC-HX-06
vCenterSSOURL:
vCenterURL: 10.xx.xx.xx
vCenterDatacenterId: datacenter-50
vCenterClusterName: HX-06

Q. What were the NTP, DNS IP's again ?

A.

root@SpringpathController4ZH     SN:~# stcli services dns show

----------------------------------------

172..xx.xx.xx

----------------------------------------

172..xx.xx.xx

----------------------------------------

root@SpringpathController4ZH    SN:~# stcli services ntp show

172..xx.xx.xx

Use -h to find other services 

root@SpringpathController4ZH    SN:~# stcli services -h

usage: stcli services [-h] {smtp,dns,ntp,asup,sch,remotesupport,timezone} ...

system services related operations

positional arguments:

  {smtp,dns,ntp,asup,sch,remotesupport,timezone}

                        stCli system services

    smtp                Commands supported in the Storage SMTP configuration

                        namespace.

    dns                 Commands supported in the Storage DNS configuration

                        namespace.

    ntp                 Commands supported in the Storage NTP configuration

                        namespace.

    asup                Commands supported in the ASUP configuration

                        namespace.

    sch                 Commands supported in the smart-callhome configuration

                        namespace.

    remotesupport       Commands supported for remote support.

    timezone            Commands supported in the Timezone configuration

                        namespace.

Q. Can I monitor the state of my cluster in real time during an upgrade ?

A.

root@SpringpathControllerGDA    LE:~# sysmtool --ns cluster --cmd healthdetail | grep -i state
State: ONLINE
HealthState: HEALTHY
Health State Reason: Storage cluster is healthy.

Q.  I know "stcli cluster storage-summary" gives me the health of my cluster, but it seems to be failing to return output. Is there any alternate command to make sure that the cluster is healthy before I press the panic button or call Cisco Support ?

A. 

If you expereince a symptom like so

stcli cluster storage-summary --detail

Get cluster storage summary failed:

Cluster invalid state

sysmtool --ns cluster --cmd info

<++snip++>

Cluster Resiliency State: HEALTHY

  Persistent device failures tolerable: 2

  Caching device failures tolerable: 2

Node failures tolerable: 1

Storage cluster is healthy.

Q. How do I view the disks that the SCVM I'm on has access to ?

A.

root@SpringpathController4ZH    SN:~# lsscsi

[2:0:0:0]    disk    ATA      SAMSUNG MZ    40 1F3Q  /dev/sdb

[2:0:1:0]    disk    TOSHIBA  AL14    0N      5705  /dev/sdc

[2:0:2:0]    disk    TOSHIBA  AL14    0N      5705  /dev/sdd

[2:0:3:0]    disk    ATA      INTEL SSD    48 CS01  /dev/sde

[2:0:4:0]    disk    TOSHIBA  AL14    0N      5705  /dev/sdf

[2:0:5:0]    disk    TOSHIBA  AL14    0N      5705  /dev/sdg

[2:0:6:0]    disk    TOSHIBA  AL14   20N      5705  /dev/sdh

[2:0:7:0]    disk    TOSHIBA  AL14   20N      5705  /dev/sdi

[3:0:0:0]    disk    VMware   Virtual disk     2.0   /dev/sda

You can also use

sysmtool --ns disk --cmd list

Disk Details:

------------:

UUID: 5000039888436425:0000000000000000

Type: DISK

Medium: ROTATIONAL

Usage: PERSISTENCE

State: CLAIMED

Runtime State: None

Blacklist Count: 0

Progress Pct: 0.000000

Path: /dev/sdg

Capacity: 1.09T

Usage: 0.00

Last Modified Time: 1553031537000

Version: 0

<++snip++>

Q. After generating the techsupport (storfs-support) bundle, How do I get it off from the controller VM ?

A.  

SCVM:

/var/support# ls
storfs-support_2019-03-22--17-08-51_SpringpathController4ZH    SN.tar.gz

For mac/linux :

scp root@14.39.54.65:/var/support/storfs-support_2019-03-22--17-08-51_SpringpathController4ZH    SN.tar.gz /Users/mohammhu/Desktop/

HyperFlex StorageController 3.0(1i)
root@14.39.54.65's password:
storfs-support_2019-03-22--17-08-51_SpringpathController4ZH    SN.tar.gz 100% 50MB 1.3MB/s 00:38

Syntax:

To copy files from a remote host to local host

scp username@remotehost:<files> /some/local/directory

To copy files from local host to remote host (if need be)

scp <file> username@remotehost:/some/remote/directory

For Windows: use WINscp or MobaXterm

MobaXterm:

1. Go to SFTP tab from the left pane.

2. Change directory to the location /var/support.

3. Select the support bundle. 

4. Click on the Download button.

WINscp:

1. Launch an SCP session to the SCVM's ip.

2. Click on the Parent Directory button to get to the root directory.

3. No navigate to /var/support.

4. Select the support bundle.

5. Click on the download button.