Mapping Blacklisted or Failed Drive in HX to UCS

Introduction

This document walks through mapping a drive which is blacklisted in HX to the drive in UCS. This helps in troubleshooting issues, identifying blacklisted drive and PID of the drive in a Hyperflex environment.  We will need both the HX and the UCS logs for this process. Alternatively, you may also run the commands provided on a live system after SSH'ing into the device.

Confirming Drive in storfs-support Bundle

/var/log/springpath/diskslotmap-v2.txt
1.2.1:55cd2e414d9c5754:Intel:INTEL_SSDSC2BX016T4K:BTHC702104YY1P6PGN:G201CS01:SATA:SSD:1526185:Inactive:/dev/sdc
1.2.2:5000c50093bb784b:SEAGATE:ST1200MM0088:Z401A1Q00000C732VC38:N004:SAS:10500:1144641:Active:/dev/sdd
1.2.3:5000c50093bb79e3:SEAGATE:ST1200MM0088:Z401A1R50000C731NZPQ:N004:SAS:10500:1144641:Active:/dev/sde
1.2.4:5000c50093bb44fb:SEAGATE:ST1200MM0088:Z4019TBD0000C734EDN2:N004:SAS:10500:1144641:Active:/dev/sdf
1.2.5:5000c50098c02517:SEAGATE:ST1200MM0088:S402MYZ30000E711CNZS:N004:SAS:10500:1144641:Active:/dev/sdg
1.2.6:5000c50093aef283:SEAGATE:ST1200MM0088:Z4017Z8S0000C7332TP0:N004:SAS:10500:1144641:Active:/dev/sdh
1.2.7:5000c50093aed897:SEAGATE:ST1200MM0088:Z401756R0000C732SZXS:N004:SAS:10500:1144641:Active:/dev/sdi
1.2.8:5000c50093afdc97:SEAGATE:ST1200MM0088:Z40185SK0000C7332WWZ:N004:SAS:10500:1144641:Active:/dev/sdj
1.2.9:5000c50093affc0f:SEAGATE:ST1200MM0088:Z4016WGF0000C7323GJD:N004:SAS:10500:1144641:Active:/dev/sdk
1.2.10:5000c50093bb1133:SEAGATE:ST1200MM0088:Z4019WEB0000C734EGAF:N004:SAS:10500:1144641:Active:/dev/sdl
1.2.11:5000c50093bb6487:SEAGATE:ST1200MM0088:Z401A2FR0000C734HM49:N004:SAS:10500:1144641:Active:/dev/sdm
1.2.12:5000c50093bb6db7:SEAGATE:ST1200MM0088:Z401A22C0000C734HPDP:N004:SAS:10500:1144641:Active:/dev/sdn
1.2.13:5000c50093bb403f:SEAGATE:ST1200MM0088:Z4019TCV0000C734EF4S:N004:SAS:10500:1144641:Active:/dev/sdo
1.2.14:5000c50093bb6633:SEAGATE:ST1200MM0088:Z401A2C40000C734HQF5:N004:SAS:10500:1144641:Active:/dev/sdp
1.2.15:5000c50093bb4423:SEAGATE:ST1200MM0088:Z4019TBR0000C734EDLY:N004:SAS:10500:1144641:Active:/dev/sdq
1.2.16:5000c50093bb75ff:SEAGATE:ST1200MM0088:Z401A1SC0000C734HMBL:N004:SAS:10500:1144641:Active:/dev/sdr
1.2.17:5000c50093a66f67:SEAGATE:ST1200MM0088:Z4016C2Y0000C7324EPZ:N004:SAS:10500:1144641:Active:/dev/sds
1.2.18:5000c50093a67813:SEAGATE:ST1200MM0088:Z4016RC20000C7324GS4:N004:SAS:10500:1144641:Active:/dev/sdt
1.2.19:5000c50093a695db:SEAGATE:ST1200MM0088:Z4016PWY0000C732A8DR:N004:SAS:10500:1144641:Active:/dev/sdu
1.2.20:5000c50093a675b7:SEAGATE:ST1200MM0088:Z4016RP30000C7323J1C:N004:SAS:10500:1144641:Active:/dev/sdv
1.2.21:5000c50093a662c7:SEAGATE:ST1200MM0088:Z4016BME0000C727L0BG:N004:SAS:10500:1144641:Active:/dev/sdw
1.2.22:5000c50093a68ac7:SEAGATE:ST1200MM0088:Z4016QHP0000C732ADRB:N004:SAS:10500:1144641:Active:/dev/sdx
1.2.23:5000c50093a66597:SEAGATE:ST1200MM0088:Z4016BGP0000C7324JEL:N004:SAS:10500:1144641:Active:/dev/sdy
1.2.24:5000c50093a686eb:SEAGATE:ST1200MM0088:Z4016BA50000C7323HYD:N004:SAS:10500:1144641:Active:/dev/sdz


/cmds_output/stcli_node_list.txt 
...
----------------------------------------
        blacklistCount: 1
        medium: solidstate
        capacity: 1.3T
        state: blacklisted                <<<<<<
        version: 0
        entityRef:
            type: disk
            id: 55cd2e414d9c5754:0000000000000000
        usage: caching
        lastModifiedTime: 1539801326000
        usedCapacity: 164.0M
----------------------------------------

Step 1. In the output of /var/log/springpath/diskslotmap-v2.txt above, verify a disk is "Inactive".  Note the disk slot, id of the disk, Vendor Model, and Serial Number of the disk.

Step 2. In the output of /cmds_output/stcli_node_list.txt confirm the drive is blacklisted and it matches the id we got in Step 1 above.

Validate info from HX system

You will need to run these commands on the live system, and then follow the corresponding HX steps above

cat /var/log/springpath/diskslotmap-v2.txt
stcli node list

Mapping Drive to Host 

Step 1. The first step will be to get the disk identifier that's failed from HX Connect. 

Step 2. Use command cat stevents.log and grep for the disk identifier to get the node ID.

Step 3. Run "stcli cluster info | less" to match the node identifier to the IP address of the host.

cat /var/log/springpath/stevents.logs | grep 55cd2e414d9c5754
2016-06-13 22:22:55,657 INFO Event Posted Successfully: DiskFailedEvent, Disk 55cd2e414d9c5754:0000000000000000 on node 1276a402564d0cb9:995b4d5ec32beabc failed, 1465856569490


Stcli cluster info | less
stNodes:
    ----------------------------------------
    type: node
    id: 5a2595a9-1678-9343-9351-e854cc98d027
    name: 172.X.X.193
    ----------------------------------------
    type: node                                  <<<<<<
    id: 1276a402564d0cb9:995b4d5ec32beabc
    name: 172.X.X.194
    ----------------------------------------
    type: node
    id: ba8f98a6-09da-2440-9609-50d91a241c86
    name: 172.X.X.192
    ----------------------------------------
    type: node
    id: be108c11-3584-0b49-94d2-18ca9e6543da
    name: 172.X.X.195
    ----------------------------------------

Mapping Drive Serial Number to UCS via UCSM sam_techsupportinfo

`show server inventory expand`
Server 1:
...
Local Disk 1:
                Product Name: 1.6TB 2.5 inch Enterprise performance 6G SATA SSD (3X  endurance)
                PID: UCS-SD16TB12S3-EP
                VID: V01
                Vendor: ATA
                Model: INTEL SSDSC2BX016T4K    <<<<<<
                Vendor Description: Intel
                Serial: BTHC652200H01P6PGN     <<<<<<
                HW Rev: 0
                Block Size: 512
                Blocks: 3125626880
                Operability: Operable
                Oper Qualifier Reason: N/A
                Presence: Equipped
                Size: 1526185
                Drive State: Unconfigured Good
                Power State: Active
                Link Speed: 6 Gbps
                Device Version: CS01
                Device Type: SSD
                Thermal: N/A

In sam_techsupportinfo, search for the Serial Number (from Step 1 above). From there look for the drive whch is failed and get the Cisco PID from it. If you want to see which server it is installed in from the UCS side, you will need to scroll up in the logs until you reach the Server number and information. You can also get the rest of the server configuration from this output.

Validate information from UCS system

You will need to check either via the UCSM GUI, or SSH into the UCSM IP address and run the below commands, then follow the corresponding UCS steps above.

To check via GUI

Navigate to UCSM > Server X > Inventory > Storage > Disks > Expand All > Serial Column

To check via CLI

To display the inventory of all servers

HX-UCSM-B# show server inventory storage detail

Or, if you know which server you want to expand the inventory on

HX-UCSM-B# scope server 1
HX-UCSM-B /server # show inventory storage detail