HX 4.0(2a) Onward DNS Monitoring and Alerting

Introduction

Starting in HyperFlex 4.0(2a), there is a new watchdog service that will monitor the resolveability of the ESXi and SCVM hostnames. The show dns command is used by the watchdog process to raise an alarm/event if HX cannot resolve the hostname or reach the DNS server. This document will cover the workaround for CSCvt13947 - One or more DNS servers not responding on HX Connect for DNS Alert from Health Monitoring.

Prerequisites

The prerequistie to hit this issue is Hyperflex Data Platform 4.0(2a).

Background Information

With this new framework, esxi hostname and scvm hostname must be in DNS in order to not have this event trigger.

{
      "message": "HX Controller VM {HOSTNAME} one or more configured DNS servers not responding",
      "type": "NODE",
      "name": "DnsServerOfflineEvent",
      "severity": "warning"
    },

There is an alarm around DNS as well:

    {
      "name": "HXA-NOD-0009",
      "description": "Triggered when one or more configured DNS servers on controller VM cannot be reached.",
      "category": "warning",
      "message": "One or more DNS servers configured on HX controller VM {HOSTNAME} not responding",
      "triggeringEvents" : ["DnsServerOfflineEvent"],
      "resetEvents" : ["DnsServerOnlineEvent"]
    }

Here is an example for the faults you will see in HX Connect:

Here is the corresponding show dns output:

root@SpringpathController3G4ZKOQ6SE:~# show dns
 +------------------------------------------+------------------+--------------+---------------------------+
| DNS Name                                  | Resolved Address |    status    |           error           |
+-------------------------------------------+------------------+--------------+---------------------------+
| HX01.rchs.local                           | None             | Not Resolved | No DNS servers configured |
| HX04.rchs.local                           | None             | Not Resolved | No DNS servers configured |
| HX03.rchs.local                           | None             | Not Resolved | No DNS servers configured |
| HX02.rchs.local                           | None             | Not Resolved | No DNS servers configured |
| SpringpathController3G4ZKOQ6SE.rchs.local | None             | Not Resolved | No DNS servers configured |
| SpringpathController5DCAL5X6C2.rchs.local | None             | Not Resolved | No DNS servers configured |
| SpringpathControllerWZ2X6H20SF.rchs.local | None             | Not Resolved | No DNS servers configured |
| SpringpathControllerGR57QZVDED.rchs.local | None             | Not Resolved | No DNS servers configured |
+-------------------------------------------+------------------+--------------+---------------------------+
 Name Servers: ['172.16.199.101'], Search Domains: - rchs.local

As you can see, the status for each is Not Resolved and the error is that No DNS servers configured. The DNS server in this output is 172.16.199.101.

If we perform a nslookup, we see that the hostname, SpringpathController3G4ZKOQ6SE, does not resolve.

root@SpringpathController5DCAL5X6C2:~# nslookup SpringpathController3G4ZKOQ6SE
Server:         172.16.199.101
Address:        172.16.199.101#53

** server can't find SpringpathController3G4ZKOQ6SE: SERVFAIL

Once the hostnames in the show dns command are added to DNS, show DNS will show the resolved address and the status will be Resolved:

root@SpringpathController3G4ZKOQ6SE:~# show dns
+-------------------------------------------+------------------+--------------+---------------------------+
| DNS Name                                  | Resolved Address |    status    |           error           |
+-------------------------------------------+------------------+--------------+---------------------------+
| HX01.rchs.local                           | 172.16.10.45     | Resolved     |  -                        |
| HX04.rchs.local                           | 172.16.10.48     | Resolved     |  -                        |
| HX03.rchs.local                           | 172.16.10.47     | Resolved     |  -                        |
| HX02.rchs.local                           | 172.16.10.46     | Resolved     |  -                        |
| SpringpathController3G4ZKOQ6SE.rchs.local | 172.16.10.41     | Resolved     |  -                        |
| SpringpathController5DCAL5X6C2.rchs.local | 172.16.10.44     | Resolved     |  -                        |
| SpringpathControllerWZ2X6H20SF.rchs.local | 172.16.10.43     | Resolved     |  -                        |
| SpringpathControllerGR57QZVDED.rchs.local | 172.16.10.42     | Resolved     |  -                        |
+-------------------------------------------+------------------+--------------+---------------------------+
 Name Servers: ['172.16.199.101'], Search Domains: - rchs.local

Workaround

The workaround is to disable the monitoring feature using the commands below.

root@hx-02-scvm-01:~# grep -i "monitor_dns_servers" /opt/springpath/hx-diag-tools/watchdog_config.json && sed -ie 's/"monitor_dns_servers": true/"monitor_dns_servers": false/' /opt/springpath/hx-diag-tools/watchdog_config.json && grep -i "monitor_dns_servers" /opt/springpath/hx-diag-tools/watchdog_config.json && restart watchdog
    "monitor_dns_servers": true,
    "monitor_dns_servers": false,
watchdog start/running, process 6350
root@hx-02-scvm-01:~#

This command will set "monitor_dns_servers" to false in /opt/springpath/hx-diag-tools/watchdog_config.json and restart the watchdog service.

To revert the change, run the following command on each storage controller VM:

root@hx-02-scvm-01:~# grep -i "monitor_dns_servers" /opt/springpath/hx-diag-tools/watchdog_config.json && sed -ie 's/"monitor_dns_servers": false/"monitor_dns_servers": true/' /opt/springpath/hx-diag-tools/watchdog_config.json && grep -i "monitor_dns_servers" /opt/springpath/hx-diag-tools/watchdog_config.json && restart watchdog
    "monitor_dns_servers": false,
    "monitor_dns_servers": true,
watchdog start/running, process 9473
root@hx-02-scvm-01:~#

In Hyperflex 4.0(2b), the feature will be diabled by default. The recommendation is for it to remain disabled until further notice.