Troubleshoot Supervisor.sock Refused Connection Error

Introduction

This document describes how to troubleshoot and fix an issue on the supervisor in the Cisco WAN Automation Engine (WAE).

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Problem

On WAE 7.3, this is the issue:

[wae@ds1waenod01 etc]$ sudo systemctl status supervisord
● supervisord.service - Process Monitoring and Control Daemon
Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-12-07 16:52:30 PST; 1 months 14 days ago
Process: 23468 ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf (code=exited, status=1/FAILURE)

Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: d.main()
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: 
File "/usr/lib/python2.7/site-packages/supervisor/supervisord.py", line 70, in main 
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: rlimit_messages = self.options.set_rlimits()
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: 
File "/usr/lib/python2.7/site-packages/supervisor/options.py", 
 line 1316, in set_rlimits
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: self.usage(msg % locals())
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: KeyError: 'min' 
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: supervisord.service: control process exited, 
 code=exited status=1
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: Failed to start Process Monitoring and Control Daemon.
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: Unit supervisord.service entered failed state.
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: supervisord.service failed.

Due to this, supervisor is rendered inactive. Therefore, it is not up:

[wae@ds1waenod01 root]$ supervisorctl status

unix:///opt/supervisor/run/supervisor.sock refused connection

The effort to restart the WAE process and the server reboot are not successful.

Troubleshoot

In order to triage this further, capture these details:

1. WAE Version
2. OS version: cat /etc/redhat-release
3. Supervisor version: rpm -qa supervisor
4. File /etc/security/limits.conf
   

Root Cause

This is a known issue observed in supervisor version 3.2 or lower. The WAE version 7.3 can have a lower supervisor version. For details, see Supervisor Issues.

Solution

The solution is to upgrade the WAE version and if that is not possible, then upgrade the supervisor. However, if an upgrade is not an option, use this workaround:

Comment these parameters in /etc/supervisord.conf and then restart supervisor/wae.

;minfds=1000000              ; (min. avail startup file descriptors;default 1024)
;minprocs=257805             ; (min. avail process descriptors;default 200)

systemctl restart supervisord.service

supervisorctl restart wae:*

Expected Outcome

WAE service starts as expected:

supervisorctl status

wae:kafka                        RUNNING   pid 11726, uptime 0:01:06
wae:logrotate                    RUNNING   pid 11648, uptime 0:01:07
wae:wae-monitor                  RUNNING   pid 11667, uptime 0:01:07 
wae:waectl                       RUNNING   pid 11634, uptime 0:01:07

wae:zookeeper                    RUNNING   pid 11633, uptime 0:01:07

Related Information

• Technical Support & Documentation - Cisco Systems