High Swap Usage or Splunk processes getting OOM killed after WLM configuration

Applicability

Description

The WLM configuration allows administrators to set limits on the CPU and memory resources for different functions within Splunk.  The Splunk WLM documentation indicates that on systemd managed systems, cgroups are configured.  These cgroups are used by systemd to enforce the limits specified in the WLM configuration.

RedHat documentation shows several commands related to cgroups that are beneficial in understanding the cgroup hierarchy and which processes are running in which slide.  

Using the systemd-cgls command on a test installation without Splunk boot-start enabled shows the Splunk processes running under the user slice:

[root@idx1 ~]# systemd-cgls
Control group /:
-.slice
├─user.slice
│ ├─user-2022.slice
│ │ ├─user@2022.service
│ │ │ └─init.scope
│ │ │   ├─5422 /usr/lib/systemd/systemd --user
│ │ │   └─5425 (sd-pam)
│ │ └─session-7.scope
│ │   ├─8337 splunkd -p 8089 restart
│ │   ├─8341 [splunkd pid=8337] splunkd -p 8089 restart [process-runner]
│ │   ├─8705 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --wiredTigerCacheSizeGB=0.450000 --port=8191 --timeStampFormat=iso8601-utc --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --setParameter=o>
│ │   ├─8767 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
│ │   ├─8858 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
│ │   ├─9054 [splunkd pid=8337] [search-launcher]
│ │   └─9055 [splunkd pid=8337] [search-launcher] [process-runner]
│ └─user-2023.slice
│   ├─user@2023.service
│   │ └─init.scope
│   │   ├─10515 /usr/lib/systemd/systemd --user
│   │   └─10519 (sd-pam)
│   └─session-8.scope
│     ├─10477 sshd: splunker [priv]
│     ├─10525 sshd: splunker@pts/0
│     ├─10526 -bash
│     ├─10602 sudo su -
│     ├─10604 su -
│     ├─10605 -bash
│     ├─10755 systemd-cgls
│     └─10756 systemd-cgls
...
output truncated for brevity

After configuring boot start with systemd, the systemd-cgls output shows Splunkd.service running under the system slice/cgroup.

[root@idx1 ~]# systemd-cgls
Control group /:
-.slice
├─user.slice
│ └─user-2023.slice
│   ├─user@2023.service
│   │ └─init.scope
│   │   ├─10515 /usr/lib/systemd/systemd --user
│   │   └─10519 (sd-pam)
│   └─session-8.scope
│     ├─10477 sshd: splunker [priv]
│     ├─10525 sshd: splunker@pts/0
│     ├─10526 -bash
│     ├─10602 sudo su -
│     ├─10604 su -
│     ├─10605 -bash
│     ├─23185 systemd-cgls
│     └─23186 (pager)
├─init.scope
│ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
└─system.slice
  ├─rngd.service
  │ └─798 /usr/sbin/rngd -f --fill-watermark=0
  ├─irqbalance.service
  │ └─769 /usr/sbin/irqbalance --foreground
  ├─amazon-ssm-agent.service
  │ ├─5196 /usr/bin/amazon-ssm-agent
  │ └─5264 /usr/bin/ssm-agent-worker
  ├─nessusagent.service
  │ ├─  847 /opt/nessus_agent/sbin/nessus-service -q
  │ ├─11067 nessusd -q
  │ └─11068 /opt/nessus_agent/sbin/nessus-agent-module -q
  ├─systemd-udevd.service
  │ └─680 /usr/lib/systemd/systemd-udevd
  ├─system-serial\x2dgetty.slice
  │ └─serial-getty@ttyS0.service
  │   └─1046 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
  ├─polkit.service
  │ └─767 /usr/lib/polkit-1/polkitd --no-debug
  ├─chronyd.service
  │ └─784 /usr/sbin/chronyd
  ├─auditd.service
  │ ├─704 /sbin/auditd
  │ └─712 /usr/sbin/sedispatch
  ├─falcon-sensor.service
  │ ├─695 /opt/CrowdStrike/falcond
  │ └─696 falcon-sensor
  ├─tuned.service
  │ └─849 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
  ├─Splunkd.service
  │ ├─23024 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
  │ ├─23069 [splunkd pid=23024] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
  │ └─23184 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_metrics_modular_input.py --scheme
  ├─systemd-journald.service
  │ └─650 /usr/lib/systemd/systemd-journald
...
output truncated for brevity

After configuring WLM pools for default_search, default_ingest, and default_misc, you can see the different cgroups with either the splunk show workload-management-status --verbose command or navigating to /sys/fs/cgroups/memory/system.slice/Splunkd.service and running the systemd-cgls command.  Below are the outputs for both commands:

[splunk@idx1 bin]$ ./splunk show workload-management-status --verbose
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.	Admission Control Status:
		Enabled: 0	Workload Management Status:
		Enabled: 1
		Supported: 1
		Error: 	Workload Category: ingest			CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/ingest
			Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/ingest
			CPU Weight: 20
			Memory Weight: 100
			Default Category Pool: default_ingest
		default_ingest:
			CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/ingest/default_ingest
			Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/ingest/default_ingest
			CPU Weight: 100
			Memory Weight: 100	Workload Category: search			CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/search
			Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/search
			CPU Weight: 70
			Memory Weight: 70
			Default Category Pool: default_search
		default_search:
			CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/search/default_search
			Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/search/default_search
			CPU Weight: 40
			Memory Weight: 40	Workload Category: misc			CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/misc
			Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/misc
			CPU Weight: 10
			Memory Weight: 10
			Default Category Pool: ############ Switched to root user ################
[root@idx1 Splunkd.service]# systemd-cgls
Working directory /sys/fs/cgroup/memory/system.slice/Splunkd.service:
├─25658 [splunkd pid=25059] [search-launcher]
├─25659 [splunkd pid=25059] [search-launcher] [process-runner]
└─ingest
  └─default_ingest
    ├─25059 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
    ├─25106 [splunkd pid=25059] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
    ├─25317 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --wiredTigerCacheSizeGB=0.450000 --port=8191 --timeStampFormat=iso8601-utc --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --setParameter=op>
    ├─25368 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
    └─25480 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
[root@idx1 Splunkd.service]# 
[root@idx1 Splunkd.service]# 
[root@idx1 Splunkd.service]# 
[root@idx1 Splunkd.service]# 
[root@idx1 Splunkd.service]# systemd-cgls
Working directory /sys/fs/cgroup/memory/system.slice/Splunkd.service:
├─25658 [splunkd pid=25059] [search-launcher]
├─25659 [splunkd pid=25059] [search-launcher] [process-runner]
├─search
│ └─default_search
│   ├─27173 [splunkd pid=25059] [search-launcher]
│   ├─27174 [splunkd pid=25059] [search-launcher] [process-runner]
│   ├─27200 [splunkd pid=25059] [search-launcher]
│   └─27201 [splunkd pid=25059] [search-launcher] [process-runner]
└─ingest
  └─default_ingest
    ├─25059 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
    ├─25106 [splunkd pid=25059] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
    ├─25317 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --wiredTigerCacheSizeGB=0.450000 --port=8191 --timeStampFormat=iso8601-utc --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --setParameter=op>
    ├─25368 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
    ├─25480 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
    ├─27281 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/supervisor_modular_input.py
    ├─27282 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py
    ├─27283 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py
    └─27288 /opt/splunk/bin/splunkd btool server list
 

Investigation

With the WLM rules in place and the cgroups configured as desired, the search processes may be OOM killed and/or swap usage may increase.  Using the free -m command, the swap usage can be monitored.  In the example below, there is no swap, however the used column will show swap usage and typically high swap usage leads to performance impacts.

[root@idx1 Splunkd.service]# free -m
              total        used        free      shared  buff/cache   available
Mem:           3676         688         332          16        2655        2702
Swap:             0           0           0

Once swap usage reaches the full size of swap or if no swap is available, there should be OOM messages in /var/log/messages on RedHat distributions or /var/log/syslog on Ubuntu distributions.

[root@idx1 Splunkd.service]# grep -i oom /var/log/messages
Jul  9 21:07:26 ip-10-202-35-233 kernel: RunDispatch invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=700
Jul  9 21:07:26 ip-10-202-35-233 kernel: oom_kill_process.cold.32+0xb/0x10
Jul  9 21:07:26 ip-10-202-35-233 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul  9 21:07:26 ip-10-202-35-233 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/Splunkd.service/search/default_search,task_memcg=/system.slice/Splunkd.service/search/default_search,task=splunkd,pid=27200,uid=2021
Jul  9 21:07:26 ip-10-202-35-233 kernel: Memory cgroup out of memory: Killed process 27200 (splunkd) total-vm:234628kB, anon-rss:24056kB, file-rss:29280kB, shmem-rss:0kB, UID:2021 pgtables:336kB oom_score_adj:700
Jul  9 21:07:26 ip-10-202-35-233 kernel: oom_reaper: reaped process 27200 (splunkd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

These messages indicate the process was killed because of the default_search configuration from WLM - oom_memcg=/system.slice/Splunkd.service/search/default_search.

Resolution

Modify the memory limit for the correct WLM pool that is causing the OOM/swapping to occur.  Follow the instructions in WLM Documentation to modify (increase) the memory limit values from the Web GUI or from the command line to prevent OOM/swapping.