High swap memory usage on Splunk servers.

Applicability

Splunk Enterprise

Description

1. What is Swap Space?
Linux divides its physical RAM (random access memory) into chunks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space are the amount of virtual memory available.

Swapping is necessary for two important reasons.
1) First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately.

2) Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

 a. Types of swap space.
Linux has two forms of swap space: the swap partition and the swap file. The swap partition is an independent section of the hard disk used solely for swapping; no other files can reside there. The swap file is a special file in the filesystem that resides amongst your system and data files.

b. To see what swap space you have to use the command swapon -s.

The output will look something like this:

Filename  Type       Size       Used Priority

/dev/sda5 partition  859436  0       -1

Each line lists a separate swap space being used by the system. Here, the ‘Type’ field indicates that this swap space is a partition rather than a file, and from ‘Filename’ can see that it is on the disk sda5. The ‘Size’ is listed in kilobytes, and the ‘Used’ field tells us how many kilobytes of swap space has been used (in this case none). ‘Priority’ tells Linux which swap space to use first. One great thing about the Linux swapping subsystem is that if you mount two (or more) swap spaces (preferably on two different devices) with the same priority, Linux will interleave its swapping activity between them, which can greatly increase swapping performance.

2. How big should my swap space be?

It is possible to run a Linux system without a swap space, and the system will run well if you have a large amount of memory — but if you run out of physical memory then the system will crash, as it has nothing else it can do, so it is advisable to have a swap space

 Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory.

Modern implementations (such as Linux) don’t require that much, but they can use it if you configure it.

A rule of thumb is as follows:

a) For a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications;

b) For a server, have a smaller amount of swap available (say half of the physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary;

c) For older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.


3. The default value for swappiness is 60. 

 /proc/sys/vm/swappiness

 a. Setting it to 0 means never using swap when there is still RAM left and 100 is swapping out memory as soon as possible.

 b. To change the value temporarily (lost on reboot):

sudo sysctl vm.swappiness=10

c. To change the value permanently, edit the file:

/etc/sysctl.conf

as root (e.g. sudoedit /etc/sysctl.conf) and change or add (if not there) the line:

vm.swappiness

to the desired value (vm.swappiness=10). If this file does not exist (e.g. in Arch Linux), then try /etc/sysctl.d/99-sysctl.conf instead.





 

Investigation

1. Check memory usage using the free command.

2. Check CPU and current memory usage using the top command.

3. Checked the swappiness value.
By below command.

cat /proc/sys/vm/swappiness

 

Resolution

1. It is normal for Linux systems to use some swap even if there is still RAM free. The Linux kernel will move to swap memory pages that are very seldom used.

2.Swap space usage becomes an issue only when there is not enough RAM available, and the kernel is forced to continuously move memory pages to swap and back to RAM, just to keep applications running. In this case, system monitor applications would show a lot of disk I/O activity.

3. Linux starts swapping before the RAM is filled up. This is done to improve performance and responsiveness:

• Performance is increased because sometimes RAM is better used for disk cache than to store program memory. So it's better to swap out a program that's been inactive for a while, and instead keep often-used files in cache.

• Responsiveness is improved by swapping pages out when the system is idle, rather than when the memory is full and some program is running and requesting more RAM to complete a task.

4. Swapping does slow the system down, of course — but the alternative to swapping isn't not swapping, it's having more RAM or using less RAM.

5. Using swap space instead of RAM can drastically slow down a PC. 

6.Splunk application has absolutely no control over what type of memory it uses. It is all controlled by the OS(Or hypervisor if Virtualised) 

7. Linux Swap usage is up to the user and their Environment.

Related Links

Linux swap space.
Usage of swap.

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release