Splunk SOAR ibackup Fails with "No such file or directory" error on backup staging directory

Applicability

Description

When initiating a backup using ibackup, the process fails immediately during the local staging phase before any data is transferred to the backup destination.

The failure is observed in the ibackup log with the following error:

rsync: recv_generator: mkdir "/opt/phantom/data/ibackup.tmp/ibackup" failed: No such file or directory (2)
*** Skipping any contents from this failed directory ***

Additionally, any manual attempt to clean up the staging directory returns the following:

rm: cannot remove '/opt/phantom/data/ibackup.tmp': Device or resource busy

Investigation

The "No such file or directory" error occurs because the ibackup process relies on /opt/phantom/data/ibackup.tmp as a clean, writable staging directory at the start of every backup run. When a previous backup terminates unexpectedly, the staging directory is left behind as a stale mount point. preventing ibackup from writing into it or removing it, causing all subsequent backup attempts to fail immediately.

Navigate to the ibackup log directory and check the latest log file for the staging failure:

tail -f /opt/phantom/var/log/phantom/ibackup/ibackup_<date>.log

Confirm the presence of the following error indicating the staging directory could not be created:

mkdir "/opt/phantom/data/ibackup.tmp/ibackup" failed: No such file or directory (2)

Resolution

The ibackup script expects to create and write to /opt/phantom/data/ibackup.tmp at the start of every backup run. If the directory cannot be created, the backup fails immediately. Follow the steps below to identify and resolve the root cause before re-running the backup.

Pre-Checks :

• Verify phantom:phantom ownership and permissions on the backup directory.
• If ownership is incorrect, restore it and proceed to initiate backup.
  

  chown -R phantom:phantom /opt/phantom/data/

Step 1: Check for a stale staging directory from a previous run :

Check whether /opt/phantom/data/ibackup.tmp is present and attempt to remove it:

rm -rf /opt/phantom/data/ibackup.tmp

• If the directory is removed successfully, proceed directly to Step 3

• If the removal returns Device or resource busy, the directory remained as a stale mount point left over from a previous backup execution, proceed to Step 2

Step 2: Unmount the stale staging directory

Confirm the directory is registered as an active mount point:

mount | grep ibackup.tmp

If a mount entry is returned, check for any processes holding the mount open and clear the processes.

fuser -mv /opt/phantom/data/ibackup.tmp

Once no active processes are found, unmount the directory:

umount -f /opt/phantom/data/ibackup.tmp

Remove the residual directory post clearing the mount status and the hung process:

rm -rf /opt/phantom/data/ibackup.tmp

Step 3: Re-run the ibackup script

Initiate and Execute the backup command & the script will automatically create /opt/phantom/data/ibackup.tmp:

Validation

Monitor the log to confirm the backup completes without error: 

tail -f /opt/phantom/var/log/phantom/ibackup/ibackup_<date>.log

Related Links

Back up a Splunk SOAR (On-premises) deployment