SOAR - iBackup setup fails with error "WAL segment was not archived before the 60000ms timeout"

Applicability

Description

When running "<PHANTOM_HOME>/bin/phenv ibackup --setup" in Splunk SOAR (On-Premises), users may encounter the following error related to WAL archiving:

Command '['pgbackrest', '--stanza=phantom', '--config=/opt/phantom/etc/pgbackrest.conf', '--log-level-console=info', '--log-level-file=info', 'check']' returned non-zero exit status 82.
Output: 2023-07-25 21:07:43.772 P00   INFO: check command begin 2.28: --config=/opt/phantom/etc/pgbackrest.conf --log-level-console=info --log-level-file=info --log-path=/opt/phantom/var/log/phantom/backup --pg1-path=/opt/phantom/data/db --pg1-socket-path=/tmp --repo1-path=/opt/phantom/data/ibackup/repo/pg --stanza=phantom
ERROR: [082]: WAL segment 00000004000001620000003D was not archived before the 60000ms timeout
       HINT: check the archive_command to ensure that all options are correct (especially --stanza).
       HINT: check the PostgreSQL server log for errors.
2023-07-25 21:08:44.703 P00   INFO: check command end: aborted with exception [082]

This issue indicates that pgBackRest timed out while waiting for PostgreSQL to archive a WAL segment. It may occur due to slow disk I/O, delayed archive operations, a temporary system bottleneck or missing backup file.

This article provides two potential solutions:

1. Increasing the pgBackRest timeout (recommended first step)

2. Disabling the WAL archiving requirement during setup (fallback option)
3. Turning off ibackup before running ibackup --delete option due to missing backup file

Investigation

By default, pgBackRest expects WAL segments to be archived within 60 seconds (archive-timeout = 60). If the archive operation is slow or temporarily blocked, the ibackup setup may fail with the above timeout error. In most SOAR deployments, this timeout is sufficient. However, if the system is under high load or the disk I/O is slow, the archive process may take longer than 60 seconds.

There could be accumulated large amounts of Postgresql WAL files under <PHANTOM_HOME>/data/db/pg_wal folders. And some of WAL files were not being archived correctly. You can verify the issue by reviewing:

• The pgbackrest logs under <PHANTOM_HOME>/var/log/phantom/backup/

• The PostgreSQL log under <PHANTOM_HOME>/data/db/pg_log folder to confirm whether WAL files are being archived successfully or the detailed error why archive command failed

• The size and content of the pg_wal directory (<PHANTOM_HOME>/data/db/pg_wal/) to determine WAL backlog

Resolution

Option 1: Increase archive-timeout (Preferred Solution)

Update the pgBackRest configuration file to increase the archive-timeout value:

1. Open the config file:

   vi <PHANTOM_HOME>/etc/pgbackrest.conf

2. Locate the [phantom] stanza (or create one if not present) and add or modify the following:

   [phantom] 
   pg1-path=/opt/phantom/data/db 
   archive-timeout=120

3. Save the file and retry:

   <PHANTOM_HOME>/bin/phenv ibackup --setup

Option 2: Disable WAL archiving during setup (Fallback/Workaround)

If increasing the timeout does not resolve the issue, you can bypass WAL archiving checks by temporarily disabling the pgBackRest-related configuration.

1. Open the PostgreSQL configuration:

   vi <PHANTOM_HOME>/data/db/postgresql.conf

2. Comment out the following line:

   #include_if_exists 'postgresql.ibackup.conf'

3. Restart PostgreSQL:

   <PHANTOM_HOME>/bin/phsvc restart postgresql-11"

4. Depending on the number of WAL files, it might take 10 minutes for Postgresql to clean up over 10,000 files under <PHANTOM_HOME>/data/db/pg_wal folder.

5. Once the WAL files are cleaned up, you can re-run ibackup setup again.

Option 3: Turning off ibackup before deleting any backup file if archive failed due to missing backup file

Please refer to this link for ibackup tools Splunk SOAR (On-premises) backup tools | Splunk Docs 

      1. Turned ibackup off: `ibackup --off`

      2. Deleted all backups: `ibackup --delete-all`

      3. Restarted Postgres: `phsvc restart postgresql`

      4. You can re-run ibackup setup again