SOAR - ibackup restore failed

Applicability

Description

The restore failed in the log:

[2025-11-28 13:56:43.318] - DEBUG    - Command: /opt/phantom/www/phantom_ui/phadmin/scripts/phmanage.pyc ibackup --ignore-env-check --restore /opt/phantom/data/backup/phantom_backup_group_0_level_0-<date and time>.tar
[2025-11-28 13:56:43.318] - DEBUG    - Initializing BackupManager
[2025-11-28 13:56:43.318] - DEBUG    - Running SQL query "SELECT 1 FROM pg_database WHERE datname='rdsadmin'" with connection {'db': 'postgres'}
[2025-11-28 13:56:43.318] - DEBUG    - SELECT 1 FROM pg_database WHERE datname='rdsadmin'
[2025-11-28 13:56:43.318] - DEBUG    - Running on a normal postgres instance
[2025-11-28 13:56:43.319] - ERROR    - An unexpected error occurred
Traceback (most recent call last):
  File "/opt/phantom/usr/python39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/phantom/usr/python39/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/phantom/usr/python39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "www/phantom_ui/db/common/base.py/base.py", line 82, in get_new_connection
  File "www/phantom_ui/db/direct/base.py/base.py", line 13, in establish_connection
  File "/opt/phantom/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?

Investigation

The postgresql start failed and the following error can be seen in postgresql-<date>.log under <$PHANTOM_HOME>/data/db/pg_log folder:

ERROR: [045]: duplicates found for WAL segment 00000008000001BD0000007D:
    repo1: 15-1/00000008000001BD/00000008000001BD0000007D-1e119181f79e5fa9d729627edcc90ee96f97264d.gz, 15-1/00000008000001BD/00000008000001BD0000007D-89d0475d404385ec554feef6f91e982fc9a5ec5b.gz
    HINT: are multiple primaries archiving to this stanza?
< 2025-12-03 07:23:30.635 PST >LOG: invalid checkpoint record
< 2025-12-03 07:23:30.635 PST >FATAL: could not locate required checkpoint record
< 2025-12-03 07:23:30.635 PST >HINT: If you are restoring from a backup, touch "/opt/phantom/data/db/recovery.signal" and add required recovery options.
    If you are not restoring from a backup, try removing the file "/opt/soar/data/db/backup_label".
    Be careful: removing "/opt/phantom/data/db/backup_label" will result in a corrupt cluster if restoring from a backup.
< 2025-12-03 07:23:30.635 PST >LOG: startup process (PID 1817717) exited with exit code 1
< 2025-12-03 07:23:30.635 PST >LOG: aborting startup due to startup process failure
< 2025-12-03 07:23:30.636 PST >LOG: database system is shut down

The error indicates a critical issue with PostgreSQL Write-Ahead Log (WAL) archiving and recovery process, likely caused by having multiple primary servers archiving to the same pgBackRest repository stanza. In this case, the different contents with the same WAL segment name are present and the DB failed to start as the recovery process not being able to find a consistent set of WAL files in the archive to start up from.

Resolution

There is no need to save any data in the secondary SOAR server. Do a reinstallation of SOAR to set up a clean DB.

• The postgresql service cannot start, running ./soar-uninstall script would fail. Remove all files and subfolders in <$PHANTOM_HOME>/ folder including .soar file.
• Remove splunk-soar folder, extract the SOAR software package tgz file and it creates a new splunk-soar folder.
• Install SOAR running soar-install script by referring to the document Install Splunk SOAR (On-premises) as an unprivileged user | Splunk Docs 
• SOAR installation is completed and verify system is up and running. 
• Restore the backup file running the commands and referring to Restore Splunk SOAR (On-premises) from a backup | Splunk Docs 

$<$PHANTOM_HOME>/bin/phenv ibackup --setup
$<$PHANTOM_HOME>/bin/phenv ibackup --ignore-env-check --restore <path/to/phantom_<number>_backup.tar>

• Disable ibackup to stop the pgbackrest process.

$<$PHANTOM_HOME>/bin/phenv ibackup --off