Universal Forwarder failing to install on Windows Servers - Fresh Install/Upgrades

Applicability

Description

When attempting to install or upgrade the Splunk Universal Forwarder on Windows (Server 2012 R2, 2016, 2019, 2022, Windows 10/11), the process may fail or roll back.

Common symptoms include:

• Setup Wizard ended prematurely or MSI exit code 1603.

• Installer hangs or rolls back automatically.

• SSL certificate generation failed.

• Error “A newer version of Universal Forwarder is already installed” even though no UF is present.

• UF listed in services.msc but cannot be started/uninstalled, and files missing.

• On Windows Server 2016, installation hangs until reboot.

Investigation

To properly debug or discard what is the exact reason the installation/upgrade process is rolling back msiexec.log, splunk.log & PML logs must be gathered for further analysis (while running Windows Installer):
 

1. Collect Process Monitor Logs (PML).

a. Launch PowerShell or Command Prompt and change to a directory OR a drive with plenty of free space available (Procmon logs can easily be many gibibytes in size).

b. Download Procmon.exe to that directory (Sysinternals Live: https://live.sysinternals.com/Procmon.exe).

c. To collect PML logs run the following:

.\Procmon.exe /BackingFile Logfile.PML /NoFilter /Quiet.

d. Then, reproduce the issue being investigated (running installer while generating msiexec logs (msiexec /l*vx msiexec.log /i <splunk.msi>).

e. To stop PML collection:

.\Procmon.exe /Terminate.

Notes: 
Double-check that Logfile.PML can be opened successfully (i.e., no error message is displayed):

.\Procmon.exe /OpenLog Logfile.PML.

Zip the PML(s) (Procmon may have generated additional files; e.g.: LogFile-1.PML, LogFile-2.PML, etc.

2. Run installer while generating msiexec logs:
This was referenced above in step d; the logic is to first enable PML collection to then collect msiexec logs while trying to install Splunk.

msiexec /l*vx msiexec.log /i <splunk.msi>

This command will run Windows Installer pointing to <splunk.msi> while generating the msiexec.log file.

3. Collect the splunk.log in the installing user’s %TEMP% directory.

The precise location may be determined from the msiexec.log file—search for >> to find an example—e.g.:

StopSplunkService:  Info: Execute string: C:\Windows\system32\cmd.exe /c ""D:\Splunk\\bin\splunk.exe" stop --answer-yes --no-prompt --accept-license >> "C:\Users\<USER>\AppData\Local\Temp\splunk.log" 2>&1"

Note: Use below Decision Tree for Root Cause Investigation.

Start
 │
 ├──► Error: "Setup Wizard ended prematurely" / MSI Exit Code 1603
 │       │
 │       ├── Run: sfc /scannow
 │       ├── Re-register msiexec
 │       └── Retry installation
 │
 ├──► Error: "SSL certificate generation failed"
 │       │
 │       ├── Remove conflicting OPENSSL_CONF or PATH variables
 │       ├── Set ComSpec = C:\Windows\System32\cmd.exe
 │       └── Retry installation
 │
 ├──► Error: "A newer version of Universal Forwarder is already installed"
 │       │
 │       ├── Delete leftover UF folder (C:\Program Files\SplunkUniversalForwarder)
 │       ├── Clean registry keys under HKCR/HKLM Installer\Products
 │       └── Retry installation
 │
 ├──► Installer hangs (esp. Windows Server 2016)
 │       │
 │       ├── Disable RSC:
 │       │     netsh int tcp set global rsc=disabled
 │       └── Reboot & retry
 │
 ├──► Installer rolls back immediately
 │       │
 │       ├── Delete Autorun registry entries:
 │       │     HKCU\Software\Microsoft\Command Processor\AutoRun
 │       │     HKLM\Software\Microsoft\Command Processor\AutoRun
 │       ├── Verify ComSpec = cmd.exe
 │       └── Retry installation
 │
 ├──► Specific UF 9.1.3 / 9.2.0 rollback
 │       │
 │       └── Install with flag:
 │           msiexec /i splunkforwarder-<ver>.msi USE_LOCAL_SYSTEM=1
 │
 ├──► UF stuck in broken state
 │       │
 │       ├── Stop and delete service:
 │       │     sc stop SplunkUniversalForwarder
 │       │     sc delete SplunkUniversalForwarder
 │       ├── Remove UF folder
 │       ├── Clean registry Installer keys
 │       └── Reboot → Install fresh MSI
 │
 └──► Still failing
         │
         ├── Gather logs (msiexec, splunk.log, Procmon)
         └── Escalate to Splunk Support

Resolution

1. Fix Windows Installer.
-Open command prompt with admin rights.
-Run: (consider this may take a few minutes)

sfc /scannow
OR
sfc /verifyonly -to check integrity of all protected system files without performing any repairs (faster)

-Then, run:

%windir%\system32\msiexec.exe /unregister
%windir%\syswow64\msiexec.exe /unregister
%windir%\system32\msiexec.exe /regserver
%windir%\syswow64\msiexec.exe /regserver

2. Delete all directories from failed UF installations.
-%SPLUNK_HOME% or tipically C:\Program Files\SplunkUniversalForwarder\

3. Delete registry keys that may be causing issues.
-Make a backup of the Registry Key HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products

-Then execute the following PowerShell:

Get-ChildItem Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products | Where-Object { $_.Name -notmatch '\\[0-9A-F]{32}$' } | Select-Object Name

-Delete each key that it lists (basically any key that is not exactly 32 hexadecimal characters).
-OR check the MSI log for an error reporting "Cannot create ftr" OR "Custom Action" GetPreviousSettingsCA" failing with Return Value 3.
-Delete the key located from that error in the logs.

4. Verify that these system variables point to the following Splunk resources & ComSpec to cmd.exe.

CA_HOME -> C:\Program Files\SplunkUniversalForwarder\etc\auth
OPENSSL_CONF -> C:\Program Files\SplunkUniversalForwarder\openssl.cnf
ComSpec -> C:\Windows\System32\cmd.exe

Notes:

To create or modify environment variables from the UI:
a. On the Windows taskbar, right-click the Windows icon and select System.
b. In the Settings window, under Related Settings, click Advanced system settings.
c. On the Advanced tab, click Environment Variables.
d. Click New to create a new environment variable. Click Edit to modify an existing environment variable.
e. After creating or modifying the environment variable, click Apply and then OK to have the change take effect.
**The graphical user interface for creating environment variables may vary slightly, depending on your version of Windows.

To create or modify environment variables from the CLI:
Use the SETX command (note the 'x' suffix) to set variables that persist after the cmd window has been closed.

setx <VARIABLE> "C:\<desired-path>" /m
Run the "set" command to verify the changes

If after correctly adjusting the values on these variables and following all steps listed above the installation process keeps rolling back WHILE also having the following ERROR about 'SSL certificate generation failed' present in splunk.log: 

2:46:07 PM C:\Windows\system32\cmd.exe /c ""C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" _internal pre-flight-checks --answer-yes --no-prompt >> "C:\Users\<USER>\AppData\Local\Temp\splunk.log" 2>&1"
The certificate generation script did not generate the expected certificate file:C:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem. Splunkd port communication will not work.
SSL certificate generation failed.
Creating: C:\Program Files\SplunkUniversalForwarder\var\lib\splunk
Creating: C:\Program Files\SplunkUniversalForwarder\var\run\splunk
Creating: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\i18n
Creating: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules\static\css
Creating: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\upload
Creating: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\search_telemetry
Creating: C:\Program Files\SplunkUniversalForwarder\var\spool\splunk
Creating: C:\Program Files\SplunkUniversalForwarder\var\spool\dirmoncache
Creating: C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\authDb
Creating: C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\hashDb

-Try bypassing the SSL generation by using the "CERTFILE" option while running the installer (& also while generating msiexec logs):

i.e.
msiexec.exe /i <splunk.msi> CERTFILE=<c:\path\to\forwarder.pem> CERTPASSWORD=<password> SPLUNKUSERNAME=admin SPLUNKPASSWORD=<YourPasswordOfChoice> AGREETOLICENSE=Yes /quiet

More guidelines on how to use these command line flags here.
 

Ultimately, if using CERTFILE to bypass SSL generation didn't help, in the Windows box:

a. Stop a service called "Manage Engine".
b. Remove the OPENSSL configs.
c. Run installer back.

Additional Comments:
To save all changes, restart the server before completing any steps that suggest running installer back.

**Check if Auto Run is enabled.

If you have enabled Windows to automatically run scripts, Splunk installation might fail if the autorun script fails.

As a workaround, you can install the forwarder with the following command: cmd /D /C msiexec.exe /i.

For example,

cmd /D /C msiexec.exe /i splunkforwarder-9.2.1-78803f08aabb-x64-release.msi /l*vx msiexec.log

Related Links

Repairing the Windows Installer service on a 64-bit OS

Installation Error: #1603

How to Get to HKEY_LOCAL_MACHINE

Create and Modify Environment Variables on Windows

Set a persistent environment variable from cmd.exe

Install a Windows universal forwarder