Universal Forwarder server on Windows OS rebooted automatically while upgrading to 9.1.1

Applicability

Description

Universal Forwarder was upgraded from v7.3.3 to v9.1.1. This upgrade process was done via System Center Configuration Manager (SCCM). But at the time of upgrade, the UF got rebooted. It was an unexpected reboot as UF was not set to automatic reboot.

*SCCM is a Microsoft software management tool that can be used to upgrade devices by automating the management, monitoring, and deployment of devices in a large organization with many devices.

Investigation

- Upon investigating the MSI logs, the logs were observed :

MSI (s) (54:9C) [12:51:31:136]: File: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe; Overwrite; Won't patch; Existing file is a lower version
MSI (s) (54:9C) [12:51:31:152]: Note: 1: 2205 2: 3: Error
MSI (s) (54:9C) [12:51:31:152]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1603
MSI (s) (54:9C) [12:53:31:294]: Verifying accessibility of file: splunk-winevtlog.exe
MSI (s) (54:9C) [12:53:31:294]: Note: 1: 2318 2:
MSI (s) (54:9C) [12:53:31:294]: Note: 1: 2318 2:
Info 1603.The file C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe is being held in use. Close that application and retry.MSI (s) (54:9C) [12:54:02:372]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1903
MSI (s) (54:9C) [12:54:02:388]: Note: 1: 2318 2:
MSI (s) (54:9C) [12:54:02:388]: Note: 1: 2318 2:
Info 1903.Scheduling reboot operation: Deleting file C:\Config.Msi\27fa9d8b.rbf. Must reboot to complete operation.

- The below log indicates that the splunk-winevtlog.exe file was still held in use by another process and the Windows installer is unable to update or replace this file.

MSI (s) (54:9C) [12:53:31:294]: Note: 1: 2318 2: Info 1603.The file C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe is being held in use. Close that application and retry.

- As per the above message, it logs an entry with the code 1603, and renames this file with a random string with .rbf extension.  The .rbf file is actually the original file logged by the 1603 message which has been renamed to the .rbf file. 
 
- And then logs a message indicating reboot is required with code 1903. 

MSI (s) (54:9C) [12:54:02:388]: Note: 1: 2318 2: Info 1903.Scheduling reboot operation: Deleting file C:\Config.Msi\27fa9d8b.rbf. Must reboot to complete operation.

- If the installer detects that it is about to overwrite a file that is in use,  then it logs the above messages. 
 
- Officially, reboot is not required on a clean install of Splunk UF. However, in some edge cases such as explained above, Windows OS may lead to an unexpected reboot. And this could happen with any separate process that might be running and required elsewhere.

Resolution

- While doing the upgrade, use the below parameter with the upgrade command to suppress all the restart prompts.

reboot=reallysupress

- The command should look like :

msiexec /i splunkforwarder-x64-release.msi AGREETOLICENSE=Yes reboot=reallysuppress

- This parameter indicates all restarts and restart prompts initiated by ForceReboot will be suppressed during the installation. 

Related Links

Reference documentation:
msi/logging-of-reboot-requests
msi/filesinuse-dialog
system-error-codes

msi reboot