Assessing Universal Forwarder Impact When Migrating Virtual Machines Between Hypervisors

Applicability

- Splunk Universal Forwarder - Splunk Enterprise - Hypervisors & Cloud: VMware vSphere, KVM, Nutanix AHV, AWS, Azure, GCP - Operating Systems: Linux (Ubuntu, RHEL, CentOS) and Windows Server

Description

A virtual machine running Splunk Universal Forwarder is migrated from one hypervisor platform to another (such as migrating from VMware to Linux KVM) using a 1:1 disk-image migration tool while preserving the operating system, Splunk installation directory, and configured network IP address.

During the migration, virtualization and hardware-layer identifiers change, including:

• Virtual network interface MAC address
• System and SMBIOS UUID (e.g., /sys/class/dmi/id/product_uuid)
• OS filesystem and disk UUIDs (e.g., values in /etc/fstab, blkid, or Windows volume GUIDs)

Impact Overview:

• Licensing: Splunk Universal Forwarder licensing is volume-based and centrally validated at the Indexer tier. Universal Forwarders do not bind to or validate host hardware, network, or filesystem identifiers.
• Instance Identity: Forwarder instance identity is defined by an internal GUID stored in $SPLUNK_HOME/etc/instance.cfg. Because a 1:1 disk-image migration transfers the existing filesystem intact, this GUID and the monitored-file read pointers (fishbucket) remain preserved.
• Golden Image / Cloning Exception: This guidance applies strictly to direct 1:1 host migrations. If an image is cloned to deploy multiple distinct forwarder instances, clear the GUID prior to template capture to prevent duplicate instance identities.

Investigation

1. Confirm that the migration is a direct 1:1 host transfer and the $SPLUNK_HOME directory structure remains intact.
2. Confirm that local directory paths for monitored log files remain identical so that the fishbucket checkpoint mechanism ($SPLUNK_HOME/var/lib/splunk/fishbucket) can resume tracking seamlessly.
3. Inspect the forwarder GUID in $SPLUNK_HOME/etc/instance.cfg to verify instance identity continuity:

   grep -i "guid" $SPLUNK_HOME/etc/instance.cfg

Resolution

1. Stop the Universal Forwarder Cleanly Before Migration:

   Flush in-memory queues and commit fishbucket checkpoint data to disk before capturing the final cutover snapshot:

   $SPLUNK_HOME/bin/splunk stop

2. Start the Service After Host Boot:

   Once the virtual machine is powered on in the target virtualization environment, start the forwarder and confirm operational status:

   $SPLUNK_HOME/bin/splunk start
   $SPLUNK_HOME/bin/splunk status

3. Verify Network Connectivity:

   Confirm that network routing and firewall rules permit outbound communication from the forwarder to:

   • Receiving Indexers on TCP port 9997 (or configured ingestion port).
   • Deployment Server on management port TCP 8089 (if centrally managed).

4. Check Local Forwarder Logs:

   Inspect splunkd.log to confirm receiver connections and rule out queue blockage:

   grep -iE "error|warn" $SPLUNK_HOME/var/log/splunk/splunkd.log

    

5. Verify Ingestion on the Receiving Indexer Tier:

   Run the following SPL search on the Search Head to verify active data stream resumption:

   index=_internal group="tcpin_connections" sourceHost="<migrated_host_name>"
   | stats latest(_time) as last_connected, values(version) as version, values(os) as os by sourceHost

    

6. Prepare Template for Multi-VM Cloning (If Applicable):

   If the VM image is used as a template to create multiple virtual machines (rather than a 1:1 migration), run the clone preparation command prior to template capture:

   $SPLUNK_HOME/bin/splunk clone-prep-clear-config

    

Related Links

• Splunk Documentation: Integrate a universal forwarder onto a system image
• Splunk Documentation: Types of Splunk Enterprise licenses

Revision History

Revision	Publish Date	Comments
1.0	08-Nov-2026	Initial Release