vManage Kernel Panic and Release 20.3.2.1

Introduction

This document describes the problem that occurs when kernel panic is experienced, instance in vManage reboot and the release of version 20.3.2.1.

Problem

It has been observed that some of the deployment has experience kernel panic and the vManage instances haves rebooted. It was observed that the APP-server has high memory footprint. The signature would be Linux processes crashing due to Out of Memory (OOM), with free memory bounding lower or around 300MB free.

Problem areas identified are:

1. IDE vs SCSI

2. OOM - Kernel Panic

Solution

This document guides you through the changes that need to be made on ESXI 6.7 (16713306).

Configure Data Disk Controller

The option will be made available depending on what the underlying hardware is capable of. If the disk type that the datastore is managed is an HDD then only SCSI option will be available. If the disk hosting datastore is of the media type SSD then you have the option of configuring both SCSI and SATA. Here we are recommending that the disk controller be configured as SATA.

Take config-db backup or VM snapshot before you proceed.

Follow these steps:

1. Use the command Request nms all stop from the vManage CLI session.
2. Power down the VM.
3. Add a new device on each of the vManage instances if it has not already been added.
4. The new device should read SCSI or SATA.
5. Edit the VM settings on the disk hosting /opt/data to SATA or SCSI.
6. Power back the VM.

Configure HDD Disk Type - IDE to SCSI

1. Select the Edit option for the VM where the controller needs to be changed.

2. Select the Add other device option as shown in the image.

Select the controller type based on the underlying hardware. 

Select the options based on Hard Disk type.

• SATA for SSD.
• SCSI for HDD.

3. Click on the disk where /opt/data is hosted. Very likely this will be IDE as indicated:

4. Finally, switch it out from IDE to SCSI or SATA.

5. Click Save.

6. Power back the VM.

Upgrade to 20.3.2.1.

With upgrading system to version 20.3.2.1, the way that vAnalytics integration with CloudServices has changed. For these improved changes to take effect, Admin has to manually enable Cloud-Services and vAnalytics.

1. Config-DB optimizations have been made to make the query time much faster.

2. System friendly optimizations have been put in place to make sure that the disk is not overwhelmed by device data and necessary safeguards have been put in place to stop data collection when 80% disk space is utilized.

3. If vAnalytics is not collecting data from vManage, the vManage system will not keep writing to the disk indiscriminately. If vManage observes that vAnalytics has a backlog of 10% disk space worth of data, it will halt writing data for vAnalytics until vAnalytics resumes its data collection.