Nexus 5000 Series Switches SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is unexpectedly high Error Message Troubleshoot

Introduction

This document describes a problem encountered with Nexus 5000 Series switches and also provides a solution and temporary workaround for the problem.

Prerequisites

Requirements

Cisco recommends that you have knowledge of Cisco Nexus CLI.

Components Used

The information in this document is based on Nexus 5000 Series switches that run any version earlier than 5.0(3)N2(1).
 
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

The Nexus 5000 Series switch reports this error every three minutes:

N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
 unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
 unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
 unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
 unexpectedly high at 80%.

When you enter the show run command (and produce more than 190 lines of output) or the show run switch-profile command (regardless of the output size), a copy of the CSM_ACFG file in /dev/shm file might generate. These files are not automatically cleaned out. Eventually they fill up the volatile memory, which causes the device to reload. Additionally, the issue is exacerbated if some sort of script is used which periodically collects or alters the configuration.

In order to troubleshoot this issue, first collect the output from the show system internal flash command. It should show high usage in the /dev/shm folder:

N5k# show system internal flash
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
/                            204800    115408       89392     57   /dev/root
/proc                             0         0           0      0   proc
/post                          2048         4        2044      1   none
/sys                              0         0           0      0   none
/isan                       1536000    452496     1083504     30   none
/var/tmp                     307200       704      306496      1   none
/var/sysmgr                 1024000      6320     1017680      1   none
/var/sysmgr/ftp              409600     48604      360996     12   none
/var/sysmgr/ftp/cores        102400         0      102400      0   none
/callhome                     61440         0       61440      0   none
/dev/shm                     524288    427248       97040     80   none  <<<<<<<<<
/volatile                    153600         0      153600      0   none
/debug                        20480         4       20476      1   none
/dev/mqueue                       0         0           0      0   none
/mnt/cfg/0                   114909      4904      104072      5   /dev/sda5
/mnt/cfg/1                   112920      4904      102186      5   /dev/sda6
/var/sysmgr/startup-cfg      307200      9940      297260      4   none
/dev/pts                          0         0           0      0   devpts
/mnt/plog                     56192      1644       54548      3   /dev/mtdblock2
/mnt/pss                     114917      5348      103635      5   /dev/sda4
/bootflash                  1609984    410536     1117664     27   /dev/sda3

In order to confirm that the contents in the /dev/shm folder are the csm_acfg files, collect the output from these commands:

• show system internal dir /dev/shm | i csm_acfg | count
• show system internal dir /dev/shm | i csm_acfg

If the Nexus 5000 Series switch crashes, it reports this message as a reload reason in the show system reset-reason command output:

Reason: Reset triggered due to HA policy of Reset
System version: 5.0(2)N2(1)
Service: syslogd hap reset

When the show logging nvram command is entered, the output shows errors such as:

N5k %$ VDC-1 %$ %PSS-0-PSS_WRITE_LOG_FAILURE: snmpd: failed to write log: No space left on device
N5k %$ VDC-1 %$ last message repeated 4 times

Solution

Upgrade to version 5.0(3)N2(1) in order to resolve this issue. For more informaton about this issue, reference Cisco bug ID CSCtn71292.

Additionally, as a temporary workaround:

• If possible, do not poll or run any commands that create new files.
• Enter the system pss shrink command in order to try and reduce the size of the /dev/shm folder.
• Contact the Cisco Technical Assistance Center (TAC) for assistance. The TAC can attempt to remove the files in the /dev/shm folder.