Troubleshoot Error "START_BACKING_THREAD:BIND: no SPACE LEFT" on Device NCS1K 7.3.2

Introduction

This document describes how to troubleshoot and clear the "START_BACKING_THREAD:BIND: no SPACE LEFT" error in NCS1K.

Problem Description

Observing start_backing_thread:bind: No space left on the device unable to monitor alarms in the device (rootfs is filled 100%). PM logs collection in rootfs directory and filling it up. Version: XR 7.3.2

Observations and Actions Performed

• Command  <show alarms b s a>  output is not displayed and observed error "start_backing_thread:bind: No space left on device" and rootfs is fully occupied.
• Rootfs is filled 100%.

Show Media Output

• Pm log files are filled in rootfs directory.PM logs stored in rootfs

Procedure to Delete PM Files

This command displays all the files in the root directory.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#run
[xr-vm_node0_RP0_CPU0:~]$cd /root
[xr-vm_node0_RP0_CPU0:/root]$ls -ltrah   

This command deletes the file from the directory.

rm “filename” ----à 

Verify the size of rootfs using  show media command.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#show media

• It is important to know why auto deletion did not occur in the system.
• It is identified there is a Python script enabled for automatic log collection every 1 hour in rootfs.
• The pm log generation is not done by the device by default, It is configured manually. Hence, this auto-deletion also needs to be done manually.
• Need to verify the source from where the pm log files are being stored in rootfs to kill this process.
• It is found that some Python scripts are running in the background to collect pm logs in rootfs in 2 nodes out of 4 ncs1k4 nodes in their network.
• These were added during deployment time.

Hard Disk Files

To kill the Python process, use the  reload  command in XR.

Resolution:

To terminate these Python processes, you need to  XR RELOAD  on the nodes.

XR RELOAD  - Command reloads the VM.

RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#reload  (Reloads the VM.)

Reloading does not impact service but you must lose management visibility for around 5-10 mins. You can also clean up the pm-related files using  rm  command:

<rm -rf "/misc/disk1/pm_data">

<rm -rf "/root/*pm.txt">

<rm -rf "/root/PM-data*">

<rm -rf *2022.tgz (deletes all the files ending with 2022.tgz)>

Similarly, you can delete files that are not needed.