Dockerd Fails to Start on IE3400 when SD-Card for IOx is FAT-Formatted

Introduction

This document describes how to fix an issue where the Docker daemon fails to start on IE3400 when the SD-card is not ext4-formatted.

Problem

The IE3400 uses the SD-card for storage related to the IOx applications. In case the SD-card is not ext4-formatted, this causes issues.

These are the Symptoms:

When you try to deploy an application through IOx GUI, this message is seen:

Application Deployment Failed: Invalid Archive file: Unable to extract docker rootfs /flash11/iox/tmpArchiverPznor to /flash11/iox/tmpExtractYXZJbE/rootfs Error:global name 'app_mount_dir' is not defined

From the log/console, these messages are seen when you enable IOx or when IOx starts after a reboot.

ie3400#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ie3400(config)#iox
Warning: Do not remove SD flash card when IOx is enabled or errors on SD device could occur.
 
*Feb 21 12:45:27.045: %UICFGEXP-6-SERVER_NOTIFIED_START: R0/0: psd: Server iox has been notified to start
*Feb 21 12:45:30.737: %PMAN-3-PROCFAIL: R0/0: root: The process dockerd has failed (rc 1)
*Feb 21 12:45:32.184: %PMAN-3-PROCHOLDDOWN: R0/0: root: The process dockerd has been helddown (rc 1)
*Feb 21 12:46:57.983: %IM-6-IOX_ENABLEMENT: R0/0: ioxman: IOX is ready.

Solution

This issue is related to the File System of the SD-card. If the card is formatted with something else than ext4, for example, FAT/vFAT, the issue seems to appear.

In order to check the current file system on the SD-card, this can be used:

ie3400#sh sdflash: file
Filesystem: sdflash
Filesystem Path: /flash11
Filesystem Type: vfat
Mounted: Read/Write

As you can see in the output, this SD-card was formatted as FAT/vFAT.

In order to fix this issue, firstly stop/disable IOx.

ie3400#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ie3400(config)#no iox
Notify IOx to stop. To safely remove SD card, please wait until IOx is stopped.
*Feb 21 12:48:01.615: %UICFGEXP-6-SERVER_NOTIFIED_STOP: R0/0: psd: Server iox has been notified to stop

 Use the show iox command to ensure that it is fully stopped.

ie3400#sh iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF)         : Not Running
IOx service (HA)          : Not Supported
IOx service (IOxman)      : Not Running
IOx service (Sec storage) : Not Supported
Libvirtd   1.3.4          : Running
Dockerd                   : Not Running

Next, format the card correctly for being used with IOx, you can use this command.

Keep in mind that this erases all data from the SD-card.

ie3400#format sdflash: ext4
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "sdflash:".  Continue? [confirm]
format completed with no errors

Format of sdflash: complete

Once completed, start/enable IOx again.

ie3400#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ie3400(config)#iox
Warning: Do not remove SD flash card when IOx is enabled or errors on SD device could occur.

*Feb 21 12:49:18.310: %UICFGEXP-6-SERVER_NOTIFIED_START: R0/0: psd: Server iox has been notified to start
*Feb 21 12:49:48.165: %IM-6-IOX_ENABLEMENT: R0/0: ioxman: IOX is ready.

As you can see, the error messages regarding dockerd are no longer seen and use this command show iox to check the status.

ie3400#sh iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF) 1.10.0.1 : Running
IOx service (HA)          : Not Supported
IOx service (IOxman)      : Running
IOx service (Sec storage) : Not Supported
Libvirtd   1.3.4          : Running
Dockerd    18.03.0        : Running