Introduction
This document describes the analysis and resolution of container installation failures caused by OCI packaging incompatibilities with Cisco Catalyst
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- App-hosting on Cisco Catalyst 9000 Series switches
- Docker container image formats, specifically Docker and Open Container Initiative (OCI) standards
- Differences in Docker Engine versions and their compatibility with Cisco IOS® releases
- Basic familiarity with Cisco IOS® CLI and troubleshooting container installations
Components Used
The information in this document is based on these software and hardware versions:
- Cisco Catalyst 9300X Series Switches
- Cisco IOS® software
- Third-party application packaged as a container using OCI format
- Solid-State Drive(SDD) drive as storage for app-hosting containers
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
App-hosting on Cisco Catalyst 9000 Series switches allows the deployment of container-based applications directly on the switch hardware, leveraging the IOx platform. These containers often use the Docker image format. Recent developments in container packaging standards have introduced the Open Container Initiative (OCI) format, which is not universally supported across all Docker Engine versions. With the release of newer Cisco IOS® versions, Cisco has incrementally added support for more recent Docker Engine versions, thereby expanding compatibility with OCI-packaged containers. However, not all Cisco IOS® releases, nor all Catalyst platforms, provide support for OCI packaging. As organizations seek to deploy third-party applications that leverage newer standards, mismatches between software versions and container formats can cause installation failures and related errors.
Problem
The objective was to upgrade or install a third-party application as a container on Cisco Catalyst 9000 Series switches, specifically on devices running Cisco IOS® 17.6.4. The application was packaged as an OCI-compliant Docker image and stored on the SSD drive. During installation, the process failed with errors related to the archive file structure and missing mandatory blobs. This is a detailed reproduction and illustration of the issue.
Step 1: Attempt to Install the Container on Cisco IOS® 17.6.4
Device#Aug 15 13:11:11.389 UTC: %IM-6-INSTALL_MSG: R0/0: ioxman: app-hosting: Install failed: Invalid Archive file: Unable to extract docker rootfs /disk0/pathview-cmp.tar to /vol/disk0/caf/tmpExtractf117jdv6/rootfs Error:Mandatory layer blobs is missing!
This error message indicates a failure during the extraction of the Docker files from the provided tar file, specifically referencing missing mandatory layer blobs. This is symptomatic of an incompatibility between the container image format and the Docker Engine version available on the switch.
Step 2: inspect the Structure of the Tar File
Device#archive tar /table flash:pathview-cmp.tar
blobs/ (directory)
blobs/sha256/ (directory)
blobs/sha256/0318da79d21ba1e610bde0c6029f424293316cb913bb8e39a673c1f344be8727 (477 bytes)
blobs/sha256/046b3f7b867b1b7b8abd8ad22b3d64d1692f670c6a6e210a03ad3d31e9e80e79 (2996736 bytes)
blobs/sha256/04f0912220f0c2c6935e817813866ba3bdadba386edc577325a83a1fdb4879d2 (2560 bytes)
blobs/sha256/1e5cb17607b5c739629c9d0be80fa48f57ccd3507962ac221c9fbb91b20af171 (3584 bytes)
blobs/sha256/26323df902297791c86ab980a2cfaa76a9939bc7cc72646d6d0fb02db2fd0c32 (477 bytes)
blobs/sha256/2e4b80ea022a9c575ff90f41c4a533a0104a0361be4f563d50ec128aea973f57 (3072 bytes)
blobs/sha256/31aedc7091e9c30e45a6b276bd3bbe922ded9791ef4a9b9c9e44d3c9ddd74183 (477 bytes)
blobs/sha256/4172f6ce43d7636f98bfca81b0dc64ab7a600bbc7c60916c70a89749d092bd28 (11776 bytes)
blobs/sha256/4654182bbf590a33df0930847ce2803792ad2822e19c630f00bb39abb0c98ef6 (11676 bytes)
blobs/sha256/4ba2623fac214b72c911695418ed7d4d3b41dc1ed9101e448ca124ed126f0cf5 (401 bytes)
blobs/sha256/625d905965265a01abf82f80bd0c9c103ab229efbffdf8e17e5408a97d9ba470 (477 bytes)
blobs/sha256/6ba030bdcefc282c164bc0c6b4dfa8f30f085487aba82c1524a55649f2741e98 (1908 bytes)
blobs/sha256/6c4c763d22d0c5f9b2c5901dfa667fbbc4713cee6869336b8fd5022185071f1c (77895680 bytes)
blobs/sha256/6def6f10cc5b9a7dc012ca9e9321013a1d947c60f7ef902af2bcdef2fc53efcb (207397376 bytes)
blobs/sha256/7a2f3e2b33a720bc5ba562639d581574c997fa47efb26486360e0cad9d311573 (4096 bytes)
blobs/sha256/96cbe396af0cb40feb2bdab0d22e2a5f5ca3865455d7c8ca28245e1462eaf2be (477 bytes)
blobs/sha256/a0b1b29152e438cee663922cf019cc9beccd1958e3385cbb0b1c4b2c8f5cf0f8 (2048 bytes)
blobs/sha256/a9775757eec8bb747ca1dd66356a643943d4c9e18cccb7135305f9f3ff3c6790 (3584 bytes)
blobs/sha256/b5a729e23ce10966aa1935040de68b24dedc7b7cbc98dd233307623f95330493 (1971200 bytes)
blobs/sha256/c21b565f0164ee4f392a2f3507b36a5f7d06c77db979e84e9fd6100f3c5f1ef5 (2220 bytes)
blobs/sha256/c5b39eb2dc5af9a446af02dd67237191f5fc164dce4bfeb2cbc63ab7802fc50f (29141504 bytes)
blobs/sha256/d0539e4616d88232f284a9d469ee309dec1850aa8b6ed4f57b2e84f413eba34d (477 bytes)
blobs/sha256/d13aa916c45e4866e0bc3af1a68ee7360b5e1c074cd0849f67b72b64e5e4d4f5 (477 bytes)
blobs/sha256/d4924836d34beac0bc207c42351234b771a3f2a0d3a0bc1e8b1aa175edb70852 (477 bytes)
blobs/sha256/d86d4fab4a9109fea084cccddefb3bb78b69d20674a1104d8ca3170ed4ad50d7 (477 bytes)
blobs/sha256/dd1f33c08d67af57858bcc4aa1cedd4cfa00560b77b248e0903ae26bb66bea87 (477 bytes)
blobs/sha256/efdcd58e09999ffc1bca3ca8256a03fdf8fa90713478d2ed13f884ddd2e3cad2 (12800 bytes)
blobs/sha256/f356261d8da7bf5ad814f757b1a51b9be2eb74c59cfcda944be2d31ab56f5ecd (477 bytes)
index.json (381 bytes)
manifest.json (4135 bytes)
oci-layout (31 bytes) <<< Indication that the container is created using OCI standard
repositories (101 bytes)
device#
Despite the extraction, the installation failed, confirming the issue was not with file integrity but with format compatibility. The container tar file was identified as using OCI structure, which is not supported by the Docker daemon(dockerd) version present in Cisco IOS® 17.6.4.
Step 3: Check IOx and Dockerd Version on the Switch
Device#show iox
IOx Infrastructure Summary:
---------------------------
IOx service (CAF) : Running
IOx service (HA) : Running
IOx service (IOxman) : Running
IOx service (Sec storage) : Not Running
Libvirtd 5.5.0 : Running
Dockerd 18.03.0 : Running <<< This version doesn't support OCI packaging
Sync Status : Disabled
This output shows that the switch is running Dockerd version 18.03.0, which does not support OCI-packaged containers. This is the root of the installation failure for OCI-formatted images.
Solution
The solution involves upgrading the Cisco IOS® software to a version that supports a recent enough Docker Engine (dockerd) to allow installation of OCI-formatted containers. The process is described step by step.
Step 1: Upgrade the Switch to a Compatible Cisco IOS® Version
Upgrade the switch to Cisco IOS® version 17.8.1 or higher. This version includes Dockerd v19.03.13-ce or later, which supports OCI packaging.
Step 2: Verify the Dockerd Version After Upgrade
Device#show iox
IOx Infrastructure Summary:
---------------------------
IOx service (CAF) : Running
IOx service (HA) : Running
IOx service (IOxman) : Running
IOx service (Sec storage) : Running
Libvirtd 5.5.0 : Running
Dockerd v19.03.13-ce : Running <<< Version that supports OCI packaging
Sync Status : Disabled
Ensure that Dockerd is running at v19.03.13-ce or higher. If so, OCI-formatted containers can now be installed successfully.
Step 3: Retry the Container Installation
After confirming the Docker Engine version, repeat the container installation process. The installation now completes successfully without errors related to archive extraction or missing blobs.
Cause
The underlying cause of the issue was the use of a container image packaged in the Open Container Initiative (OCI) format, which is not supported by the version of Docker Engine (dockerd 18.03.0) included with Cisco IOS® 17.6.4. Only Cisco IOS® versions 17.8.1 and higher, which include Docker Engine v19.03.13-ce or later, support OCI packaging. Upgrading the Cisco IOS® software resolves the incompatibility.
Related Information
Feedback