Field Notice: FN74442 - Some Network Convergence Series 540 Series Routers May Experience Router Access and System Admin VM Authentication Failure After Initial Boot - Workaround Provided

Cisco has identified an issue that affects a limited number of Cisco Network Convergence System (NCS) 540 Series Routers. The problem manifests differently depending on the product variant:

• Cisco N540-ACC-SYS, N540X-ACC-SYS, and N540-24Z8Q2C-M Routers: Users are unable to gain access to the System Admin virtual machine (VM). Despite having configured credentials, the system fails to authenticate the user within the admin plane.
• Cisco N540X-6Z and N540X-8Z Routers: Users are unable to log in to the router upon boot. The console does not trigger the expected root system authentication flow, effectively locking the user out of the device.

This issue has been traced to specific manufacturing scripts that were utilized during the production of these units. The scripts did not properly initialize the user access control configurations in the firmware image for these specific batches. Cisco has already updated its manufacturing processes and corrected the scripts to ensure that this issue does not occur in future shipments.

Note: This is not a hardware defect. The affected units are fully functional. However, the software-level configuration requires a manual recovery process to restore user access. 

Customers can verify if their unit is affected by observing the following behaviors:

• For Cisco N540-ACC-SYS, N540X- ACC-SYS, or N540-24Z8Q2C-M Routers: When attempting to access the admin plane via RP/0/RP0/CPU0:ios#admin, the system returns the following error: Unable to login - Group root-system not found in admin-plane.
• For Cisco N540X-6Z18G-SYS-A and N540X-8Z16G-SYS-D Routers: During the boot sequence, the console fails to prompt for the root system user. Instead, it presents a standard generic prompt, Getting Username, instead of Enter root-system username.

To regain control of the device, users must perform a network-based recovery using the iPXE boot environment. This allows for a clean configuration of the required credentials.

Recovery Procedure Using iPXE

1. Initiate boot: Power cycle the router.
2. Access menu: Repeatedly press Esc during the boot sequence until the U-Boot menu appears.
3. Select iPXE: Choose iPXE boot.
4. Enter shell: When prompted, press Ctrl+B to access the iPXE CLI.
5. Network configuration: Configure the network interface to reach the TFTP server by executing the following commands (users should replace placeholders with their environment-specific values):
   • ifclose net0
   • set net0/gateway <network_gateway>
   • set net0/ip <management_ip>
   • set net0/netmask <netmask>
   • ifopen net0
6. Execute recovery: Run the boot command to load the image:
   • boot tftp://<server_ip>/<image_path>/<image_name>
7. Post-boot configuration: Once the system boots using the reloaded image, proceed to reconfigure the administrative credentials as required.

Recovery Procedure Using USB

This method reimages the router using a bootable USB drive.

Prerequisites:

• A bootable USB drive that is prepared with the required Cisco IOS XR image
• Physical access to the active USB RP on the router
• A backup of the existing configuration before initiating the recovery process (highly recommended)

Step 1

Put the prepared bootable USB drive into an active USB RP on the router.

Step 2

Force the router to boot from the USB drive using one of the following two methods.

Method 1: Manual boot manager

1. As the router begins to reload, press Esc repeatedly until the Boot Manager window appears.
2. Choose Boot Manager from the menu.
3. From the list of available devices, choose the USB drive and press Enter.

The router will load the image from the USB drive and install it onto the internal hard disk, as shown in the following example:

a    Cisco BIOS Setup Utility - Copyright (C) 2019 Cisco Systems, Inc
    b    
    c    Boot Override
    d    UEFI: Micron_M600_MTFDDAT064MBF, Partition 4
    e    UEFI: Built-in iPXE
    f    URFI: Built-in Shell
    g    URFI: Built-in Grub
    h    EFI USB Device (Sandisk) 
    i    UEFI: IPv4 0 Intel® I210 Gigabit Network Con
    j    UEFI: IPv4 0 Intel® Ethernet Connection x552
    k    UEFI: IPv4 1 Intel® Ethernet Connection x552
    l    UEFI: IPv4 2 Intel® Ethernet Connection x552
    m    UEFI: IPv4 3 Intel® Ethernet Connection x552

The router boots from the hard disk after the installation.

Method 2: The reload bootmedia usb command

1. Run reload bootmedia usb.
2. Confirm the prompt with yes or confirm.

The system will perform a configuration backup and automatically initiate the reboot process using the USB media, as shown in the following example:

RP/0/RP0/CPU0:ios# reload bootmedia usb
Proceed with reload? [confirm]
Preparing system for backup...
Status report: node0_RP0_CPU0: BACKUP HAS COMPLETED SUCCESSFULLY
[Done]

A return material authorization (RMA) is not recommended for this issue. For further diagnostic support, contact the Cisco Technical Assistance Center (TAC).

For instructions on identifying an affected device using the serial number, see the Serial Number Validation section of this field notice.

Once the installation is complete, the router will automatically reboot from its internal storage. If the device does not recognize the USB drive, verify the following:

• Drive formatting: Ensure the drive is formatted exactly as described in the Create a Bootable USB Drive Using Compressed Boot File section of the System Setup and Software Installation Guide for Cisco NCS 540 Series Routers.
• Port selection: Insert the drive into the designated USB service port on the front panel.
• Image integrity: Verify that the ISO image on the drive is not corrupted to avoid potential boot loops.