Configure NXOS to ACI Auto-Conversion Using POAP

Introduction

This document describes how you can switch OS (Operating System) from Nexus Operating System (NXOS®) to ACI  with Auto Provisioning (POAP) Auto-Conversion.

When it is needed?

In existing process, Return Material Authorization (RMA)/New switches if recieved with NXOS® image, It becomes very time consuming and complex to convert these switches to ACI mode and then adding them to the ACI fabric.

There are three case where you require this feature,

1) Replacement of faulty Leaf/Spine Switch

2) Adding new Leaf/Spine to extend the ACI Fabric

3) Addition/Replacement of Remote Leaf

Prerequisite

The exisitng process which is in place to convert NXOS® to ACI mode is very complex and time consuming so to simplify the procedure and make it close to plug and play (PowerOn Auto Provisioning) POAP Auto-conversion feature was introduced in ACI version 5.2(3). POAP feature is available on NXOS® starting 7.X relaease which concludes that the new switch you are trying to add must be running an higher release.

What is POAP?

POAP stands for Power on auto provisioning. Poap process is triggered automatically on Nexus switches when no startup config is found.

When Poap is triggered, switch is going to start Dynamic Host Configuration Protocol (DHCP) process from mgmt interface first and if no  DHCP ack is recieved this DHCP discovery is going to be started on all other ports. In ACI, APIC act as an DHCP server, which is going to provide IP address and python script location to the switches. Once python script is downloaded, this invokes ACI image download and conversion automatically take place on the switches. ACI runs Poap in infra network.

Topology

In this topology you have spine node 201, leaf node 101 and nxos switch which needs to be converted. This nxos switch can directly be connected to spine or leaf switch which means you can onboard it directly to the fabric once the conversion is successful.

For this configuration example, you are going to connect ACI Leaf 101 Port E1/53 to new switch node port E1/47. Please ensure that only Fabric Ports on Leaf and Spine can be used for POAP.

How to Configure POAP?

You can refer the steps demonstrated and make sure to power on the new nexus switch node and connect it to Leaf101 as dispalyed in the topology section.

Step1: Enable Nxos to ACI Conversion.

Navigate to Fabric --> Fabric Membership --> Registered Nodes --> Add with Nxos to ACI Conversion

Step2: Add the appropriate Node 101 (Existing ACI switch) and  interface E1/53 on Leaf 101 to be used for POAP (Only Fabric Links can be used for POAP on Leaf/Spine).

Step3: Double click on Leaf Node which you are using for POAP.

Step4: Validate POAP status of eth1/53, you can see POAP is enabled for port E1/53.

Step5: Connect Nexus node to the Leaf 101 interface E1/53 then you are going to see the new switch node under Nodes Pending registration. In this example we are using port E1/47 on new NXOS switch.

Step6: Register the new switch node, right click and select register.

Step7: Add node ID, node name and register the leaf.

Step8:During startup of NXOS switch, a prompt appears asking if you want to abort POAP and continue with a normal setup. choose no continue with POAP.

           (If the device has a startup configuration, you have to perform a write erase and reload the device to force nexus switch to POAP mode.)

Abort Power On Auto Provisioning [yes - continue with normal setup, skip - bypass password and basic configuration, no - continue with Power On Auto Provisioning] (yes/skip/no)[no]: 

>>>> This message appears on the console which means POAP process started on new switch node, Do not break this sequence.

Step9: Watch the console of new switch node, you are going to see these logs.

 %$ %POAP-2-POAP_DHCP_DISCOVER_START: [FDO233002HC-A4:53:0E:3D:D9:A3] - POAP DHCP Discover phase started     <<< POAP Process started, new switch node is sending DHCP discover message

2024 Jun 24 13:21:31 switch %$ VDC-1 %$ %POAP-2-POAP_INFO:   - Abort Power On Auto Provisioning [yes - continue with normal setup, skip - bypass password and basic configuration, no - continue with Power On Auto Provisioning] (yes/skip/no)[no]:

2024 Jun 24 13:21:32 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Recieved DHCP offer from server ip  - 10.0.0.1     <<< DHCP offer has been recived from APIC

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Recieved DHCP offer from server ip  - 10.0.0.1 (message repeated 1 time)

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Using DHCP, valid information received over Eth1/47 from 10.0.0.1   <<< This is the interface used on new switch node

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Assigned Host Name: poap-leaf   <<< Hostname assigned to new switch node based on node-name you specified while registering

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Assigned IP address: 10.0.232.68 <<< New switch node got an IP Address

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Netmask: 255.255.0.0

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - DNS Server: 10.0.0.1

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Default Gateway: 10.0.0.30

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Script Server: 10.0.0.1

2024 Jun 24 13:21:39 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Script Name: aci_poap_bootfile.py <<< This script is responsible for performing NXOS to ACI mode conversion

2024 Jun 24 13:21:49 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - The POAP Script download has started    <<< Downloading script

2024 Jun 24 13:21:49 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - The POAP Script is being downloaded from [copy tftp://10.0.0.1/aci_poap_bootfile.py bootflash:scripts/script.sh vrf default ]

2024 Jun 24 13:21:50 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_DOWNLOADED: [FDO233002HC-A4:53:0E:3D:D9:A3] - Successfully downloaded POAP script file     <<< Script downloaded

2024 Jun 24 13:21:50 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - Script file size 106768, MD5 checksum 8b31543966b6dd518441a45d3546e0e1

2024 Jun 24 13:21:50 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FDO233002HC-A4:53:0E:3D:D9:A3] - MD5 checksum received from the script file is 8b31543966b6dd518441a45d3546e0e1

2024 Jun 24 13:21:50 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_STARTED_MD5_VALIDATED: [FDO233002HC-A4:53:0E:3D:D9:A3] - POAP script execution started(MD5 validated)      <<< Validating MD5 Checksum before downloading ACI image aci-n9000-dk9.15.2.8h.bin

2024 Jun 24 13:21:52 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - Starting to download image - /script.sh       <<< Starting ACI image download as script is executed in the backend

2024 Jun 24 13:21:52 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - The command is : terminal dont-ask ; terminal password <removed> ; copy http://10.0.0.1:7777/fwrepo/aci-n9000-dk9.15.2.8h.bin bootflash:aci-n9000-dk9.15.2.8h.bin vrf default - /script.sh

184633.793: 2024 Jun 24 13:25:50 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - Downloading the image completed - /script.sh      <<< ACI image download completed

184633.819: 2024 Jun 24 13:25:50 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - Image file found in bootflash folder - /script.sh

184635.739: 2024 Jun 24 13:25:52 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - no boot nxos o/p : - /script.sh

184640.147: 2024 Jun 24 13:25:56 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - copy running-config o/p : [# ] 1% [# ] 2% [## ] 3% [## ] 4% [### ] 5% [### ] 6% [### ] 7% [#### ] 8% [#### ] 9% [##### ] 10% [##### ] 11% [##### ] 12% [###### ] 13% [###### ] 14% [####### ] 15%184640.147:  [####### ] 16% [####### ] 17% [######## ] 18% [######## ] 19% [######### ] 20% [#########

184642.147: 2024 Jun 24 13:25:58 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - boot aci o/p : Warning: Please check list of all ACI supported hardware before doing this operation, not all hardware are supported. Performing image verification and compatibility check, pleas184642.147: e wait.... Unable remove internal file Image verification successful. - /script.sh

184649.973: 2024 Jun 24 13:26:06 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG:  - md5sum o/p : e9065f12d6eac79d15091f0c595ed9e5 - /script.sh       <<< Post download MD5 checksum validation

184657.960: 2024 Jun 24 13:26:14 switch %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Successfully deactivated virtual service 'guestshell+' 

184701.033: 2024 Jun 24 13:26:17 switch %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface <<< Rebooting the device to perform the conversion

Step10: New switch node named as Poap-Leaf , starts showing in Registered Nodes.

Step11: Validate from CLI of new switch node it has been converted to ACI mode.

User Access Verification

(none) login: admin

********************************************************************************

     Fabric discovery in progress, show commands are not fully functional

     Logout and Login after discovery to continue to use show commands.

     Run show discoveryissues for more details.

********************************************************************************

(none)#  <<< Device is in ACI discovey mode now

Step12: Now connect the new switch node to the ACI fabric appropriately.

Step13: Delete the NXOS conversion policy post poap successful conversion.

++ You also encounter an Fault F427 when you configure POAP, please ensure that fault is cleared post step 13.
This fault is an reminder that you have configured an port on ACI Leaf switch for POAP usage

POAP dhcp logs on APIC

To review DHCP logs on the APIC, refer to the given location and log files.

apic1# cd /var/log/dme/log/
apic1# less dhcpd.bin.log | grep ISC | grep a4:53:0e:3d:d9:a3


30167||2024-06-24T13:12:04.170069236+00:00||dhcp||INFO||||ISC dhcpd: DHCPDISCOVER from a4:53:0e:3d:d9:a3 via 10.0.232.64||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68    <<< DHCP Discovered recieved from Leaf101 Infra Loopback IP , Leaf 101 is acting as DHCP Relay 

30167||2024-06-24T13:12:04.170129519+00:00||dhcp||INFO||||ISC dhcpd: DHCPOFFER on 10.0.232.68 to a4:53:0e:3d:d9:a3 via 10.0.232.64||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68      <<< DHCP Offer forwarded from APIC

30167||2024-06-24T13:12:07.219176308+00:00||dhcp||INFO||||ISC dhcpd: Received host decl = FDO233002HC { uid "FDO233002HC"; dynamic; option host-name "poap-leaf"; fixed-address 10.0.232.68; option cisco.node-role 2; option cisco.node-type 0; pool subclass "pod" 00:1; option cisco.fabric-id 1; option cisco.pod-id 1; option cisco.node-id 1891; option subnet-mask 255.255.0.0; option routers 10.0.0.30; option domain-name-servers 10.0.0.1; filename "aci_poap_bootfile.py"; } ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68   bico 04.421. <<< DHCP Server provides IP, DHS and TFTP server IP address to download python script and ACI image

30167||2024-06-24T13:12:07.220213143+00:00||dhcp||INFO||||ISC dhcpd: Received host decl = FDO233002HC { uid "FDO233002HC"; dynamic; option host-name "poap-leaf"; fixed-address 10.0.232.68; option cisco.node-role 2; option cisco.node-type 0; pool subclass "pod" 00:1; option cisco.fabric-id 1; option cisco.pod-id 1; option cisco.node-id 1891; option subnet-mask 255.255.0.0; option routers 10.0.0.30; option domain-name-servers 10.0.0.1; filename "aci_poap_bootfile.py"; } ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68

30167||2024-06-24T13:12:10.167297645+00:00||dhcp||DBG4||||ISC dhcpd: Before updateUid 12:||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||73   bico 07.220

30167||2024-06-24T13:12:10.167454624+00:00||dhcp||DBG4||||ISC dhcpd: not our server id ours 0.0.0.0 received 10.0.0.1||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||73

30167||2024-06-24T13:12:10.167588015+00:00||dhcp||DBG4||||ISC dhcpd: Before updateUid 12:||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||73

30167||2024-06-24T13:12:10.167639084+00:00||dhcp||INFO||||ISC dhcpd: DHCPREQUEST for 10.0.232.68 (10.0.0.1) from a4:53:0e:3d:d9:a3 via 10.0.232.64||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68

30167||2024-06-24T13:12:10.167706966+00:00||dhcp||INFO||||ISC dhcpd: DHCPACK on 10.0.232.68 to a4:53:0e:3d:d9:a3 via 10.0.232.64||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68

30167||2024-06-24T13:12:10.167806426+00:00||dhcp||DBG4||||ISC dhcpd: updateUid 11:FDO233002HC, ../common/external/dhcp/dhcp-4.1-ESV-R6/server/svcapi.c:410||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||73

30167||2024-06-24T13:12:12.221336321+00:00||dhcp||INFO||||ISC dhcpd: Received host decl = FDO233002HC { uid "FDO233002HC"; dynamic; option host-name "poap-leaf"; fixed-address 10.0.232.68; option cisco.node-role 2; option cisco.node-type 0; pool subclass "pod" 00:1; option cisco.fabric-id 1; option cisco.pod-id 1; option cisco.node-id 1891; option subnet-mask 255.255.0.0; option routers 10.0.0.30; option domain-name-servers 10.0.0.1; filename "aci_poap_bootfile.py"; } ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||68   bico 10.436