Understanding and Troubleshooting PXE-boot across the ACI fabric

Preboot eXecution Environment (PXE, sometimes pronounced as pixie)

PXE or Network Booting allows BIOS to download and boot an Operating System (OS) over the network. In Cisco NX-OS routers, this is equivalent to the boot loader [loader>] aka Kickstart.

In PXE boot, the PXE Client’s BIOS looks at the configured boot order and, if PXE boot is configured, determines if the hardware supports PXE boot. If yes, it tries to DHCP an IP address off the NIC. In the DHCP request, the PXE client requests certain DHCP options such as 67 [Filename]. In the DHCP reply, the DHCP server replies with the requested options. The PXE client then downloads the filename [Network Bootstrap Program (NBP)] specified in the DHCP response and runs it to get the OS loaded onto the server.

Redirect PXE client to different TFTP server

By default, the PXE client will try to TFTP download the filename in the DHCP response off the DHCP server unless it receives an option 66, next-server, or option 60/43 in the DHCP response to redirect it to a different ip address. The next-server address is used in bootp and PXE to specify to use different servers for DHCP and TFTP.

The next-server (option 12) and bootfile (option 0) DHCP options typically specify the TFTP server and filename of the initial boot loader. Bootfile is optionally used by a client to request a particular type of boot file in a DHCPDISCOVER message. It is used by a DHCP server in a DHCPOFFER to fully specify a boot file directory path and filename.

Option 67 must also be specified for the filename. The difference between bootfile and option 67 is where in the DHCP response the filename is found. With the reserved fields, the responses are not numbered in the Options part of the response packet but given in a specific order.

For Example, Server Host Name and Bootfile appear before the numbered options in the image below.

Important Troubleshooting Notes

If the PXE boot client does not request a specific DHCP option in the Option 55 Parameter Request List in the DHCP request packet, the DHCP option WILL NOT BE in the DHCP server response packet even if it is configured on the DHCP server. It is important to get a sniffer trace to verify the options requested and given.

NOTE: if ACI is configured for DHCP ip helper address (DHCP Relay) to an DHCP server, then tcpdump may be used on the first hop leaf switch. Because the leaf switch must act as a DHCP relay, the CPU must process the DHCP packets to forward them to the DHCP server. You can login using the admin account and use the command

tcpdump -i any -w /tmp/packet.pcap "port 67"

to capture the DHCP packets between the host and DHCP server. If the leaf switch does not have any OOB ip address, you can transfer the captured file to the APIC by logging into the apic using the admin username and using

scp admin@<switch TEP>:/tmp/packet.pcap /home/admin

You can then transfer the file off the APIC using the GUI ip address.

There can be many different filenames depending on what PXE server supports. The most common boot loader filename for a linux server is called "/pxelinux.0", and is typically located in the /tftpboot directory.

DHCP Options That Might Be Used in PXE booting

Option 43 is returned to the PXE client if the DHCP server matches its option 60 string to the option 60 string sent in by the client. Option 43 has several sub-options in ascending order (option 6 then 8, etc.):

Sub-option 6 is defined as

06 : sub-option 6

01 : 1 byte long

07 : use PXE_BOOT_SERVERS list, disable multicast and broadcast discovery (unicast only)

Sub-option 8 is defined as

08 : sub-option 8

07 : size of 08 option in bytes

0000 : PXE server type 0 (PXE boot server)

01 : number of PXE servers

0afbd015 : 10.251.208.21

The option looks like this in the DHCP configuration

Option 43 hex 06010708070000010afbd015

According to https://support.microsoft.com/en-us/kb/259670, when the DHCP server has these options set:

60 = Client Identifier (set to "PXEClient")

66 = Boot Server Host Name or IP address

67 = BootFile Name

and when the initial DHCP offer from the DHCP server contains these boot options, an attempt is made to connect to port 4011 on the DHCP server from the PXE client and the error “Proxy DHCP service did not reply on Port 4011” occurs.

NOTE: Microsoft does not support the use of these options on a DHCP server to redirect PXE clients.

Customer Topology

In this a sample case, the user had 2 different ACI fabrics. In ACI fabric #1, the PXE clients were DHCPing and booting off a Microsoft server within the fabric. In the other ACI fabric, the PXE clients were DHCPing off an external IOS-XE 4507 router via an L3Out but using the same Microsoft server to download the file so that the DHCP server was not acting as the TFTP server.

The resolution the customer finally adopted was to move to a linux DHCP server. Analyzing the DHCP packet traces, it looks like the customer mis-configured the option 43 string and the IOS-XE DHCP server never worked.

For this to work in ACI:

1. a DHCP relay must be configured to relay the DHCP client packets
2. Contracts must allow DHCP and TFTP packets between the L3Out and the EPG the PXE client is in

You may see contract drops on the leaf switches for the TFTP packets using the command

calo2-leaf2# show logging ip access-list internal packet-log | grep <client ip address>

This command will change in version 2.0 to add the deny or permit keyword and becomes

calo2-leaf2# show logging ip access-list internal packet-log deny | grep <client ip address>

In the example case, the PXE client was not requesting option 66 but was specifying option 60 and requesting option 43 in the option 55 parameter list. It was also requesting next-server (option 12). Notice it requested the PXE filename via option 67.

The response shows the bootfile and option 67

TCPdump on leaf switch: