Contents

Introduction

Background Information

Linux Server Configuration

DHCP Configuration

HTTP Server Requirements

Troubleshooting Tips

Enable Router To Get IP Address With DHCP

Verify HTTP Server Is Functional

How To Start ZTP?

Manually Initiate ZTP

Boot The Router With iPXE

Post-Install Automation

Related Information

Introduction

This document describes the troubleshooting tips for first-time deployments of Zero Touch Provisioning (ZTP).

ZTP was introduced to reduce human interaction in the provision of XR devices. ZTP uses an implementation of the Preboot eXecution Environment (iPXE).

Background Information

ZTP can perform:

- Auto-Configuration update: Download and apply the XR configuration after image installation

- Script execution: Download and execute the user-defined script files. Various methods/functions can be implemented as a part of script execution:

- Multiple packages update

- SMU installation

- Verifications

- eXR decides which of the actions to perform based on the contents of the first line in the file

- If the file starts with !! IOS XR denotes a configuration file and performs apply_config.

- Similarly, the files that start with #! /bin/bash or #! /bin/sh or #!/usr/bin/python denotes a script file, and perform script execution.

- Support for python is introduced from eXR 6.2.2

- eXR iPXE supports Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), and Hypertext Transfer Protocol (HTTP).

- Hypertext Transfer Protocol Secure (HTTPS) is not supported as it cannot predict which signature to verify.

Linux Server Configuration

iPXE is an enhancement over PXE that requires TFTP/FTP/HTTP for image/configuration download and uses Dynamic Host Configuration Protocol (DHCP) to get/provide information for the image and configuration.

DHCP Configuration

In a later example, there is a review of a packet capture to confirm the DHCP operation.

HTTP Server Requirements

HTTP server must be reachable from the Mgmt Ethernet interface.

Troubleshooting Tips

When the Linux server is set up, Perform DHCP/HTTP server reachability and functionality Check.

In this setup, a single Linux server is used as DHCP/HTTP server. If there are separate servers for these functions, verify these steps on all the servers, if needed.

[root@xxxxxxxxxx]# service dhcpd status

Redirecting to /bin/systemctl status  dhcpd.service

dhcpd.service - DHCPv4 Server Daemon

Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled; vendor preset: disabled)

Active: active (running) since Mon 2017-05-29 10:30:59 PDT; 15h ago

Docs: man:dhcpd(8)

man:dhcpd.conf(5)

Main PID: 26913 (dhcpd)

Status: "Dispatching packets..."

CGroup: /system.slice/dhcpd.service

└─26913 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid

<SNIP>

Hint: Some lines were ellipsized, use -l to show in full.

[root@xxxxxxxx]# service httpd status

Redirecting to /bin/systemctl status  httpd.service

httpd.service - The Apache HTTP Server

Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)

Active: active (running) since Fri 2017-05-26 05:50:30 PDT; 3 days ago

Docs: man:httpd(8)

man:apachectl(8)

Process: 28088 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)

Process: 11036 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)

Main PID: 28095 (httpd)

Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"

CGroup: /system.slice/httpd.service

├─11037 /usr/sbin/httpd -DFOREGROUND

├─11038 /usr/sbin/httpd -DFOREGROUND

├─11039 /usr/sbin/httpd -DFOREGROUND

├─11040 /usr/sbin/httpd -DFOREGROUND

├─11041 /usr/sbin/httpd -DFOREGROUND

├─26998 /usr/sbin/httpd -DFOREGROUND

├─27426 /usr/sbin/httpd -DFOREGROUND

├─27427 /usr/sbin/httpd -DFOREGROUND

├─27428 /usr/sbin/httpd -DFOREGROUND

├─27889 /usr/sbin/httpd -DFOREGROUND

└─28095 /usr/sbin/httpd -DFOREGROUND

May 26 05:50:30 xxxxx systemd[1]: Starting The Apache HTTP Server...

May 26 05:50:30 xxxxx systemd[1]: Started The Apache HTTP Server.

May 27 03:16:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.

May 28 03:37:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.

Enable Router To Get IP Address With DHCP

Interface MgmtEth 0/RP0/CPU0/0

Ipv4 address dhcp

Shut/no shut

Verify HTTP Server Is Functional

- Test that http://<ip-servername> is works.

- Open a browser to the IP address/server name.

If DHCP or HTTP fails to work, there can be firewall issues or reachability problems.

To check the firewall properties on the server perform these commands or add the specific protocols:

Use IP tables command to verify firewall rules:

Iptables –L –n

Chain IN_public_allow (1 references)

target     prot opt source               destination

ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67 ctstate NEW

ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW

ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW

Use this firewall command to list the entries allowed:

[root@xxxxxxxxx ~]# firewall-cmd --list-all

public (active)

target: default

icmp-block-inversion: no

interfaces: enp2s0f0

sources:

services: dhcp dhcpv6-client http ssh

ports:

protocols:

masquerade: no

forward-ports:

sourceports:

icmp-blocks:

rich rules:

Use the commands to permanently allow ports:

firewall-cmd --permanent --add-service=http

firewall-cmd --permanent --add-service=dhcp

Enable Packet capture on DHCP server:

tcpdump –i <interface id> port (bootpc & bootps & port http) –s 0(buffer size) –w <dest. File name>

ex: tcpdump -i enp2s0f0 port 67 or port 68 or port 80 -s 0 -w iPXEboot.pcap

How To Start ZTP?

Manually Initiate ZTP

Use ZTP initiate exec CLI command to invoke ZTP.  By default, the interface used for ZTP is MGMTEth. To initiate this process on other interfaces use this option:

ZTP initiate interface <type> <number> <cr>

OR

Ztp initiate <cr>

Boot The Router With iPXE

1. From Calvados, initiate the reload CLI command:

This command causes a router to reload, which initiates the iPXE boot.

Note: User-class information is interpreted as a Malformed option due to a bug in Wireshark.

2. The router initiates DHCP Discover, notice that user-class is populated with iPXE option:

3. The DHCP Offer from the server includes the boot file name in Option 67:

4. The router initiates the image download:

5. Once the image is successfully downloaded, the installation of the images on the device begins.

6. Once the router successfully boots with the downloaded image, it initiates another DHCP request:

In this Discover, notice that user-class info includes *.exr-config. Since the DHCP is configured to either return Config file or Script (ie. Else statement in etc/dhcp/dhcpd.conf).

7. DHCP server returns the required file information in option 67:

8. The router downloads the config:

The script can be included as a response from the DHCP server and have them copy the image/configuration.  This can also be used as a post-install script like in the below example.

Post-Install Automation

Various automation can be performed post-installation on the eXR software.

In this example, this script installs all required packages and applies the Config. This is a slightly modified version of ztp_helper.sh.  This script is invoked from xr-linux-shell:

RP/0/RP0/CPU0:NCS-5502-A#more disk0:/ztp/ztp_helper_file.sh

Wed May 31 00:55:54.529 UTC

#!/bin/bash

################################################################################

# Install config and additional packages

################################################################################

source /disk0:/ztp/ztp_helper.sh

export HTTP_SERVER=http://10.10.10.10

export RPM_PATH=images

export CONFIG_PATH=images

#Config

export INITIAL_CONFIG=NCS-5502-A.cfg

export FINAL_CONFIG=NCS-5502-A.cfg

#Packages

K9SEC_RPM=ncs5500-k9sec-2.2.0.0-r612.x86_64.rpm

MCAST_RPM=ncs5500-mcast-2.0.0.0-r612.x86_64.rpm

ISIS_RPM=ncs5500-isis-1.1.0.0-r612.x86_64.rpm

OSPF_RPM=ncs5500-ospf-1.1.0.0-r612.x86_64.rpm

MGBL_RPM=ncs5500-mgbl-3.0.0.0-r612.x86_64.rpm

MPLS_RPM=ncs5500-mpls-2.1.0.0-r612.x86_64.rpm

MPLSTE_RPM=ncs5500-mpls-te-rsvp-2.2.0.0-r612.x86_64.rpm

function download_config(){

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${CONFIG_PATH}/${FINAL_CONFIG} -O /harddisk:/new-config 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading system configuration ###"

else

printf "### Downloading system configuration complete ###";

fi

}

function apply_config(){

# Applies initial configuration

printf "### Applying initial system configuration ###";

xrapply_with_reason "Initial ZTP configuration" /harddisk:/new-config 2>&1;

printf "### Checking for errors ###";

local config_status=$(xrcmd "show configuration failed");

if [[ $config_status ]]; then

echo $config_status

printf "!!! Error encounter applying configuration file, review the log !!!!";

fi

printf "### Applying system configuration complete ###";

}

function install_pkg(){

#Download packages

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${K9SEC_RPM} -O /harddisk:/$K9SEC_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $K9SEC_RPM ###"

else

printf "### Downloading $K9SEC_PKG complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MCAST_RPM} -O /harddisk:/$MCAST_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $MCAST_RPM ###"

else

printf "### Downloading $MCAST_RPM complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${ISIS_RPM} -O /harddisk:/$ISIS_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $ISIS_RPM ###"

else

printf "### Downloading $ISIS_RPM complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${OSPF_RPM} -O /harddisk:/$OSPF_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $OSPF_RPM ###"

else

printf "### Downloading $OSPF_RPM complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MGBL_RPM} -O /harddisk:/$MGBL_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $MGBL_RPM ###"

else

printf "### Downloading $MGBL_RPM complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLS_RPM} -O /harddisk:/$MPLS_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $MPLS_RPM ###"

else

printf "### Downloading $MPLS_RPM complete ###";

fi

printf "Downloading Packages"

ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLSTE_RPM} -O /harddisk:/$MPLSTE_RPM 2>&1

if [[ "$?" != 0 ]]; then

printf "### Error downloading $MPLSTE_RPM ###"

else

printf "### Downloading $MPLSTE_RPM complete ###";

fi

xrcmd "install update source /harddisk:/ $K9SEC_RPM $MCAST_RPM $ISIS_RPM $OSPF_RPM $MGBL_RPM $MPLS_RPM $MPLSTE_RPM" 2>&1

local complete=0

while [ "$complete" = 0 ]; do

complete=`xrcmd "show install active" | grep k9sec | head -n1 | wc -l`

printf "Waiting for k9sec package to be activated"

sleep 5

done

rm -f /harddisk:/$K9SEC_RPM /harddisk:/$MCAST_RPM /harddisk:/$MCAST_RPM /harddisk:/$ISIS_RPM /harddisk:/$OSPF_RPM /harddisk:/$MGBL_RPM /harddisk:/$MPLSTE_RPM /harddisk:/$MPLS_RPM

printf "### XR PACKAGE INSTALL COMPLETE ###"

}

printf "Start Auto provision"

install_pkg;

download_config;

apply_config;

Related Information

- https://xrdocs.github.io/software-management/tutorials/2016-08-26-working-with-ztp/

- https://xrdocs.github.io/software-management/tutorials/2016-07-27-ipxe-deep-dive/

- https://xrdocs.github.io/software-management/blogs/2016-10-14-ios-xr-packages-and-security/