Zero Touch Provisioning Implementation and Troubleshooting Tips

Introduction

This document describes the troubleshooting tips for first-time deployments of Zero Touch Provisioning (ZTP).

ZTP was introduced to reduce human interaction in the provision of XR devices. ZTP uses an implementation of the Preboot eXecution Environment (iPXE).

Background Information

ZTP can perform: 

• Auto-Configuration update: Download and apply the XR configuration after image installation
• Script execution: Download and execute the user-defined script files. Various methods/functions can be implemented as a part of script execution:
  • Multiple packages update
  • SMU installation
  • Verifications
  • eXR decides which of the actions to perform based on the contents of the first line in the file
    • If the file starts with !! IOS XR denotes a configuration file and performs apply_config.
    • Similarly, the files that start with #! /bin/bash or #! /bin/sh or #!/usr/bin/python denotes a script file, and perform script execution.
  • Support for python is introduced from eXR 6.2.2
  • eXR iPXE supports Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), and Hypertext Transfer Protocol (HTTP). 
  • Hypertext Transfer Protocol Secure (HTTPS) is not supported as it cannot predict which signature to verify.

Linux Server Configuration

iPXE is an enhancement over PXE that requires TFTP/FTP/HTTP for image/configuration download and uses Dynamic Host Configuration Protocol (DHCP) to get/provide information for the image and configuration.

DHCP Configuration

In a later example, there is a review of a packet capture to confirm the DHCP operation.

HTTP Server Requirements

HTTP server must be reachable from the Mgmt Ethernet interface.

Troubleshooting Tips

When the Linux server is set up, Perform DHCP/HTTP server reachability and functionality Check.

In this setup, a single Linux server is used as DHCP/HTTP server. If there are separate servers for these functions, verify these steps on all the servers, if needed.

[root@xxxxxxxxxx]# service dhcpd status
Redirecting to /bin/systemctl status  dhcpd.service
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-05-29 10:30:59 PDT; 15h ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
 Main PID: 26913 (dhcpd)
   Status: "Dispatching packets..."
   CGroup: /system.slice/dhcpd.service
           └─26913 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid
<SNIP>

Hint: Some lines were ellipsized, use -l to show in full.

[root@xxxxxxxx]# service httpd status
Redirecting to /bin/systemctl status  httpd.service
   httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-05-26 05:50:30 PDT; 3 days ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 28088 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 11036 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 28095 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─11037 /usr/sbin/httpd -DFOREGROUND
           ├─11038 /usr/sbin/httpd -DFOREGROUND
           ├─11039 /usr/sbin/httpd -DFOREGROUND
           ├─11040 /usr/sbin/httpd -DFOREGROUND
           ├─11041 /usr/sbin/httpd -DFOREGROUND
           ├─26998 /usr/sbin/httpd -DFOREGROUND
           ├─27426 /usr/sbin/httpd -DFOREGROUND
           ├─27427 /usr/sbin/httpd -DFOREGROUND
           ├─27428 /usr/sbin/httpd -DFOREGROUND
           ├─27889 /usr/sbin/httpd -DFOREGROUND
           └─28095 /usr/sbin/httpd -DFOREGROUND

May 26 05:50:30 xxxxx systemd[1]: Starting The Apache HTTP Server...
May 26 05:50:30 xxxxx systemd[1]: Started The Apache HTTP Server.
May 27 03:16:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.
May 28 03:37:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.

Enable Router To Get IP Address With DHCP

Interface MgmtEth 0/RP0/CPU0/0
Ipv4 address dhcp
Shut/no shut

Verify HTTP Server Is Functional

• Test that http://<ip-servername> is works.
• Open a browser to the IP address/server name.

If DHCP or HTTP fails to work, there can be firewall issues or reachability problems.

To check the firewall properties on the server perform these commands or add the specific protocols:

Use IP tables command to verify firewall rules:

Iptables –L –n
Chain IN_public_allow (1 references)
target     prot opt source               destination        
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW

Use this firewall command to list the entries allowed:

[root@xxxxxxxxx ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp2s0f0
  sources:
  services: dhcp dhcpv6-client http ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  sourceports:
  icmp-blocks:
  rich rules:

Use the commands to permanently allow ports:

firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=dhcp

Enable Packet capture on DHCP server:

tcpdump –i <interface id> port (bootpc & bootps & port http) –s 0(buffer size) –w <dest. File name>

ex: tcpdump -i enp2s0f0 port 67 or port 68 or port 80 -s 0 -w iPXEboot.pcap 

How To Start ZTP?

Manually Initiate ZTP

Use ZTP initiate exec CLI command to invoke ZTP.  By default, the interface used for ZTP is MGMTEth. To initiate this process on other interfaces use this option:

ZTP initiate interface <type> <number> <cr>

OR

Ztp initiate <cr>

Boot The Router With iPXE

1. From Calvados, initiate the reload CLI command:

This command causes a router to reload, which initiates the iPXE boot.

Note: User-class information is interpreted as a Malformed option due to a bug in Wireshark. 

2. The router initiates DHCP Discover, notice that user-class is populated with iPXE option:

3. The DHCP Offer from the server includes the boot file name in Option 67:

4. The router initiates the image download:

5. Once the image is successfully downloaded, the installation of the images on the device begins. 

6. Once the router successfully boots with the downloaded image, it initiates another DHCP request:

In this Discover, notice that user-class info includes *.exr-config. Since the DHCP is configured to either return Config file or Script (ie. Else statement in etc/dhcp/dhcpd.conf).

7. DHCP server returns the required file information in option 67:

8. The router downloads the config:

The script can be included as a response from the DHCP server and have them copy the image/configuration.  This can also be used as a post-install script like in the below example.

Post-Install Automation

Various automation can be performed post-installation on the eXR software.

In this example, this script installs all required packages and applies the Config. This is a slightly modified version of ztp_helper.sh.  This script is invoked from xr-linux-shell:

RP/0/RP0/CPU0:NCS-5502-A#more disk0:/ztp/ztp_helper_file.sh
Wed May 31 00:55:54.529 UTC
#!/bin/bash
################################################################################
# Install config and additional packages
################################################################################

source /disk0:/ztp/ztp_helper.sh

export HTTP_SERVER=http://10.10.10.10
export RPM_PATH=images
export CONFIG_PATH=images
#Config
export INITIAL_CONFIG=NCS-5502-A.cfg
export FINAL_CONFIG=NCS-5502-A.cfg

#Packages
K9SEC_RPM=ncs5500-k9sec-2.2.0.0-r612.x86_64.rpm
MCAST_RPM=ncs5500-mcast-2.0.0.0-r612.x86_64.rpm
ISIS_RPM=ncs5500-isis-1.1.0.0-r612.x86_64.rpm
OSPF_RPM=ncs5500-ospf-1.1.0.0-r612.x86_64.rpm
MGBL_RPM=ncs5500-mgbl-3.0.0.0-r612.x86_64.rpm
MPLS_RPM=ncs5500-mpls-2.1.0.0-r612.x86_64.rpm
MPLSTE_RPM=ncs5500-mpls-te-rsvp-2.2.0.0-r612.x86_64.rpm

function download_config(){
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${CONFIG_PATH}/${FINAL_CONFIG} -O /harddisk:/new-config 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading system configuration ###"
    else
        printf "### Downloading system configuration complete ###";
    fi
}

function apply_config(){
        # Applies initial configuration
    printf "### Applying initial system configuration ###";
    xrapply_with_reason "Initial ZTP configuration" /harddisk:/new-config 2>&1;
    printf "### Checking for errors ###";
    local config_status=$(xrcmd "show configuration failed");
    if [[ $config_status ]]; then
        echo $config_status  
        printf "!!! Error encounter applying configuration file, review the log !!!!";
    fi
    printf "### Applying system configuration complete ###";
}

function install_pkg(){
        #Download packages
        printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${K9SEC_RPM} -O /harddisk:/$K9SEC_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $K9SEC_RPM ###"
    else
        printf "### Downloading $K9SEC_PKG complete ###";
    fi
    
    printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MCAST_RPM} -O /harddisk:/$MCAST_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $MCAST_RPM ###"
    else
        printf "### Downloading $MCAST_RPM complete ###";
    fi
  
    printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${ISIS_RPM} -O /harddisk:/$ISIS_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $ISIS_RPM ###"
    else
        printf "### Downloading $ISIS_RPM complete ###";
    fi    
    
    printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${OSPF_RPM} -O /harddisk:/$OSPF_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $OSPF_RPM ###"
    else
        printf "### Downloading $OSPF_RPM complete ###";
    fi
    printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MGBL_RPM} -O /harddisk:/$MGBL_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $MGBL_RPM ###"
    else
        printf "### Downloading $MGBL_RPM complete ###";
    fi   
   
     printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLS_RPM} -O /harddisk:/$MPLS_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $MPLS_RPM ###"
    else
        printf "### Downloading $MPLS_RPM complete ###";
    fi   
      printf "Downloading Packages"
        ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLSTE_RPM} -O /harddisk:/$MPLSTE_RPM 2>&1
        if [[ "$?" != 0 ]]; then
        printf "### Error downloading $MPLSTE_RPM ###"
    else
        printf "### Downloading $MPLSTE_RPM complete ###";
    fi    
    
    xrcmd "install update source /harddisk:/ $K9SEC_RPM $MCAST_RPM $ISIS_RPM $OSPF_RPM $MGBL_RPM $MPLS_RPM $MPLSTE_RPM" 2>&1 
  local complete=0
  while [ "$complete" = 0 ]; do
        complete=`xrcmd "show install active" | grep k9sec | head -n1 | wc -l`
        printf "Waiting for k9sec package to be activated"
        sleep 5
    done
    rm -f /harddisk:/$K9SEC_RPM /harddisk:/$MCAST_RPM /harddisk:/$MCAST_RPM /harddisk:/$ISIS_RPM /harddisk:/$OSPF_RPM /harddisk:/$MGBL_RPM /harddisk:/$MPLSTE_RPM /harddisk:/$MPLS_RPM
    printf "### XR PACKAGE INSTALL COMPLETE ###"
}

printf "Start Auto provision"
install_pkg;
download_config;
apply_config;

Related Information

• https://xrdocs.github.io/software-management/tutorials/2016-08-26-working-with-ztp/
• https://xrdocs.github.io/software-management/tutorials/2016-07-27-ipxe-deep-dive/
• https://xrdocs.github.io/software-management/blogs/2016-10-14-ios-xr-packages-and-security/