Mass Scale Deployment Using Zero Touch Onboarding White Paper

Abstract: In late 2019, several vendors were asked to participate in a discussion on creating a new network based on the latest 5G technologies, including a fully Open Radio Access Network (O-RAN) architecture. Dish Wireless started this conversation, and brought to it plans that were bold, aggressive, and, at the time, questionable. The current market was competitive, flooded with mobile service providers fighting for every customer. Dish Wireless had a vision of doing it differently and not being your average mobile service provider. Buzz phrases like “Fully Open RAN (Radio Access Network),” “Public/Private Cloud,” “Full Automation with a lean workforce,” and “network slicing at scale,” which had only been considered in the past, became visionary goals of a company ready to disrupt the existing market. Since then, Dish Wireless has navigated several obstacles on a bumpy, yet successful journey as they developed and built upon new products and processes to be used in innovative, exciting ways. This includes Zero Touch Onboarding, which combines several automation techniques to deliver a real-time process that takes a cell site from start to service-ready in a matter of hours, versus the days or weeks required with traditional processes. With Zero Touch Onboarding, Dish was able to create thousands of cell sites in several months and meet their aggressive goal.

Introduction

Once T-Mobile and Sprint merged in 2020 and Boost Mobile was acquired, the next milestone goal for Dish Wireless was to have 20 percent United States (U.S). Coverage no later than July 14, 2022, with 70 percent U.S. coverage no later than June 2023. But at the time, Dish Wireless was a greenfield network without an existing infrastructure. This meant that, to meet that goal of 20 percent coverage, Dish would have to install several thousand cell sites with radio and networking equipment while also hosting a user and control plane 5G packet core in a public cloud. Additionally, Dish Wireless could be perceived as a startup company. While some came from the traditional Dish Network part of the organization, much of their lean workforce was from outside of the company. Therefore, Dish Wireless would have to rely heavily on automation developed by Cisco^® and other vendors to meet those goals.

Traditional Experiences with Zero Touch Provisioning

Zero Touch Provisioning is not a new concept. Once connected to the network, many customers and vendors can use some traditional tools to call home and pull down a fully prepared configuration that could have been created days, or more likely weeks/months, ago. This is a workable solution for some large, embedded service providers with large workforces and mature management systems. However, that process does not suit a company like Dish Wireless that is new, fragmented, and has just-in-time deliverables. Also, many of the management systems were still being tested but had to be used for automation. Dish Wireless needed to develop a new process that began with the experiences and knowledge of the traditional zero touch implementations. By leveraging the power of DevOps and a lot of innovation, they created a highly automated, multivendor solution that moved past provisioning to make them service-ready through what is called Zero Touch Onboarding (ZTO).

Zero Touch Onboarding can be broken down into multiple areas of focus, each of which will be covered in this white paper.

1.     Planning and Standards Definition

2.     ZTO Preparation (pre-ZTO)

3.     Smart Discovery

4.     Real-Time Provisioning

5.     Automated Verification

6.     Administrative Onboarding

Planning and Standards Definition

Every device on the network obviously needs a hostname for identification. Yet it is difficult with manual processes to keep consistent naming standard across all device types and ensure all product owners use the same standard (APIs, backend databases [DBs], Tracking Systems, IP Address Manager [IPAM], Network Management System [NMS]).

But this practice is an integrated part of successful automation. Port mapping standards for each device type will make configuration template creation consistent and enable accurate application of a populated template.

ZTO Preparation

As with any technical solution, a certain bit of planning and preparation is needed. Imagine a situation where a low-skilled technician can install a router into a rack, cable and power that router, verify some lights, and walk away. This is what must be possible if a mass-scale deployment of thousands of routers is going to be done quickly at a low cost. So the question is, what preparation is needed so that this scenario can become a reality?

To scale across thousands of devices, there needs to be a template of configuration. However, in order to make a device unique in the network, there are specific variables required for that site. With both templates and variables defined, a workflow automation process is used to combine those objects to make a custom configuration solution for a particular cell site device. Using machine-to-machine APIs, the workflow automation tool can take an input of identifiers (in this case, cell site hostnames) and not only retrieve variables from the different databases, but also define and assign specific objects to those variables in real time. This enables the management system owners to define high-level pools of resources, while letting the workflow automation tool do the detailed work of assigning the custom information.

For example, an IPAM server may have a pool of IPs for a specific function, such as a pool of IPs for a specific VPN (Virtual Private Network) service in a particular market. The workflow automation tool can create a specific subnet from that pool, assign a default gateway address in that subnet, and create a set of DHCP addresses within that subnet. The process is replicated through APIs until all defined variables have been created and assigned. Using this information, the workflow automation tool can then prepare and configure the networking environment with the specific configuration needed when the specific router connects to the network. This is called the pre-ZTO process, and it can be done for a single device or for hundreds of devices per day.

In a multivendor environment, as shown in Figure 1, care should be taken with the scale of the workflow automation. The process can be resource-intense and could have adverse effects on computing and database systems. Thus, it is recommended that you always start slowly and ramp up your processes.

ZTO Preparation

Smart Discovery

Once the preparation of the environment and systems is complete, the network is ready to accept new devices. The pre-ZTO process and the start of the ZTO process can be done independently, but pre-ZTO for any particular device must be completed before ZTO can begin.

ZTO Smart Discovery

The ZTO process begins with the process ownership on the router that is powering on, but the ownership is then passed to an orchestration application like Cisco Network Services Orchestrator (NSO) to own and orchestrate the rest of the process. Once the device is connected and powered, it must be able to communicate with a connected network that can allow communications to a DHCP server. This may sound easy enough, but the details require some underlying investigation. Different scenarios must be addressed for different customers or use cases. These use cases consist of connecting with an out-of-band connection, an untagged physical interface, or a VLAN tagged virtual interface. Each of these scenarios is a possibility and requires specific attention. Once these use cases are defined, developed, tested, and implemented, the router is prepared so that any network connection into it, with the proper configuration, will kickstart the process.

The standard defining DHCP options (RFC2132) includes a vehicle (Option 67) to be able to download a specific text file or script for the host and use as it needs. Traditional ZTP solutions would download a prepared text configuration file already customized for the particular site. Due to the changing requirements and parameters for any particular site in their network, Dish Wireless needed a new paradigm. With Option 67, any text file can be downloaded, whether it is a configuration file or a script file. In this case, it is a Python script, which is specific device–agnostic and is downloaded to each router upon starting the ZTO process. This results in a highly scalable solution without the need for many text files, one for each of the thousand devices to be installed.

Using a Python script opens up more options of what the router can do. For the router to pass on ZTO ownership to Cisco NSO, a specific identifier unique to that device must be defined. This could be a MAC address, a serial number, an IP address, or anything that allows the orchestrator to specifically identify the device.

Thinking outside of the box, Dish has opted to attach a GPS antenna to each cell site router. The initial configuration (Day 0^[1]) downloaded to the router is the same configuration for all routers. It contains general configuration, of which is Global Navigation Satellite System (GNSS)-receiver configuration, turning on the connection to the GPS antenna. Once the GPS coordinates are established, the script then hands off ZTO ownership to the orchestrator using an API call, along with all its information, including the GPS coordinates. Knowing the GPS coordinates of all the planned cell sites, the orchestrator can now identify which site it is and begin creating the custom configuration in real time.

Real-Time Provisioning

In the past, configurations were built days, weeks, or even months ahead of the implementation. However, there may be use cases where the configuration needs to be built in real time. The data may not be ready to build the configuration, or the management systems may not be completely ready. In this case, the configuration needs to be built at the last possible moment to ensure that the data to populate the configuration is available and accurate. Therefore, instead of creating the configurations well ahead of time, there must be a process developed to create the configuration in real time before applying it to the specific device that needs it immediately.

To begin, proper solution testing must be completed. From that testing, a device golden configuration will be created, which will then be parameterized so that it is agnostic to any location. These variables are then defined in the orchestrator and told where the data for these variables are located. The data could be retrieved from an external management or inventory system, or it could be retrieved from a local or remote database. Once the orchestrator has a working template and all the locations of data needed to complete the configuration, then the process is ready.

ZTO Real-time Provisioning

As previously stated, when the device has finished its Smart Discovery and sent an API request to the orchestrator, along with a specific identifier, then the orchestrator can begin to create the permanent, unique, site-specific Day 1 configuration. Once the final configuration has been created, the orchestrator can then apply it to the device. When the configuration has been applied successfully and the orchestrator is satisfied that it has been committed, the orchestrator can verify that the device is running the proper software. If the device needs to be upgraded, the orchestrator will push the certified golden ISO or specific software packages to the device and initiate an upgrade. At any time in the software lifecycle of these devices, the certified software can be updated in a specific location, allowing all future ZTO devices to immediately start using the new software.

Once the software upgrade is complete, the orchestrator can then proceed to the next step in the process: Automated Verification.

Automated Verification

If up to this point everything has been successful, then the new device should be installed, configured, and ready to accept services. But how do you know? Also, a cell site is home to not only routers but also mobile radios, servers with RAN software, and environmental test equipment, among other things. A customer is not satisfied with a single device configuration. It must fit into a holistic, multivendor onboarding process that is truly an end-to-end product.

Logically, the next step is to verify the environment using automation. One way to verify the environment is to use a Cisco-supported Test Automation Framework (TAF). TAF is a framework that employs customized test cases or templates using an extensive list of open-sourced test products to generate test reports in an automated fashion. The TAF service began as a software-centric solution that initially focused on mobility software solutions but has now been expanded to include more network-type platform testing as well.

ZTO Auto Verification

Administrative Onboarding

The last step needed before services can be immediately added to the environment is to onboard the devices into the customer’s management tools. Using machine-to-machine APIs, all the information used in the creation of the unique configurations can now be added into the different management and inventory systems for use during service creation, including:

●      Inventory management

●      Performance management

●      Fault management

●      Customer databases

●      Parent orchestrator

If the environment includes other vendor workloads, a parent orchestrator may be used to take the information from the routing infrastructure to use for server or mobility workloads to continue building the environment.

Conclusion

With the innovation of Zero Touch Onboarding, the industry is moving from smart hands to smarter processes. Customers no longer want a single vendor solution; they are looking for something that can tie their multivendor environments together. By leveraging different layers of orchestration, the automation process can be modular, flexible, and managed by a lean workforce. This has beneficial use cases when mass scale deployment is needed in IoT (Internet of Things) or greenfield deployments, such as:

●      Large enterprise with many branches

●      Cellular service providers

●      Industrial environments