Published: April 2019
Cisco IT manages 30,000 Cisco Virtual Office routers in employees' home offices around the world. We make dozens of configuration changes every month--for security updates, operating system updates, new DNS or DHCP servers, new QoS policies, and so on.
Since 2006 we've automated configuration updates using custom scripts that push out the latest configuration file to 50-100 routers at a time. This worked well when we had 10,000 routers--but less well as the deployment grew.
"Pushing out a completely new file for every configuration change is like replacing every nail in a board instead of only the bent ones," says Joseph Bradley, Cisco IT senior engineer.
If the ninety-ninth of 100 routers failed to update, the configuration for all routers in the batch would sometimes revert to the previous state, and the script would try again the next day. In the case of a security update, the delay left the routers vulnerable.
Errors occurred often enough that eight engineers were needed to support Cisco Virtual Office. They spent 40-50 hours monthly testing configuration templates, another 5-6 hours updating router configurations, and still more time remediating errors.
We wanted a more efficient way to automate Cisco Virtual Office router configuration. Our goals included:
We automated Cisco Virtual Office router configuration and compliance checks using Cisco Network Services Orchestrator (NSO) and Python scripts. We did not have to use APIs to connect the scripts because Python support is built right into NSO. We deployed NSO in three theaters, starting with router configuration management in August 2018 and using it for Plug and Play provisioning in February 2019. Here's how it works.
Day 1: initial provisioning
When we ship out a router, someone at the depot enters its serial number, which is recorded in the NSO database and mapped to a router configuration in the Plug and Play Connect cloud (available from software.cisco.com).
"Our user just connects the router to the network and powers it on--and they're off to the races," Bradley says. The user can walk away at that point. The router sends its serial number to the Plug and Play cloud, which directs the request to the correct NSO server and pulls the current configuration. About 20 minutes later, the router is ready to use--down from 45 minutes before.
"Employees like it because now they don't have to sit in front of a web interface during the setup," says Ana.
Day 2: regular updates
Several times a week, NSO checks each router's configuration against its database to see if updates are needed. It does the same whenever a router is powered back on. If the configuration is out of date, a Python script pushes out just the needed changes--not the entire configuration file. If one router fails to update, the others in the group are not affected.
The entire router configuration lifecycle is now fully automated. We're continuing to fine-tune NSO and add new functions. One is automating changes that users have requested and Cisco IT has approved. Ana concludes, "In a matter of months, automating Cisco Virtual Office router configuration has improved security, reduced IT caseload, and improved our user experience. It's a win-win--for our users and for IT."
For more information