Published: February, 2020
Cisco has more than 1500 computer labs around the world to test new products and provide customer support for a very wide range of possible IT combinations. The compute, storage, and networking infrastructure in these labs must be ready to support a fully functional, production-like environment that is flexible, easily extendable, and can quickly mirror almost any customer configuration. But configuring a complete infrastructure for each testing or support scenario can take days, even for a knowledgeable engineer.
Cisco IT needed a way to create ready-to-go infrastructure resources that can be configured with templates and allocated as needed for specific projects in our labs. By using the Cisco HyperFlex HX-Series systems, we have automated much of this configuration activity in a Cisco Technical Assistance Center (TAC) lab. This automation reduces the time to recreate a complex customer environment from 120 hours to less than one hour, which speeds resolution of support cases.
Traditional lab operations make it difficult to standardize infrastructure resources and configuration tasks. Specific challenges include:
To address these challenges and needs for lab testing, the Cisco IT team now uses Cisco HyperFlex systems to create a dynamic server and storage infrastructure.
The Cisco HyperFlex systems support multiple hypervisors and combine Cisco Unified Computing System™ (Cisco UCS®) servers for computing with software-defined storage and networking. A single infrastructure engineer manages the Cisco HyperFlex systems from the cloud, using Cisco Intersight™ software as a service management platform.
Cisco HyperFlex supports fast and cost-effective infrastructure deployment and management, allowing us to maintain a stable environment while supporting frequent and temporary reconfigurations of that infrastructure. This flexibility allows us to offer a “testing lab as a service” to our Cisco TAC support engineers.
The initial lab infrastructure supports the Cisco TAC Americas collaboration support team, which includes nearly 400 engineers. This team needs flexible infrastructure to recreate customer environments, test and troubleshoot product functionality, and deliver training to TAC, partner, and customer engineers.
For example, one TAC team needed to re-create a particular customer environment that had about 60 servers with many virtual machines and terabytes of data. Yet TAC engineers don’t have the expertise to do this provisioning themselves, especially when the mix of elements and configurations is different for every customer.
By using Cisco HyperFlex with Cisco Intersight automation, we’re able to easily and quickly create those unique environments on-demand. And with the monitoring and configuration visibility provided by Cisco Intersight, the infrastructure engineer can focus on supporting the customer project, not on maintaining the servers and storage resources.
As this infrastructure has proven its value for complex Cisco TAC support activity, the Cisco HyperFlex cluster has grown from 8 nodes initially to 32 nodes today. For the TAC lab, the Cisco HyperFlex infrastructure supports up to 3600 virtual machines (VMs), with up to 300 VMs per host and over 120 TB of storage capacity. These resources can be allocated as needed for one-to-many testing projects and, when no longer needed, released for reconfiguration to use in another project.
The lab infrastructure allows TAC engineers to use a defined set of service-profile templates to create a particular type of environment for planning and testing activity. The policies and profiles are maintained in Intersight. (Figure 1) These profiles have been designed and configured by product subject-matter experts to incorporate best practices and typical customer deployments. Intersight works with HyperFlex Connect to quickly configure and deploy the HyperFlex infrastructure. Although these profiles won’t perfectly mirror a particular customer’s configuration, they are extremely close, and we expect them to cover the majority of TAC troubleshooting and testing needs.
Cisco IT has already realized several benefits from the dynamic infrastructure created with Cisco HyperFlex for our lab testing.
Improved customer experience. Cisco support engineers can spend more time focused on solving problems instead of configuring and maintaining a lab infrastructure.
Simple lab creation. Reduced lab complexity and an automated configuration process mean fast and easy reconfiguration of the lab infrastructure for different customer environments and testing needs.
Easy access. The lab infrastructure can be leveraged by automated processes and is globally accessible and extendable, making it easy to use by TAC engineers worldwide.
Realistic training. The Cisco TAC lab infrastructure can be used to deliver training and a working test environment for Cisco and partner support engineers.
Insights for product development. By working with a similar lab infrastructure, the TAC teams can give helpful input to the development teams about product testing.
Given its power, flexibility, and simplicity, the Cisco HyperFlex lab infrastructure has become a full contributor to our Cisco TAC Americas collaboration team operations. We expect to continue growing the environment to support more TAC teams and regions, as well as for more purposes, such as validating proposed network designs.