Cisco IT Nearing Final Phase of ACI Journey

What

Cisco IT has come a long way in our Cisco Application Centric Infrastructure (ACI) journey.  ACI is the next-generation SDN platform that is application-aware and capable of supporting physical, virtual, and cloud integration with centralized management. We have already deployed ACI in all our data centers around the world that host Cisco key business applications. Our latest migration milestone—moving our very large Platform as a Service (PaaS) application development and Dev/Ops private cloud to our ACI fabric—is a major step forward to our bringing greater automation, security, ease of use and cost-savings to our IT data center infrastructure through open source and open APIs. As we deploy more applications, we continue to gain more skill and the ability to automate more migration functions, so our migration continues to accelerate.

“Our teams have been working diligently to migrate groups of our business applications to the Cisco ACI platform,” says Anitha Parimi, Cisco IT Principle Engineer.  “Moving an application to ACI can take hours or days, depending on the size and complexity of the application. We have been improving our processes all along the way, applying both lessons learned and automation to help accelerate ACI migration. For example, we are now able to perform bulk migrations for larger groups of applications.”

Why

Cisco IT chose to migrate all our business data centers to ACI for five simple reasons:  greater speed and agility, reduced cost, and greater simplicity, security, and scalability.  We are well on our way to achieving all of these, and we will continue to share our results and best practices as we complete our migration.

Cisco ACI migration is essential to achieving our vision of a “fast” IT environment.  The centrally controlled ACI platform separates network identity from the physical infrastructure and automates policy to meet application needs, which simplifies and speeds our deployment and management of applications at scale within our existing data center. ACI simplifies operations by automating layers of configuration, providing application developers with a self-service portal to more easily provision application-specific infrastructure services that Cisco IT once handled manually, further reducing operating expenses.  Application provisioning today is about 67% faster, and provisioning network services like load balancers a little faster (about 75%). Time to resolve issues in the ACI data center is about 64% faster. 

Our ACI migration reduces the cost of our data centers.  Networking capital costs for our new ACI data centers has dropped about 67% (although this reduction was only 17% in a very small data center, so size makes a difference).   ACI also improves existing resource utilization for servers, storage, space and power. The ACI fabric enables greater speed and scalability, pooling all ACI data center infrastructure together to enable better infrastructure utilization, reducing stranded resources and overall infrastructure costs. Server and storage utilization is improved by the single data-center-wide VXLAN fabric and converged infrastructure by about 20%.  Resource space requirements (“footprint”) in the data center are reduced by about 66%, and power and cooling requirements are also significantly reduced.   

ACI simplifies our lives, too.  With ACI, Cisco IT can specify our data center infrastructure services once, through policy, and let application developers specify their application needs once, across multiple data centers, and let the ACI fabric support each application accordingly. The centralized Application Policy Infrastructure Controller (APIC) pushes out and monitors application policy and performance end to end from that moment on.  This automation and self-service reduces operational time and error, and greatly reduces application deployment times. As we automate that process and make use of application profile templates, this reduction in time (currently about 67%) is increasing.  IT operations can make changes to the underlying infrastructure for upgrades or repair with far less application downtime, improving business resilience.  Upgrading servers requires no more than moving virtual machines to any other spare compute resource in the data center while we change out equipment; . The end-to-end view provided by the APIC also simplifies and reduces the time it takes to troubleshoot application and infrastructure issues, further reducing operating expenses.  

Finally, our ACI migration is improving the security and the scalability of our existing data centers. ACI policies and micro-segmentation enables far greater granular segmentation security and control.  Cisco IT chose to implement our ACI as “zero-trust”, using allowed list policies, which stops attacks from spreading across multiple applications or tenants. We are also migrating applications using application policy contracts that allow only the necessary application inter-communication required by the application, providing more barriers to the spread of malware in the data center. Also, our existing spine-leaf ACI architecture allows Cisco IT to continue to scale the network infrastructure horizontally to more than double its current size.

Cisco ACI Migration Process: Overview

Figure 1 outlines the key steps for moving applications from traditional network infrastructure to the ACI fabric.

 

In late 2015, we started the one-time infrastructure migration tasks by building a simple and scalable ACI fabric in our existing data centers based on Cisco Nexus 9000 Series Switches, in parallel with our traditional data center network. We then installed “seed” compute and “seed” storage into the ACI fabric, and then connected the ACI fabric to the traditional network using load-balanced gateways between the existing data center core switch and a pair of leaf switches in the new ACI fabric.

During FY16 and FY17, the key steps we have focused on include:

·       Application dependency mapping: Most of Cisco IT’s applications are multi-tier, and a typical application may have one or more virtual machines providing presentation services, application processing, and data management functions that connect to one or more databases. Overlooking a single inter-dependency among these virtual machines could result in the application not working properly after migration to the ACI platform. ACI defines each application according to application policies that includes a secure connection between the application and its necessary infrastructure services as well as with other applications. That’s why knowing each application’s set of interdependencies is critical to building an ACI solution. 

Cisco IT began using the Cisco Tetration Analytics solution for application dependency mapping and endpoint grouping, and other tasks, in 2016. Tetration delivers behavior-based application insight with deep forensics, providing us with real-time visibility into traffic across all Cisco data centers as well as a far better-detailed understanding of dependencies among applications. Tetration Analytics enabled us to identify the best candidates for early migration (less complex, fewer interdependencies), and more.

·       Configuring endpoint groups (EPGs), contracts, and more: Cisco IT uses the Tetration Analytics data to build endpoint groups—sets of devices that connect to the network directly (like servers) or indirectly (like web clients)—which greatly simplifies managing the vast number of connected devices. The interdependency mapping from Tetration Analytics shows Cisco IT how to build secure application policies that enable applications to interact using an allowed list. Only required interactions are enabled by ACI; all other interactions are blocked. This set of allowed list policies, along with the services contracts for firewall, load balancing, or other network services, make up the full application service policy. The allowed list policy provides an unprecedented level of security within the ACI data center and replaces thousands of lines of hard-to-manage access control lists (ACLs), saving significant operations time and improving reliability and security.   

·       Moving applications to the ACI platform and prioritizing other applications for migration: Our ACI migration strategy was to migrate multi-tier infrastructure as a service (MT-IaaS) applications before tackling more complex platforms. “Cisco IT staff had to confirm each application was stable in the ACI environment before moving on to the next application,” says Carol Goh, Cisco IT Network Services Director.  “Often, they faced workload resource timing and capacity issues and had to rethink their timing for moving certain applications.”

Figure 2 is an overview of Cisco IT’s road map for ACI migration during FY2017:

 

 

 

ACI Migration Milestones

As of August 2017, more than 5000 hosts and 100 applications were running on Cisco ACI-enabled networks. Migrations include:

●    Big data and businesses intelligence platforms like Hadoop, SAP Hana, and SAP Business Objects.

●    Continuous delivery platforms such as uDeploy and uRelease.

●    Large enterprise resource planning (ERP) assets like Cisco Customer Care and the Cisco Sales Commission database.

●    Large-scale cloud platforms, including our private cloud containers for PaaS based on Openshift 2.0 (about 700 applications) and the identity authentication tool, OneID (greenfield).

Specific ACI-related achievements by Cisco IT in FY16 and FY17 include:

●    Installation of Tetration Analytics in Cisco IT data centers located at our San Jose and Mountain View, CA, campuses, and our Research Triangle Park, NC, (RTP) campus.

●    Tetration Analytics 2.0 upgrade at our RTP data center.

●    MT-IaaS migrations at our RTP and ALLN (Allen, TX) data centers.

●    Smaller data center infrastructure build (2 border leaves, 4 spines, 4 compute leaves, and 1 APIC cluster) in our European data center near Amsterdam —completed in less than 4 weeks, followed by full ACI migration of all applications in the following 4 weeks.

●    Migration of all MT-IaaS clients and BRMS (business rule management systems) to the Amsterdam ACI network.

●    Refresh of UCS Fabric Interconnect hardware at the ALLN data center, which required 29 consecutive hours of execution.

●    Congo 2.0(2n) upgrade in almost all ACI fabric data centers, giving those assets the latest advanced security features.

●    Completion of Informatica production and disaster recovery migrations.

 

And finally, in June 2017, Cisco IT completed the migration of our PaaS private cloud to ACI. This effort included the provisioning of 866 new virtual hosts via our Application Centric Cloud (ACC) front end, and the migration of more than 14,000 application instances from production and non-production (eg. dev and test applications) to ACI.  ACC enables Cisco application developers to deploy their new applications automatically and directly into ACI, while Cisco IT works to migrate existing applications to ACI.

Benefits of ACI Migration

Following are just some of the benefits that Cisco has realized at this point in our ACI journey so far:

●    Dramatically increase network line speeds: from 10 Gbps to 40Gbps and soon 100 Gbps

●    Network latency improvement: 30 percent

●    Data processing improvement: up to 34 percent

●    Reduction of application provisioning time by 67 percent (to date)

●    Reduction of total capital expenses in the data center of 25 percent due to better utilization of server and storage resources and less expensive switching

●    Reduction of overall data center management costs by 21 percent (to date)

●    Estimated final total reduction in data center TCO of more than 40 percent

●    Institution of policy-based security, resulting in far greater allowed list security and, for one application, a reduction of more than 2000 ACL entries to 10 policy contracts

●    Migration automation: 1 application, 3 hosts, and 3 lifecycles in 10 minutes

Cost Savings for ALLN Data Center and AMS Data Center

We recently migrated our entire (rather small) production data center in Amsterdam.  This gave us a chance to compare the impact of ACI on a small data center versus the far larger one in Allen Texas.  We also reviewed our earlier pre-migration projections of ACI cost savings to our more recent measurements in Allen, and found our earlier projections were far too conservative.  We found that our savings are much higher than our projections for this large data center. For example:

●    We projected a reduction in application time-to-provision of 58 percent, but have so far realized far more:  67 percent savings.  This improvement is in large part due to our added automation of the APIC user interface with our Application Centric Cloud provisioning tool, and with this and other automation tools we are seeing our provisioning times continue to accelerate.  Even our smaller data center saw 54 percent.

●    We projected capital expenditure savings of 25 percent, but have so far realized far more:  67 percent savings.  (In our smaller data center this savings was only 17 percent, so size clearly makes a difference in capital savings.)

●    We projected 45 percent reduction in network, power and cooling costs, but have realized a  91 percent reduction;. Our smaller data center had similar results, savings of 83 percent.

●    We anticipated a drop in the number and data center footprint of our networking infrastructure based on our move to ACI.  We had thought we’d see about a 19% reduction in data center footprint for our ACI networking gear, but have seen a 66 percent reduction. In our smaller data center, we saw a similar but slightly smaller reduction of 55 percent. 

●    We estimated reaching a 12 percent increase in resource optimization, but have so far measured a 20 percent increase.

 

We will compare our earlier projections for management costs reductions (21 percent) once network automation features at the ALLN data center have been in use long enough for us to measure performance.

ACI Migration Fueling Cultural Transformation at Cisco

One of the most significant benefits to Cisco, in parallel with our ACI migration is the cultural change that it has helped to facilitate within Cisco IT. To achieve our goals for ACI, we recognized early the need to embrace Agile development techniques to accelerate our entire work process. Adoption of the Agile methodology is a broader trend in the industry, so it was important for Cisco IT to make this move to keep pace with transformative forces in IT like digitization and virtualization.

With more PaaS automation, Cisco IT is now able to provide full continuous delivery application development platforms, and application developers have been taking continuous delivery and DevOps to new heights within Cisco IT. These developers are building as many new Cisco business applications as possible using a “cloud native” architecture—applications composed of reusable microservices within containers—which allows applications to make full use of cloud capabilities to automatically grow, shrink, or relocate their components for lower costs, higher reliability, and more.  It was this entire continuous delivery cloud and all the business applications developed within it, based on a Platform as a Service environment, that we most recently migrated to ACI.

Since we have moved to continuous delivery and DevOps practices, we have broken down traditional “silos” within our IT department. That means our specialists can now expand the breadth of their knowledge to other domain areas. The results include better collaboration, faster iteration, and more innovation within Cisco IT—including on projects other than Cisco ACI migration.

Next Steps

Cisco IT also recently achieved another significant milestone in our ACI journey: the first phase of migration of thousands of compute and storage units from traditional network fabric to the ACI platform. “As part of our ACI migration journey, we embrace network programmability for automation.  This accelerates the speed of migration.” says Su Tsai, Cisco IT Sr. Manager ACI Network Services. “We expect to reach more than 50 percent of our planned host migration scope by the end of FY18 by combining automation and an innovative POD migration approach.”

For More Information

ACI Whitepapers - https://www.cisco.com/c/en/us/products/cloud-systems-management/application-policy-infrastructure-controller-apic/white-paper-listing.html

Which include:

●    Cisco IT ACI Design - https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Design.html

●    Cisco IT Migration to ACI - https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_Migration_to_ACI.html

●    Cisco IT ACI Storage Deployment – https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Storage_Deployment.html

●    Cisco IT ACI Compute at Scale - https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Compute.html

 

And for more content of interest, see Cisco IT Data Center and Cloud - https://www.cisco.com/c/en/us/solutions/cisco-on-cisco/data-center.html

To read additional Cisco IT case studies about a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT.

To view Cisco IT webinars and events about related topics, visit Cisco on Cisco Webinars & Events.

Note

This publication describes how Cisco has benefited from the deployment of its own products. Many factors may have contributed to the results and benefits described. Cisco does not guarantee comparable results elsewhere.

CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Some jurisdictions do not allow disclaimer of express or implied warranties; therefore, this disclaimer may not apply to you.