Nearly 1200 Cisco IT applications are considered critical for continuity of Cisco® business operations. Many of these applications normally run in our mirrored data centers in Richardson and Allen, Texas. If a major disaster simultaneously disrupted both of these facilities, we would activate those applications in our recovery data center located in Research Triangle Park, North Carolina.
As part of this business continuity plan, we perform resiliency tests to verify that applications can be recovered quickly and completely in the backup data center. To test such a large number of applications, we have made several improvements to our resiliency testing program.
No production downtime. “All disaster recovery resiliency testing is solely contained within the disaster recovery data center without impacting production applications,” says Sal Pearce, Cisco IT resiliency manager. It also means we can conduct testing during normal work hours instead of scheduling a downtime window at night or on the weekend.
Large-group testing. Previously, we tested applications either individually or in small batches when deemed ready by the application teams. This approach was inefficient because the same amount of testing preparation and activity was required no matter the number of applications to be validated. Additionally, application teams weren’t always required to commit to a testing date. We now test a large number of applications on a specified date. This approach significantly reduces the preparation time and effort for both the application and testing teams and has helped us accelerate the overall testing schedule. In an initial large-group test, we were able to measure the resilience of 77 applications in the recovery data center.
Designated Resiliency Primes. A primary contact person (called the Resiliency Prime) works with the applications teams to support their preparation for the resiliency testing process. The Prime helps the team develop a schedule and gather the application information needed to identify the required infrastructure and scope for testing. “The Primes are able to accelerate adoption of our resiliency goals by being proactive in reaching out to the application teams, capturing key information and supporting their participation in the testing,” says Pearce.
Streamlined process. We consolidated and automated activities to reduce the time required for the complete resiliency testing process by 50 percent (Figure 1). For example, a standard template simplifies review of the application architecture diagram, a key document that identifies the infrastructure needed for resiliency testing. An interface to our network management system automatically populates the diagram with information on components used by the application. The number of hand-offs between the different groups that play a critical role in the testing process has been reduced from five to two, resulting in a more positive client experience for the application teams.
Figure 1. Major Steps in the Cisco IT Resiliency Testing Process
We are working toward a goal of completing resiliency testing for 75 percent of Cisco IT business applications by mid-2016. By improving the resiliency testing approach, we are able to test more applications in less time and maintain confidence in Cisco’s ability to sustain key business operations.
To measure participation in the resiliency testing effort, the application’s test status and results will be included in the quarterly review data presented to IT executives. These metrics will also help in establishing a resiliency focus across the Cisco IT organization.
For More Information
To read additional Cisco IT case studies about a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT.
This publication describes how Cisco has benefited from the deployment of its own products. Many factors may have contributed to the results and benefits described. Cisco does not guarantee comparable results elsewhere.
CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Some jurisdictions do not allow disclaimer of express or implied warranties; therefore, this disclaimer may not apply to you.