Cisco on Cisco
Setting Your Sights on a Data Center
Business and Technology Decisions Drive Cisco's Data Center Site Selection Process
In choosing where to build its newest North American production data center, Cisco narrowed a field of 420 potential locations, already prescreened, down to a single site in Richardson, Texas.
As with many business decisions, Cisco's data center site selection objectives were set by management and a formal process was established. Screening the pool of initial potential sites was based largely on technical considerations such as accessibility, impact of natural disasters, power efficiency, etc. However, decisions based on what is best for the business ultimately determined the final location. The Richardson data center site aligned with business goals outlined in Cisco's global data center strategy, a multiyear, collaborative effort between IT, the Cisco Workplace Resources group (facilities management), and global business process teams. Their focus? Enable Cisco's growth and business diversity for the next decade and beyond (see sidebar, "Global Data Center Strategy").
Vice President, Network and Data Center Services, Cisco
“Just as there is no perfect geographic location for a data center, there is no one-size-fits-all set of selection criteria,” says Andy Broer, IT manager for data center infrastructure at Cisco. “Every enterprise will base its decision on different business needs, a different set of business drivers.”
This is an overview of Cisco’s top data center site selection criteria.
Cisco first began looking to consolidate its North American data centers into a single pair, because the bulk of its processing and storage equipment (along with the majority of employees and customers) are in North America. The initial field of 420 possible North American locations was screened by size of the metropolitan area and accessibility. Accessibility, in terms of nearby airports and frequency of flights, mattered because Cisco wanted the data center to be easily accessible to customers and employees.
In addition to finding a location within the U.S. or Canada, the selection team focused on the following must-have criteria: a site with low risk of impact by natural disasters, situated along existing fiber paths, and served by at least two long-distance providers.
The team studied natural and environmental hazards in depth. Such disasters can be a fast route to serious downtime for a data center, so the Cisco site selection team visited and revisited the list, in increasing levels of detail. They examined the overall impact that a natural or environmental disaster could have on the region, and its effect on employees, utilities, transportation, and other business and operational factors."We collected hurricane data back to 1851," says Broer. "We then sorted locations by occurrence of Category 3 to 5 hurricanes and looked at how close they came to our possible sites." The team looked at Category 3 to 5 tornadoes, although records went back only about 50 years. This thorough screening also included the site's proximity to chemical and nuclear plants and to railway lines that might carry hazardous materials, including noting into the direction of prevailing winds. After considering a complete list of natural and environmental risks, the number of possible sites was pared down from 420 to 105. Factoring in a minimum of two separate service providers reduced the list to 89 sites.
The remaining locations were evaluated primarily using the following criteria:
- Plenty of electrical power, more than one provider, and conditions right for long-term price stability
- Availability of technical labor
- Close to customers and proximity to other Cisco operations
- A regulatory environment supportive of long-term business planning
- Other cost considerations, such as real estate, taxes, and possible governmental incentives
Tax considerations led the team to focus on U.S.-only locations, while education and workforce factors, including the need for a large pool of technical labor, further cut the list to 39 possible sites. "We looked for a pool of at least 4000 computer or IT professionals," Broer says. The construction price index, which compares building costs in various cities, eliminated one more site.
Of course, power was a big consideration. Because the site would be the primary production data center for Cisco's North American operations, it was going to need reliable power and plenty of it. The data center itself occupies 30,000 square feet of raised floor space. The site selection team wanted two 10-megawatt power feeds from separate substations, for sheer capacity, as well as the likelihood of few or no outages.
In addition, the group balanced possible tax and financial incentives from government entities. Attracting data centers to a particular region is competitive, and cities will offer whatever they can to companies that will bring this type of business to their area.
"Part of a data center's cost can be mitigated by tax breaks and other incentives," Broer says. "A lot of companies focus primarily on energy costs, but incentives from local governments should not be overlooked."
During this stage of the selection process, an unexpected consideration narrowed the field further: latency for applications hosted in the Asia Pacific region. The nature of Cisco's global operations provided this additional consideration as many North American applications also serve users in Asia Pacific. This meant latency from the data center dictated that it had to be in the western half of the U.S.
Eventually, the list of possible locations was whittled down to three U.S. metropolitan areas: Phoenix, Arizona; Denver/Boulder, Colorado; and Dallas, Texas. The team began to look for existing buildings and sites of sufficient size with the potential for advantageous tax breaks or other incentives; it went over natural and environmental hazards again, this time in even greater detail.
A huge plus for the Dallas area was an unoccupied building of the right size in Richardson, which Cisco had begun constructing several years earlier. Phoenix did not have appropriately sized sites or a building on the market at the time. The Denver-Boulder area lacked the established Cisco presence and would require completely new construction, which would add to Cisco's timeline, says Broer. Because the Richardson building was part of a Cisco campus, there were a number of other Cisco employees already there.
"It was important that our data center employees and contractors be near other employees, so they didn't feel disconnected from the company," says Sean Worthington, vice president of IT Business Services at Cisco.
Close to the Richardson data center, the Dallas-Ft. Worth airport is a hub for American Airlines, so accessibility by customers and employees is assured. Even with cutbacks to other cities, direct flights to Dallas from most major cities continue.
The data center has two incoming fiber cables from different service providers, each with OC-48 bandwidth of 2.488 Gbps and the ability to increase that capacity as needed. The Richardson site also has two power feeds, one from a substation to the north and one from a substation to the south, each of 10 megawatts. Texas is particularly reliable because its electrical power grid is not connected into any national or regional grid. Its autonomy protects it from cascading outages. At the time of selection, cost per kilowatt hour was about half the cost of Cisco's production data center in San Jose, California. While the cost has risen since, it is still meaningfully lower.
Richardson also satisfied another business criterion. From the early stages of development, there was a strong possibility that the data center would be paired with another not far away in an active-active configuration. This created the need for a suitable space within a synchronous-capable distance, 25 miles or so, for the secondary site. In July 2008, Cisco made the final decision to go ahead with the paired center, which will significantly enhance application and business process resiliency.
What's not perfect about the Dallas metropolitan area? "Mostly the heat and humidity," Broer says. "People complain and it means we can't often use ambient air for free cooling." Ambient air cooling can be a significant power-saving device.
In addition, Texas is the largest state in the continental U.S. and has the largest number of natural disasters. Historically, however, the site selection team found that natural disasters have not significantly threatened Richardson. Texas gets tornadoes, but the ominous "Tornado Alley" begins in Oklahoma, and history shows only low-level funnel clouds near the Richardson site by the time a tornado reaches that area.
Hurricanes? "Katrina was a huge storm, the size of the whole state of Louisiana, and it had one of the deepest penetrations away from water of any hurricane to hit the country," Broer says. "But when it went inland, it went up the Mississippi river. These killer storms need a warm water source and may follow rivers to maintain their power. There's no major river that runs from the Gulf of Mexico to Dallas."
Nonetheless, the building team reinforced the Richardson site to withstand winds of 125 to 145 mph. In addition, there are now 86 30-ton air handlers inside the building surrounded by a reinforced cement wall protecting the data center on the first floor and a steel-reinforced secondary wall interior to the external wall on the upper floors. Windows on the same upper walls are laminated and furnished with bladders in case rain is forced through their frames.
"Criteria such as the risk of natural disasters absolutely should be considered when deciding on the optimal geographic location for a data center," says Worthington. "But in the end, business factors might negate the risk posed by any particular disaster."
While the business drivers for data center site selection will differ by company, the following recommendations can be applied to any company's selection process.
Begin with a global data center strategy. This plan will help govern business decisions such
as levels of reliability and the site selection process itself.
Document the entire process and the reasons why you made certain choices, why some criterion were or were not on the list. This documentation can be invaluable during a multi-year program that will most likely include staffing rotations. "Documenting the rationale behind past decisions will help to address the cycle of repeat questions that new management or staff will have upon entering the program mid-flight, and will also help break the cycle of constant review and allow the program to keep moving forward," says Broer.
Know your business stakeholders well. Understand their business requirements and communicate costs with them. With good communication at the outset, there is greater assurance that issues will be addressed, expectations set, and the requirements understood across all stakeholder groups.
Enlist a senior-level champion who will support the project in the long haul, to maintain consistency in goals from year to year (this is a valuable ally for many of the processes involved in building a data center).
If the data center is being planned by two or more groups, make sure to build close cooperation from the start, so everyone understands everyone else's documented needs and business commitments.
Enact robust change management for both IT and physical construction designs. This approach will help minimize redesigning on the fly, which can be costly and onerous on your timeline.
The Richardson data center is one of the primary facilities in Cisco's global data center strategy. The cornerstone of this strategy is to enable Cisco's growth for at least the next 10 years, by addressing how to improve capacity and resiliency, and evolve network and data center services toward a service provider model.
As with the Richardson site, decisions based on what is best for the business will finally determine the location of future data centers. "The site selection process often begins by screening the pool of potential sites based on technical and geographic considerations, such as accessibility, impact of natural disasters, power efficiency, service providers, etc.," says John Manville, vice president of Network and Data Center Services at Cisco. "But the final decision should be driven by business objectives."
"People tend to optimize their choices for resiliency and availability, rather than efficiency," adds Sean Worthington, vice president of IT Business Services at Cisco. "A global plan can help change the balance for more efficiency and less cost. For example, if you have a pair of active-active data centers, you might not need to build them to Tier 3 or Tier 4 standards. Given that you have two, you might save a good deal of money building to Tier 2 standards and still have the resiliency you need. It's a matter of risk management versus cost effectiveness."
"The Richardson data center was part of the impetus for creating the global strategy as much as it was an answer to an urgent business need," says Andy Broer, IT design and build manager for the Richardson data center. "One of the fundamental principles guiding us then was availability. Today, for instance, the weighted selection criteria will have an increased "green" focus. In our new data centers, we will seek to leverage free cooling technology advancements, and that will have an effect on future site selection."
Ultimately, according to Manville, "You need to have a very clear understanding of where your business might go in the next 10 to 15 years and what it will demand in data services. Then you can develop a global plan that will enable business growth and support business diversity, which includes global expansion, new business models, and emerging markets."