The data center is one of the most impressive areas to visit on the Cisco San Jose, California campus, and customers who visit Cisco frequently request to see the main production data center. A tour of the data center is one of the most popular attractions at Cisco. This guided online “tour” of the Cisco main production data center conveys some of the experience of what it is like to tour the data center. The tour includes 45 photos with accompanying text descriptions of the network, servers, and storage in the Cisco production data center, including:

  • How Cisco IT staffs, monitors, and performs change management
  • How the data center is designed to remain available despite network or power outages
  • How the data center guards against damage from fires and earthquakes

By reading the following documents you can experience a virtual data center tour. Click a link to take the complete tour from beginning to end, or in any order of your choice.

Welcome to the Cisco Data Center And Operational Control Center

Welcome to the Cisco Data Center And Operational Control Center

Popular Tour Now Available Online

What follows is a guided tour of the Cisco Systems main production data center, located in San Jose, California. This entire document is 64 pages long, with 45 photographs that help bring an experience of the Data Center tour to reader. This tour is based on a series of interviews with Cisco IT data center operations managers Dick Corso and Ian Reddy.


Customers visiting Cisco usually request to visit the main production data center, and this tour remains one of the most popular, year after year. The data center is still one of the most impressive areas to visit on the Cisco San Jose campus.


Typically, to take this tour would require visiting San Jose and making an appointment with the Executive Briefing Center to schedule Dick or Ian to show you some portion of the data center. By reading this set of documents (or reading this document) you can experience a "virtual data center tour." Click the links on the left to take the tour from start to finish, or in any order of your choice. Links to printable PDFs of this tour are available at the end of each section, and a link to the entire document is available at the end of the tour.


This description of the Cisco Data Center environment and management practices provides a good background into the inner workings of a global data center. It describes the servers, storage and network found in the Cisco production data center; but more importantly it shows:

  • How Cisco IT staffs, monitors, and performs change management;
  • How the data center is designed to remain available despite network or power outages, and
  • How the data center guards against damage from fire and earthquake.

The tour consists of the following nine sections. Select where you want to start and enjoy.

Virtual Data Center Tour 01- Introduction
(PDF Document - 95 KB)

Back to Top

Data Center and Operations Control Center Background

Overview of the Cisco Data Center and Operations Control Center

All 40+ Data Centers and Server Rooms are Controlled from One Place

All of Cisco IT runs on applications and servers that live in the 40+ Cisco Data Centers and Server Rooms located around the world.. The Operations Control Center (OCC) is located in the largest production data center in Cisco, and has management and support responsibility of all the other data centers and networks and networked resources from that one location. This 8-page PDF with 7 pictures shows:


  • The scope of the Cisco data centers; and
  • How the Operations control center is built, staffed, and managed.
  • How the parallel OCC in India supports the San Jose OCC for 24x7 uninterrupted support

Virtual Data Center Tour 01 - Data Center and Operations Control Center Background
(PDF Document - 298 KB)

Back to Top

Technology in the OCC: Telephony, Monitoring and Backup

The OCC Has Backups of Backups using IP Telephony and SRST

How Cisco IT Monitors, Alarms, Prioritizes, and responds to events

The Operations Control Center responds and coordinates troubleshooting whenever there is a problem with any part of the Cisco network or data centers. They maintain backup sites and fail-over telephony to help ensure that they are up and running no matter what else is happening within Cisco. Monitoring these network and data center resources is handled by many tools, primarily by an internal Enterprise Management system. This 8-page PDF with 6 pictures shows:


  • How trouble calls are prioritized and handled by the Operations Control Center and
  • How the internal Enterprise Management tool monitors 10,000+ resources.

Virtual Data Center Tour 02 - Technology in the OCC: Telephony, Monitoring and Backup
(PDF Document - 273 KB)

Back to Top

In the OCC: Staff, Process, and Ongoing Process Development

Improved Processes Reduce Staff Needs

Cisco IT OCC Increases Scope Over Ten-fold, Decreases Staff

Cisco IT's OCC started with the 5 production data centers, and took over responsibility for the added 40+ engineering data centers. Trouble calls went from 500 to 9000 incidents per year, but the OCC was able to reduce headcount. Strict adherence to well-honed process made this possible. This 10-page PDF with 5 pictures shows:


  • How the OCC is staffed, and how the process has been automated and streamlined; and
  • Job responsibilities of the OCC.
  • Communication, Coordination, and Documentation processes in the OCC.

Virtual Data Center Tour 03 - In the OCC: Staff, Process, and Ongoing Process Development
(PDF Document - 305 KB)

Back to Top

Root Cause Analysis and Change Management

Fixing the Problems is Just the Beginning

Managing Changes, and Managing the Improvement Process, Reduces Downtime

Once the immediate problem is fixed, there is still a lot more work ahead. The Cisco Operations Control Center coordinates getting top IT engineers to determine what weaknesses allowed the incident to happen, and to find the best solutions to keep it from happening again. In addition, the Cisco IT Data Centers and Networks are continually being improved. Controlling this improvement process is critical to keeping resource availability high. This 4-page PDF with 3 pictures describes:


  • The root cause analysis process, and
  • Our change management process.

Virtual Data Center Tour 04 - Root Cause Analysis and Change Management
(PDF Document - 181 KB)

Back to Top

Inside the Build Room

Cisco IT Burns In New Equipment and Features Network Architecture in the Build Room

Build Room is Designed for Fast Reconfiguration

Before installing equipment in the production data center, Cisco IT burns in the equipment in a small Data Center test space called the Build Room. This room is designed for fast reconfigurations, and the network architecture here is contrasted with the different network architecture in production data centers. In addition, being in the build room gives us a good place to look under the floor tiles to view the cabling standards, and the console network. This 6-page PDF with 5 pictures shows:


  • Where new data center equipment is burned in before deployment; and
  • How data center network connectivity is designed; and
  • A look at cabling and console network standards

Virtual Data Center Tour 05 - Inside the Build Room
(PDF Document - 317 KB)

Back to Top

Data Center Storage and Servers

Cisco IT's Largest Production Data Center - 14,000 square feet of servers and storage

Cisco IT Holds Over 800 Terabytes of Storage in the Production SAN

This part of the tour focuses on all the hardware on the raised data center floor: the racks and racks of storage and servers that support Cisco. Cisco IT's migration to a Storage Area Network based racks of MDS 9509s has greatly improved utilization, and the tour takes a look at them. This paper also shows tape storage, and some examples of the hundreds and hundreds of servers in the Data Center. It also looks at remote monitoring of some of the larger servers, and ends with a view of the CallManager servers handling call processing for San Jose. This 8-page PDF with 7 pictures shows:


  • Storage networking switches; and
  • The physical server environment in the data center.

Virtual Data Center Tour 06 - Data Center Storage and Servers
(PDF Document - 331 KB)

Back to Top

Data Center Power

Power to the Data Center Is Backed Up By UPS, Generators

Power Distribution is N+1 To All Equipment

Power supply is critical for an available Data Center. Each Cisco Data Center is supported by a direct feed from the local power company, backed up by a room full of battery backup systems, which in turn are backed up by diesel generators. Power is distributed to racks via three separate circuit sets, so that dual-powered equipment has added backup. This 8-page PDF with 5 pictures shows:


  • How power is groomed, generated, and distributed, and
  • How and why it's important to signal the transition from battery to diesel; and
  • How power and network cables are separated under the floor, and more.

Virtual Data Center Tour 07 - Data Center Power
(PDF Document - 188 KB)

Back to Top

Overheating, Fire, and Earthquake Protection

Sensors, Detectors, and Seismic Frames Keep the Data Center Safe

Cisco IT Makes a Serious Investment in Protection

Overheating can harm servers, and fire and earthquake can devastate a multimillion dollar Data Center. Cisco IT has put a lot of extra time and money into protecting against these hazards. Added cabinet IP temperature sensors allow the OCC to monitor and manage cabinet temperatures in data centers around the world. Sensitive smoke detectors (so sensitive that extra measures must be taken to keep minor amounts of dust from setting them off) stand ready to trigger dual-system fire suppressants. And state-of-the-art earthquake protection isolates server racks from the building. This 8-page PDF with 6 pictures shows:


  • How temperature sensors are deployed, and how overheating is avoided
  • The measures taken to avoid false smoke alarms, and to respond to true fire hazards
  • The earthquake isolation platforms (and how it affects rack cabling)

Virtual Data Center Tour 08 - Overheating, Fire, and Earthquake Protection
(PDF Document - 196 KB)

Back to Top

Conclusion

Process and Redundancy Keep the Data Center Working

Keeping the Data Center Running is a Success Story

At the end of this Data Center Tour, Dick Corso and Ian Reddy sum up the lessons learned from running a Data Center: Consistent standard process, and the constant measurement that helps steer continuous process improvement, is the key to successful management of this complex and critical enterprise system, This 2-page PDF with 2 pictures shows:


  • How standards allow for outsourcing Data Center support; and
  • How continuous improvement can become a standard process

Virtual Data Center Tour 09 - Conclusion
(PDF Document - 196 KB)

Back to Top

Entire Data Center Tour

An In Depth and Personal Look Into Cisco IT's Largest Production Data Center

One of the most popular tour attractions at Cisco is the main production data center and Operations Control Center in San Jose. For years the only people to be allowed to visit this area were special customers touring the Cisco site to bring back ideas into their own enterprise areas.

Now you can go inside: inside the data center, inside the Operations Command Center, and inside the minds of two of Cisco's top data center managers. You can do it any time, by taking the Virtual Data Center Tour. This virtual tour is on the Cisco on Cisco - Inside Cisco IT web site, where Cisco IT shares internal experiences with Cisco products. Inside this 64 page PDF, illustrated with 45 pictures, you will find:


  • How Cisco manages over 40 data centers from two locations, with reduced and outsourced support engineers
  • How Cisco prepares for disasters with redundancy and process
  • How Cisco architects the networks, the power, and the processes to maintain very high availability for all data center and network resources
  • And much more.

Virtual Data Center Tour 10 - The Entire Data Center Tour
(PDF Document - 1.56 MB)

Back to Top