Cisco on Cisco
Optical Networking Case Study: How Cisco IT Used CWDM to Interconnect Japanese Data Center Sites
Cisco Systems® maintains two sales offices in Tokyo, Japan. One is located in the government district of the city known as Akasaka and the other in a commercial area known as Shinjuku. As is typical for enterprises in the region, Cisco® Japan IT colocated data centers in each of the offices to support the IT needs of sales, Cisco Technical Assistance Center (TAC), and engineering staff within the facilities, but needed to connect these two locations together. In Japan, many service providers offer managed services like Gigabit Ethernet service to end customers. Cisco leased a Gigabit Ethernet circuit from a service provider at a cost of one million yen per month (about US$9000 at an exchange rate of 111 yen per dollar) to tie the two data centers (and offices) together.
Although these sites had been in place for some time, it is common for sales offices to change locations-to relocate to larger facilities or move closer to customers, etc. Reestablishing a colocated data center can be costly and disruptive. In 2003, Cisco Japan IT considered relocating the two data centers to a single, dedicated facility. A more permanent facility would allow Cisco Japan IT to engineer a higher level of availability with more robust redundancy. Situating the data center outside the expensive city center also would reduce leasing costs.
Although a dedicated data center appeared to be a good solution, the cost of providing reliable connectivity between the data center and the two sales offices seemed prohibitive. To provide an acceptable level of reliability and redundancy, a circuit would be needed from the data center to Akasaka, from Akasaka to Shinjuku, and from Shinjuku back to the data center. With three circuits required, the one-million-yen-per-month Gigabit Ethernet lease cost would triple to approximately $27,000.
Cisco IT manager,
As early as 2001 Cisco Japan IT had investigated leasing dark fiber from carriers as an alternative to the costly Gigabit Ethernet service. Because the great majority of the cost of laying fiber optic cable is labor, not the fiber, telecom carriers typically install more fiber strands than they need. In many areas of the world, private enterprises sometimes can lease these "dark" unused strands from carriers at low rates to connect company sites in metropolitan areas. They had abandoned this solution in 2002, however, because carriers in Tokyo were not offering dark fiber to enterprises. Furthermore, the dense wavelength-division multiplexing technology that enabled multiple channels over a pair of optical fibers, a critical requirement for Cisco future bandwidth needs, was expensive and difficult to manage.
To provide viable connectivity between the new data center and the two sales offices, Cisco Japan IT needed a high-speed, low-cost solution that would be reliable and easy to manage.
In 2003, with the construction of a dedicated data center under serious consideration, Cisco Japan IT again investigated dark fiber. In Japan, Class 1 carriers can provision and lease fiber, but can lease it only to Class 2 carriers and service providers. Class 2 carriers can in turn lease them to corporations. This time Cisco Japan IT found that there were several carriers in Tokyo who were receptive to leasing dark fiber, and at prices far lower than the current managed Gigabit Ethernet services. In addition, the Cisco coarse wavelength-division multiplexing (CWDM) gigabit interface converter (GBIC) solution was now available, which could provide economical optical bandwidth scalability with little or no management requirements. CWDM employs multiple light wavelengths to transmit signals over a single optical fiber. CWDM technology is a crucial component of Ethernet LAN and MAN networks because it maximizes the use of installed fiber infrastructure at an attractive price point.
The dark fiber and CWDM GBIC solution offered another benefit. "If we had chosen to lease Gigabit Ethernet circuits from a service provider, we would have a single provider with the risk of a single point of failure," says Zhengming Zhang, Cisco IT network engineer. "With dark fiber, however, we have the ability to select circuits from different providers, ensuring physically diverse routes, which was an important requirement for us."
A building was located for the new data center about 25 kilometers from Akasaka and 30 kilometers from Shinjuku. Efforts to lease the dark fiber links began in October 2003. The task of relocating nearly 50 racks of equipment from Akasaka and 30 from Shinjuku to the new data center began on February 28, 2004, with the first move on March 6.
Although Cisco IT uses CWDM for network access in other locations, the Tokyo Internet data center (IDC) is the first place in which Cisco takes advantage of CWDM to interconnect an IDC with multiple Cisco offices. The advantage of CWDM technology is that it can transmit and receive signals over a single strand of fiber. With Cisco CWDM GBICs, a maximum of four channels can be multiplexed over the single fiber. CWDM provides bandwidth for growth and secure traffic separation with only one single fiber. The Tokyo IDC currently utilizes one channel, which is used for a single gigabit circuit.
Several dark fiber providers are located in Tokyo, and Cisco Japan IT included the dark fiber vendor selection process as part of the overall data center evaluation process. Seven venders were sent Requests for Proposal (RFPs) for the data center project, and Cisco Japan IT selected three separate fiber providers to ensure redundant paths to all three sites. A single provider also was responsible for supporting service-level agreements (SLAs) on all three fibers and for terminating the fibers in each of the sites. "Installation was very simple," says Greg Duncan, Cisco IT Manager. "They pulled the dark fiber into our racks and attached the SC connector, and that was it." Although the fiber providers had estimated eight to nine weeks for completion, they installed the first circuit in less than four weeks and the remaining circuits within another week.
Because the CWDM equipment is passive, it does not amplify the signal traveling through the fiber. The signal naturally weakens, or attenuates, over the length of the fiber based on factors such as the quality of the fiber, distance, and number of splices. If too much loss occurs, the signal at the receiving end will be too weak to detect and could cause packets to be dropped. Testing carried out by Cisco Japan IT showed that the CWDM equipment could tolerate a loss of 30 dB with no packets dropped. The fiber provider SLA guaranteed that these fibers would not exceed 24 dB. "This is one of those few instances where everything has gone exactly to plan," says Duncan. "Our fiber provider installed every fiber link with less loss (less than 16 dB for all three paths) and in less time than what they promised. It went very, very smoothly."
Connecting the fiber to the LAN environment at each of the three locations is the Cisco CWDM GBIC solution. The primary components of the CWDM GBIC solution are the Cisco CWDM GBIC and Cisco CWDM optical add/drop multiplexer (OADM) modules. The Cisco CWDM GBICs are active components that convert Gigabit Ethernet electrical signals into an optical single-mode fiber (SMF) interface. The CWDM GBIC plugs into standard GBIC ports on Cisco switches and routers. No dedicated or additional routers were required for the deployment. "The CWDM solution is very cost effective," says Zhengming Zhang. "You can use existing routers or switches as long as the hardware has the Gigabit Ethernet module." At the Akasaka and Shinjuku offices, existing Cisco 7603 routers were used, as shown in Figure 1. A Cisco 7603 Router also was used at the data center.
Figure 1. CWDM Network Diagram
The CWDM OADM modules used in this deployment (CWDM-MUX-4-SFx) are passive optical components that multiplex multiple wavelengths from multiple SMF pairs into one SMF strand. Other CWDM OADM modules are designed to multiplex multiple wavelengths into a pair of SMF fibers where a dual-fiber topology is used. The CWDM OADM modules are connected to the CWDM GBICs with SMF using dual SC connectors. Because they are passive devices, no power is required. Neither the CWDM GBIC nor OADM modules require any configuration. The technicians simply matched the GBIC color with the color of the channel interface on the respective OADM module. As with a Gigabit or Fast Ethernet interface, an IP address must be configured for the GBIC interface if it is used as a Layer 3 router port. If a Layer 2 switch is used, spanning tree configuration may be required.
The CWDM (CWDM-MUX-4-SFx) solution supports up to four channels over a single fiber. When additional channels are needed, technicians simply plug another CWDM GBIC into a GBIC port on the Cisco 7603 Router, as shown in Figure 2. No new fibers or changes to the dark fibers are required. The technician simply plugs the second CWDM GBIC into the Cisco 7603 Router and connects it to the OADM with a pair of single mode fibers. "Adding a Gigabit takes only about five seconds and costs about $750," says Zhengming Zhang.
Figure 2. Adding a Second CWDM GBIC
The Cisco CWDM GBIC solution supports point-to-point, ring, hub-and-spoke, and mesh network topologies. The solution offers both path protection (using two fiber paths for the same wavelength) and client protection at the channel endpoints through the CWDM GBICs. Availability redundancy schemes such as EtherChannel technology, Spanning Tree Protocol, and Hot Standby Router Protocol (HSRP) can be used to provide redundancy.
Cisco Japan IT chose a multisite point-to-point topology for the Tokyo network deployment because of its simplicity and cost. Using Enhanced Interior Gateway Routing Protocol (EIGRP), the network detects a failure in one of the links and automatically reroutes traffic to the redundant path. A full-mesh solution would have required two fibers between each location, more extensive hardware, and greater management requirements, such as Spanning Tree Protocol. "If two fibers were as inexpensive as one, we might have chosen a full mesh solution, but this solution is also good," says Zhengming Zhang.
The Tokyo IDC hosts all the regional mission-critical services and applications and supports all WAN connectivity for Cisco Japan offices. Some of these services include Internet access, extranet connections, VPN concentrators for site-to-site and user-based IPSec VPN connections, content networking and IP/TV® streaming video broadcasts, CallManagers, storage filers, printing servers, and many more. high performance of the network in IDC makes all services available to users as if the resources are located nearby. In addition, critical user and application data in Japan data is replicated to our Hong Kong IDC, which provides redundancy in the event of a critical hardware failure. Cisco quality of service technology allows for near-real-time replication without consuming other critical services such as Web, video, and voice services.
Circuit diversity was an essential factor for building a highly available IDC in Tokyo. In this case both physical circuit routes and carriers are diversified. Normally, circuit backup is sufficient and physical diversity of circuit paths is valuable but sometimes hard to achieve; carrier diversity is even more valuable but usually too difficult to achieve. Carrier diversity is valuable because there are rare instances of multiple outages on a single carrier's network (for example, due to a port module board failure that causes multiple circuits to fail at the same time). In Tokyo, Cisco IT was able to select a unique carrier for each circuit. This was fortunate because high availability is crucial. In addition to hosting all mission-critical services and applications for Cisco Japan, the IDC connects Cisco Japan's large locations to the rest of the network, and to services and applications located in other regions. This diversity helps ensure that mission-critical services and applications are always available no matter which circuit or carrier fails.
Before being relocated to Tokyo IDC, all mission-critical devices were hosted in either the Shinjuku office or Akasaka office. Every year each building conducted a necessary electricity maintenance process that would bring down entire power supplies for 48 hours. During this maintenance, all the customer devices in the building lost power and none of the applications or services were available. Cisco IT used uninterruptible power supplies (UPSs) to provide backup for a few hours, but battery backup for 48 hours was not realistic, since it would require a huge number of batteries, and the cost, weight, space, and safety factors made this too expensive to consider. During the 96 hours of power outage (different dates for each building outage), users were unable to use file and printing servers, DHCP, DNS, ACS, DC (directory service), and local VPN concentrators. Some field sales offices in Japan were unable to connect to the corporate network through site-to-site-based IPSec VPN. Users would have to connect to the VPN concentrators in San Jose to access corporate resources and the Internet, and the long distances between Japan and the United States made VPN performance slow.
Relocating all servers and network equipment to a single Tokyo IDC has resolved all these problems. The Tokyo IDC has three power generators, each one with an independent power source. The Tokyo IDC supplies power to each server rack with at least two separate power feeds (and some have three separate power feeds). The N+1 redundancy has greatly improved service availability.
Had Cisco Japan IT chosen to lease Gigabit Ethernet circuits from a service provider, the cost would have been approximately 3 million yen (about $27,000) per month. Instead, the three dark fibers cost approximately 1.1 million yen (about $9900)-a saving of more than 60 percent. And by adding relatively inexpensive CWDM GBICs and OADM modules to the existing infrastructure, bandwidth can be doubled, tripled, or even quadrupled without additional monthly fiber leasing expense.
Total route diversity has eliminated single points of failure and ensures high availability between sites. The network has been in operation since March 2004 with no problems. On May 15, Cisco Japan IT took offline one of the Cisco 7603 routers to replace line modules. The network rerouted automatically and connectivity between sites was never affected.
Improvements in other areas were achieved as well:
More usable space available at the two Tokyo offices: Relocation of shared services to the Tokyo IDC allows us to reuse the expensive downtown Tokyo office space for services showcasing, labs, and customer support staff. When new Cisco technology or products go to market, the existing spaces can be used for sales and marketing purposes.
Removed duplicate hardware costs: Some shared services and applications were duplicated in the Shinjuku and Akasaka offices (for example, storage filers and printing servers). When new services were deployed, the same hardware and software had to be installed in both offices. With the Tokyo IDC, existing services are combined into fewer, higher-capacity hardware devices, which perform the same tasks at a lower price per task.
Simpler management: With the centralized colocation of services in the Tokyo IDC, management, troubleshooting, and maintenance are easier than when equipment was located in two separate offices.
Reduced IT labor: Cisco IT Japan used to spend a significant amount of time on cabling, mounting hardware, simple hardware replacement, and circuit installation. These tasks are now handled by the Tokyo IDC support staff, and Cisco IT Japan can concentrate on new service design and implementation.
Unlike most deployments, the CWDM project went exactly as planned. "I can't think of a single instance during the CWDM deployment that trapped us or caused us to reconsider our plans," says Duncan. Several factors made this deployment simple and trouble-free. Among them was the willingness of different service providers that, even as direct competitors in the same market, were willing to share cable path information to provide diversified routing. And the fiber vendor performed as promised. "They lived up to their word without exception, even beating their schedule," notes Duncan. And finally, the CWDM equipment offered no surprises. "I think the lesson learned for me is, it was as easy to do as what the product information said it would be," says Duncan.
Cisco Japan IT plans to use the Tokyo CWDM solution for several new applications over the next year. Prior to CWDM, separate access paths had to be provided for labs that needed direct demilitarized zone (DMZ) access, resulting in additional Internet access points distributed throughout the different labs. One of the four channels on the CWDM already is being used to carry secure, segregated lab traffic into the DMZ, located at the new data center, from the existing engineering lab at the Shinjuku office. Without CWDM, a dedicated leased line, which might cost at least 300,000Yen ($2700) monthly, would have been required to connect the DMZ lab in Shinjuku to DMZ backbone in the Tokyo IDC. Other labs will follow, replacing their separate Internet trunks with a channel on the CWDM.
If more circuits between the same sites are required in the future, Cisco IT Japan can easily add CWDM GBIC modules to expand bandwidths to 2 Gbps, 3 Gbps, or 4 Gbps. Because each channel is utilizing a different wavelength to transmit and receive, each one can be treated as an independent physical circuit. This allows us to use another channel of CWDM to interconnect the DMZ lab in the Shinjuku office to the DMZ backbone in the Tokyo IDC over the same dark fiber without compromising security.
Another advantage of the CWDM dark fiber solution is its ability to support technology demonstrations without negatively affecting production traffic. Before customers spend a large sum of money for a Cisco solution, they want to see that it works. Cisco sales and engineering teams set up demos for different solutions. Often, the customer might be at the Akasaka sales office while the servers and critical resources that make the demo work are in Shinjuku. Engineers would have to connect the two sites but they could not use the existing Gigabit Ethernet circuit because of existing IT policies and information security policies, which caused them to find another circuit. With CWDM, they will be able to use a separate fiber channel without raising security concerns. In addition, the extra bandwidth will allow Cisco IT Japan labs and sales locations to interconnect servers to storage using SAN iSCSI, FCIP, or other applications.
Cisco TAC and engineering groups currently occupy a sizeable portion of the leased space at the Shinjuku facility. At some point within the next year, those groups probably will be relocated to lower-cost facilities outside of Shinjuku. "That's going to be a lot simpler for us to do because we just need to hook them into the new data center and extend another dark fiber to their new location," says Zhengming Zhang.