10 Gigabit Ethernet DWDM Interconnections in Enterprise Campus Networks
PDF(924.5 KB) View with Adobe Reader on a variety of devices
Updated:Oct 10, 2006
This document provides technical description about 10 gigabit ethernet dense wavelength-division multiplexing (DWDM) pluggable optics and application information for deploying 10 gigabit ethernet dwdm interconnections in enterprise campus networks
Since the IEEE 802.3ae standard was ratified in 2002, the number of 10 Gigabit Ethernet deployments has grown rapidly. The 10 Gigabit Ethernet market has grown from US$385 million in 2004 to $885 million in 2005 and has a potential of $3.4 billion in 2009, according to the Dell'Oro Group.
The current trend in the networking industry is consolidation. Even without specific applications requiring 10 Gbps, bandwidth needs within the campus network have grown quickly because of network consolidation feeding the demand for higher bandwidth. Today 10 Gigabit Ethernet is deployed through the campus core, the campus distribution, and also the campus access network, to achieve a reasonable oversubscription ratio, if deploying Gigabit Ethernet to the desktop or to servers. (See Figure 1.)
Figure 1. 10 Gigabit Ethernet Campus Design
To scale the core bandwidth beyond 10 Gigabit Ethernet, a new IEEE standard is needed. But standardization has not been started yet and is not expected to be ratified within in the next four years. The standard Ethernet service 802.3ad link aggregation can be used to aggregate up to eight 10 Gigabit Ethernet links into a virtual 80 Gbps connection. DWDM can be used to transport various channels over an optical infrastructure. DWDM is an optical technique to transmit multiple channels on the same fiber strand. Using the standard ITU 100 GHz grid in the C-band with the 4-skip-1 channel plan, 32 different channels can be transported over the same fiber, resulting in a fiber transmission capacity of up to 320 Gbps.
The goal with convergence is to build network architectures that are simpler to operate with enhanced flexibility to deliver new services with reduced capital expenditures (CapEx) and operating expenses (OpEx). In order to address the CapEx and OpEx aspects, the trend in DWDM network architecture is to drive cost elements out of the photonic layer and into the IP routers and Ethernet switches and to achieve network convergence on the photonic layer.
NETWORK CONVERGENCE ON THE PHOTONIC LAYER
Optoelectronics technologies over the past few years have evolved dramatically. Not only can 10 Gigabit Ethernet optics be packaged in Small Form-Factor Pluggable (SFP) transceivers such as Xenpak, X2, or XFP, but also even more sophisticated long-reach DWDM optics can now be accommodated within the size and power constraints of hot pluggable form factors such as the Xenpaks. These technical advances are opening up a new paradigm to build enterprise core networks using the power of DWDM.
DWDM functions used to be typically carried out in dedicated optical equipment such as transponders, capable of operating the wavelength conversion from, for example, a "traditional" 1310 nm laser to a DWDM channel with very tight control over the wavelength stability. The Cisco Systems
® DWDM Xenpak enables 10 Gigabit Ethernet over DWDM without the need of dedicated external equipment. The sophisticated DWDM function is squeezed into a Xenpak module, which is equivalent from a Layer 2 and higher perspective to any other Xenpak available on Cisco
The capabilities of current hot-pluggable Xenpak transceivers allow the migration of DWDM functions directly into routers and switches, and customers can experience operational and CapEx savings by removing required costly optical equipment to perform the wavelength translation just mentioned. IP over DWDM (IPoDWDM) is the Cisco paradigm to design cost-effective, converged, scalable optical networks based on Cisco Catalyst
® 6500 Series Switches for Ethernet equipped with DWDM pluggable optics. (See Figure 2.)
Figure 2. IPoDWDM on Ethernet Switches
Moreover, consolidating the DWDM function in an Ethernet switch also provides significant operational advantages:
• Less power and rack space is now required because of elimination of transponders.
• Fewer optics and electronics also mean fewer points of failure in the network.
• The DWDM layer can be managed by a LAN administrator without the need of specialized personnel to run the optical layer, which can very well be reduced to pure passive elements to multiplex and demultiplex the Xenpak wavelengths.
DWDM is the unifying technology that allows consolidation of data center and enterprise campus applications over a common transport infrastructure. The Cisco DWDM Xenpak pluggables allow enterprise companies to provide scalable and easy-to-deploy 10 Gigabit Ethernet services in their networks. The Cisco DWDM Xenpak input/output devices are hot-swappable, plug into an Ethernet Xenpak port of a Cisco switch or router, and are not wavelength-tunable. These DWDM pluggables are delivered with 32 different product numbers (SKUs) supporting the standard C-band ITU 100-GHz wavelength channel plan, as shown in Figure 3. The Cisco DWDM Xenpak solution allows the reach of these networks to scale up to over 200 km by means of optical amplifiers known as EDFAs and dispersion compensation units using single-mode ITU-T G.652 fibers.
Figure 3. ITU 100 GHz DWDM Channel Plan and 10 GbE DWDM Xenpak Optic
The Cisco DWDM XENPAK supports digital optical monitoring (DOM) capability according to the industry-standard SFF-8724 Multi Source Agreement (MSA). This feature gives the end user the ability to monitor real-time parameters of the Xenpak, such as optical output power, optical input power, temperature, laser bias current, and transceiver supply voltage. Digital optical monitoring enables network administrators to view the optical network physical layer performance through proactive monitoring and management of ports and links and enables optical management capabilities, without the usage of complex optical tools that are not well known and supported by enterprise network operators. Digital optical monitoring is supported on DWDM Xenpaks with Cisco IOS
® Software Release 12.2(18)SXE on Cisco Catalyst 6500 Series Switches. The real-time digital optical monitoring parameters can be gained using a command-line interface or by using the SNMP interface.
Figure 4 shows the output of a 10 Gbps DWDM Xenpak device. The measurement has been performed with a single DWDM Xenpak interconnected with a loopback cable via a 10 dB attenuator. The internally calibrated transmitted power is 0.0 dBm and is above the -1.0 dBm minimum transmitter power specified in the data sheet. The measured received optical input is -10.1 dBm caused by the attenuator and connector losses. The specified minimum optical input power is -24 dBm, giving a power budget of 23 dB. A more detailed command line output including the threshold values for temperature, current, voltage and optical TX/RX values is shown by executing the command "show interface transceiver detail". To enable threshold notification for all sensors in the transceivers the command "snmp-server enable trap transceiver" has to be made. Enabling transceiver monitoring is done with the command "transceiver type all".
Figure 4. DOM Output of a DWMD Xenpak Device at ITU Channel 26
Cisco offers two different optical architectures to implement 10 Gigabit Ethernet using Xenpak DWDM pluggables:
• Auto-adjusting intelligent DWDM solution based on Cisco ONS 15454 Multiservice Transport Platform (MSTP) with R-OADM technology
• Fixed-channel filter solution based on Cisco ONS 15216 Metropolitan DWDM 100-Ghz Fleylayer Filter Solution
Traditional DWDM solutions are using fixed-channel filters that can add and drop specific channels or bands of wavelengths. The components of these systems are passive and do not require any electrical connections, enabling a very low-cost optical solution if combining with DWDM transceiver pluggables. Such a system consists of add-drop modules covering various channel ranges, optical splitters or combiners to connect add-drop modules, and optical amplifiers to amplify the signal and variable optical attenuators (VOA) to equalize the optical power of channel groups.
Figure 5 shows the components of an add path, where six 10 Gigabit Ethernet Xenpaks are multiplexed on a single fiber. The system consists of two 2-channel add modules, one 8-channel add module, a 4:1 combiner, a variable optical attenuator, and an optical amplifier to move beyond typical campus distances. Additional channels can be added easily without service interruption to the existing 8-channel add module, but also an additional module can be connected to the free combiner port. The system shown is connected in a parallel layout with x:1 combiners, but also linear layouts are possible. The layout depends on the balance between the highest degrees of flexibility compared with the better channel power equalization.
Figure 5. Fixed-Filter Based DWDM Components (Add Path)
The Cisco ONS 15216 DWDM Flexlayer solution enables enterprises to deploy low-cost fixed-filter based DWDM networks in the campus and metro area. The ONS 15216 provides all the components to build a 32 channel fixed-filter DWDM solution, including optical amplifier, variable optical attenuators, and the support of an optical supervisor channel for management connectivity. (See Figure 6.)
Figure 6. ONS 15216 DWDM Flexlayer Chassis (4 Module Slots) Filled with a 8-Channel OADM, an Optical Splitter/Coupler, and a 4-Channel VOA
In fixed-filter DWDM solutions, changes to the optical services must be engineered in advance and subsequent deployment most likely
will disrupt existing services. A reconfigurable optical network offers the possibility to grow services between sites with no advanced engineering or planning. In the past, reconfigurable optical networking technology was too expensive or delicate to be widely deployed. With newly mature silicon-based integrated Planar-Lightwave-Circuit (PLC) components, reconfigurable optical add/drop multiplexers
(R-OADMs) can be build on waveguides utilizing manufacturing processes from the semiconductor industry allowing easy manufacturing at low cost. PLC technology offers the integration of all components (switch, multiplexers, de-multiplexers, variable optical attenuators and photodiodes) into a compact device. Dynamic channel equalization is available through integrated power measurement and control of every channel. The optical switching time to provide future protection mechanisms is less than 50 msec.
Figure 7 illustrates the 32 channel R-OADM architecture as 4 channel device for simplification. Incoming DWDM signals are split into a pass-through path and a drop path. With software control each of the 32 channels can be dropped in the de-multiplexers. The pass-through wavelengths are feeding in multiplexers doing wavelength adding via software control. While using 10 Gigabit Ethernet DWDM pluggables no transponders are used in this setup. In both East and West directions 2 x 10 Gigabit Ethernet services are terminated, while the remaining wavelengths are fed through the wavelength selective switch.
Figure 7. R-OADM Architecture
The fast and simple provisioning of networks using cost-effective R-OADM technologies has reset the standard of DWDM transmission in the last years and has been deployed widely in the industry.
The Cisco ONS 15454 Multiservice Transport Platform (MSTP) shown in Figure 8 offers a comprehensive reconfigurable optical networking solution, including 32 channel R-OADMs available in C and L bands, multi-rate tunable transponders and muxponders (supporting any service, including Ethernet, TDM, and storage) and intelligent WDM software that can automatically manage and adjust optical power levels as new services are added or deleted and offer end-to-end management of wavelengths.
Figure 8. Cisco ONS 15454 MSTP and R-OADM Cards
Compared with the fixed-filter design, the R-OADM design allows any wavelength to be dropped at any site without affecting any of the other services delivering the highest flexibility. The in-service growth from 1 to 32 wavelengths and the simplified network design and provisioning will result in lower operational expense, faster service provisioning, but will need additional CapEx costs, compared with the low-cost fixed optical filter solution.
Enterprise customers should compare both solutions using fixed filter or R-OADM design, before they decide which technology fits best to build a converged optical network with 10 Gigabit Ethernet DWDM pluggables.
10 GIGABIT ETHERNET DWDM DEPLOYMENTS IN CAMPUS NETWORKS
The main driver for DWDM deployment in metropolitan enterprise networks in the past has been data center interconnection, offering multiple services over rented fibers. In campus enterprise networks, fiber exhaustion can be the driver for deploying DWDM technologies, but most enterprises have enough fiber available to scale their network needs today. This situation changes dramatically if the campus is segmented over several areas crossing public ground. Enterprises have to rent fibers from local service providers giving the opportunity to deploy campus DWDM to save cost, ease the deployment and accelerate the service setup time (see Figure 9).
Figure 9. Campus Segmentation and Fiber Layout
In this kind of scenario, the fixed-filter based DWDM solution with Cisco ONS 15216 DWDM Flexlayer combined with 10 Gigabit Ethernet DWDM Xenpaks is offering a very attractive solution.
In the example shown in Figure 10 a full mesh network with 2 x 10 Gigabit Ethernet between each switch on the four campus sites is built. Standard non-DWDM 10 Gigabit Ethernet Xenpaks are used for connections build over private ground, where enough fibers are available. Between the campus sites, where rented fiber is used, a fixed-filter DWDM system using 2-channel add-drop-multiplexer is designed to build the full mesh of 2 x 10 Gigabit Ethernet between each of the switches. Even if one external fiber is broken, all switches can be reached via not more than one additional switch hop. Taking into account the maximum insertion loss for the 2-channel OADMs of 2.5 dB for add/drop, the maximum insertion loss of 1.6 dB for pass, an additional optical power margin of 2 dB and the fiber attenuation of 0.25 dB/km the total maximum loss between sites is 16 dB. DWDM Xenpaks have an optical power budget of 23 dB and can be used in this kind of scenario even without amplification.
Figure 10. Segmented Campus with Fixed-Filter DWDM Solution
Another area for Enterprise DWDM solution is the interconnection of primary and backup data centers to protect information and have 24-hour access to critical application services. Cisco offers different optical solutions allowing these customers to select the transport service-whether IP/Ethernet, SONET/SDH, or WDM-that best meets their recovery time and point objectives for business continuity. Cisco also offers several DWDM platforms for metro networks, such as the Cisco ONS 15216 Flexlayer and Cisco ONS 15454 MSTP. In addition to transparently supporting various services, benefits of these systems include increased capacity (more services, such as Gigabit Ethernet and Fiber Channel/FICON on a fiber pair), high-density service aggregation (many services on a single wavelength), scalability from 2.5- to 10-Gbps wavelengths, and service mixing (different services on the same wavelength).
In this white paper we like to focus on DWDM application in the data center itself. Data centers often have elaborate fire prevention and fire extinguishing systems. Fire prevention and detection systems are typically zoned, and high-quality fire-doors and other physical fire-breaks used, so that even if a fire does break out it can be contained and extinguished within a small part of the facility, as illustrated in Figure 11.
Figure 11. Data Center Fire-Break Zones and Backup Data Center
When deploying new fibers through fire-breaks, the fire break need to be opened and sealed again within maintenance windows involving various infrastructure, security and operational teams. This complex workflow may be expensive, depending on the facility conditions and infrastructure. Network planners and operators like to be more independent from these infrastructure restrictions. Deploying R-OADM technology in every fire-zone enables the network operators to deploy wavelength services between the fire zones providing in-service growth, simplified network design and provisioning and fast reaction to service requests. This concept will help to lower operational expenses and allows the flexibility needed in data center environment.
Figure 12 shows a R-OADM deployment scenario between 4 different fire zones within a data center. In this example 6 wavelengths are used to interconnect the 4 zones in a full mesh with 2 x 10 Gigabit Ethernet services.
Figure 12. R-OADM Deployment Between Data Center Fire Zones
Tools like the Cisco MetroPlanner offer the possibility to model and test DWDM optical networks in a graphical environment to validate, analyze and optimize network designs. After entering the number of network sites, the fiber distances, fiber type and service demands (e.g. 2 x 10 Gigabit Ethernet from site 1 to site 3) the Cisco MetroPlanner will do a power consumption calculation, design the wavelength routing and define the required add/drop filters, optical amplifiers and dispersion compensation units to fit the user traffic demands at minimum costs. The Cisco MetroPlanner will generate reports about optical node shelf configuration, bill of material lists, installation parameters and the optical results. The tool is able to calculate a network upgrade after freezing existing network parameters to implement forecasted channels.
Figure 13 shows a screenshot of the MetroPlanner tool for the network described in Figure 12. The network layout consists of 4 sites connected with a fiber ring. The distances between the sites are fixed to 1 km and a loss factor of 0.25 db/km is used to calculate the power budgets. As services between the sites a full mesh with 2 x 10 Gigabit Ethernet is chosen, with an upgrade option of additional 2 x 10 Gigabit Ethernet channels. As client interfaces 10 Gigabit Ethernet DWDM Xenpaks have been selected. The wavelength routing table ?
(6 wavelengths) and the shelf configuration for site 1 are also shown.
Figure 13. MetroPlanner 7.0 Screenshot
The Cisco ONS 15454 MSTP supports besides Gigabit and 10 Gigabit Ethernet DWDM pluggable interfaces and also a large variety of transponder line cards and muxponder line cards offering fiber channel, SDH, ESCON and other services, so that the system can be extended to a multi-service optical network offering additional convergence on the optical layer.
SUMMARY: DEPLOYING IP OVER DWDM IN ENTERPRISE NETWORKS
Deployment of 10 Gigabit Ethernet in campus networks is the only available technology today to serve the demand for higher bandwidth in consolidated campus networks. An Ethernet standard beyond 10 Gigabit Ethernet is not likely to be ratified within the next 4 years, forcing enterprise network planners to aggregate 10 Gigabit Ethernet links. To achieve network consolidation on the photonic layer, cost-effective optical transmission technique is combined with available 10 Gigabit Ethernet DWDM pluggables and optical switching: IP over DWDM.
IPoDWDM is a paradigm for cost effective, scalable, simplified and more efficient optical DWDM networks. This concept derived for service providers can be extended to enterprise networks, offering the optical layer as convergence layer within the campus infrastructure.
DWDM deployments in enterprise campus networks are often driven by fiber exhaustion between campus segments or if fibers passing public ground have to be rented between the campus areas. A cost-effective solution using 10 Gigabit Ethernet DWDM pluggables and fixed-channels OADM filters (ONS 15216) offers more flexibility to campus network operators. The ease and speed of service delivery can be further accelerated using R-OADM technology (ONS 15454), where service installation and provisioning is fast and simplified.
In data centers the segmentation into fire zones often is restricting fast and easy deployment of new services, if fibers are exhausted between the zones. The R-OADM technique allows data center operators to deploy a scalable optical transport system, with fast and easy service installation and provisioning and makes them more independent from infrastructure constraints in an environment, where fast reaction is an obligation.