Executive Summary
In today's service-oriented metropolitan networks, IP traffic is growing exponentially as customers migrate to IP-based applications. As these networks evolve to include bandwidth-intensive IP-based voice, video, and data services, carriers must boost capacity in response to demand, knowing that the collected revenue will not scale at the same rate. Therefore, carriers must find ways to optimize the operating and cost efficiency of service networks and drastically reduce costs per bit. However, the inherent inefficiencies of many of today's core network architectures can hinder service providers from achieving either objective.
Currently, most carriers rely on one of two approaches for transporting IP traffic in the core network. They can transport IP traffic over dedicated wavelength trunks, which Cisco® refers to as the IP-over-optical approach; or they can aggregate IP traffic with other types of traffic for transport over a SONET/SDH dense wavelength-division multiplexing (DWDM) network, which Cisco refers to as the IP-over-SONET approach.1 Both strategies have allowed carriers to meet today's IP bandwidth demands, but both have limitations.
Cisco recommends an alternative architecture called IP over DWDM, which eliminates the need for optical to electrical to optical (OEO) conversion equipment between the router and the optical layer (including both transponders and SONET gear) by combining DWDM router interfaces with an optically switched DWDM layer. This streamlined core architecture improves the flexibility, power efficiency, and reliability of the core network, while providing significant operating and capital cost savings.
The main question when considering this alternative architecture is this: Does IP over DWDM provide significant enough benefits to make this change worthwhile? This document compares the IP-over-DWDM approach with conventional approaches to core networks and presents detailed, quantitative research demonstrating the advantages of this architecture. For a detailed technical discussion of the IP-over-DWDM architecture, visit http://cisco.com/en/US/products/ps5763/products_white_paper0900aecd80395e03.shtml.
Challenge: Limitations of Current Core Networks
Most of today's core networks are layered architectures, in which the optical transport layer is designed and managed separately from the IP layer (often by separate management organizations). Although these architectures have met carriers' needs in the past, their inherent limitations are likely to prove a hindrance in tomorrow's core networks, which will face increased demands for IP capacity and manageability.
IP-over-optical architectures, for example, rely on transponders to convert the short-reach non-DWDM interfaces on IP routers into wavelengths. As a result, carriers must bear significant capital costs for transponders and short-reach router interfaces, as well as high ongoing operational costs to power, maintain, and repair this equipment. This extra layer of electronic equipment also increases the potential for failures, affecting the overall reliability of the network.
Due to these issues, many carriers have taken the IP-over-SONET approach. This strategy can reduce the need for additional transponders and short-reach router interfaces and, in networks that do not have large router-to-router interconnects, can allow for additional automation and grooming capabilities. However, IP-over-SONET architectures suffer inefficiencies as well. Much like the transponders in the IP-over-optical model, the IP-over-SONET approach requires a costly additional layer of equipment to groom and aggregate all traffic for SONET transport, and requires costly SONET-to-router tie lines. This extra SONET layer also increases the complexity of the network and adds potential points of failure.
Additionally, the original intent of the SONET layer - to groom and aggregate traffic - is now redundant in modern architectures. SONET networks can aggregate IP packets and conventional time-division multiplexing (TDM) traffic onto a single line system, but today's IP trunks are of the same or higher capacity compared to SONET line systems. As a result, traffic aggregation within the SONET layer offers no benefit.
Both the IP-over-optical and IP-over-SONET models also rely on fixed, rigid network configurations, generally consisting of point-to-point DWDM links, so that making network changes requires extremely intensive manual work. In an IP-over-optical architecture, carriers must install and configure equipment at each node any time they need to make a change. Although carriers using an IP-over-SONET architecture can use a digital cross-connect (DXC) to remotely reconfigure nodes, they can do so only at low bit rates - for example, when reconfiguring 50-Mbps synchronous transport signal 1 (STS-1) connections. After carriers begin using 10 Gigabit Ethernet connections, any network change still requires carriers to manually add or reconfigure interfaces. Given these concerns, many carriers are looking for an alternative approach to IP transport in core networks.
Solution: The IP-over-DWDM Alternative
Many of the problems with conventional core transport networks are a result of the inherent separation of the IP and optical layers of the network and the need to convert traffic for transport between these layers. Unlike traditional architectures, which attempt to patch the distinct layers together through various means, the IP-over-DWDM approach merges the IP and optical layers into a single, flexible network. This approach eliminates the need for fixed DWDM links, instead employing a flexible, reconfigurable optical transmission layer and tunable DWDM router interfaces. As a result, carriers can minimize the amount of electrical conversion equipment (and the associated costs) and provide true IP-over-DWDM transport (Figure 1). At the same time, service providers retain the capability to manage bandwidth at both the wavelength level through photonic switching devices such as those used by the Cisco ONS 15454 Multiservice Transport Platform (MSTP) and the packet level within the IP router. Carriers using an IP-over-DWDM architecture can maintain the separate IP and optical management infrastructures they use today, if they choose, but can also enjoy the benefits of a fully integrated management solution.
Figure 1. Side-by-Side Comparison of Core Network Architectures
Although full integration of IP and optical network layers was not feasible just a few years ago, recent optical networking innovations have made this architecture a viable option for service providers today. These advances include the emergence of innovations such as the wavelength division multiplexing physical interface (WDM-PHY) on the router and state-of-the-art reconfigurable optical add/drop multiplexing (ROADM) technologies.
WDM-PHY Router Interfaces
When choosing router interfaces in the past, carriers had two options: the LAN-PHY 10 Gigabit Ethernet standard and the Packet over SONET (PoS) OC-192 interface for IP-over-SONET transport. The LAN-PHY standard provides a means of mapping Ethernet frames directly onto a 10 Gigabit Ethernet optical circuit. This strategy is an effective and relatively inexpensive solution for in-station use, but it cannot support essential operations, administration, maintenance, and provisioning (OAM&P) features necessary for carriers to manage core networks. As a result, many carriers use IP-over-SONET interfaces, which can support OAM&P features. However, because this model packages IP-layer capabilities within the TDM framing and overhead function of SONET, it requires substantial additional investment.
An IP-over-DWDM architecture relies on a new generation of 10 Gigabit Ethernet router interfaces, which Cisco refers to as WDM-PHY. WDM-PHY interfaces combine basic LAN-PHY with the International Telecommunication Union (ITU) G.709 standard, which allows Ethernet frames to be mapped to digital wrappers that carry the OAM&P information that service providers require. As a result, providers can gain the cost advantages of Ethernet without giving up essential network monitoring and management capabilities.
Advances in OADM Technology
Recent innovations in DWDM technology, including a new generation of ROADM architectures, allow service providers to create a more flexible, cost-effective, and feature-rich optical transport layer. Today's ROADM architectures provide transparent interconnectivity among nodes, flexible add/drop configurations, and support for multiple service rates, including 2.5 Gbps, 10 Gbps, and now, 40 Gbps. In the future, ROADM architectures will support more sophisticated mesh networks and will enhance control-plane function through technologies such as Generalized Multiprotocol Label Switching (GMPLS). Ideally, these advances will allow the IP layer to dynamically configure and reconfigure the optical transmission layer in response to IP traffic.
Building on these WDM-PHY and ROADM technology innovations, today's IP-over-DWDM architectures can provide many advantages over conventional core network architectures.
Advantages of IP-over-DWDM Architectures
An exhaustive comparison of the IP-over-DWDM model with IP-over-optical and IP-over-SONET approaches, conducted in 2006 by the Cisco Research Group, demonstrates that IP over DWDM provides substantial advantages in most service environments. These advantages include the following:
• Capital savings
• Operating savings
• Improved network reliability
• Increased network flexibility
• Truly transparent optical transport
• More intelligent and proactive traffic protection
• Enhanced network scalability
Capital Savings
By integrating IP and optical layers and providing a means of preserving OAM&P capabilities without SONET, the IP-over-DWDM model can substantially reduce capital expenditures (CapEx) in core transport networks. With this architecture, carriers no longer require an extra layer of SONET equipment or transponders, the shelves that house this equipment, and the short-reach router interfaces or SONET-to-router tie-line interfaces that connect it.
The Cisco Research Group study highlights the differences in equipment and operating costs for IP-over-optical, IP-over-SONET, and IP-over-DWDM architectures. Table 1 compares the bill of parts and the part costs for various components of these architectures as a percentage of the cost of a 10-Gbps WDM-PHY ITU router interface in an IP-over-DWDM architecture. These numbers reflect current market prices.
Table 1. Comparison of Capital Expenses for Different Architectures
*Cost comparison not included, as this component is required in all architectures
The Cisco Research Group study also measured CapEx savings as a function of changing network loads for IP-over-SONET and IP-over-DWDM architectures, in both U.S. and European networks, as compared to an IP-over-optical architecture (Figures 2 and 3). The networks in these comparisons are real-world networks. The first network, a European national core network, consisted of 10 cities connected by 15 trunks, with each city having between two and five outgoing trunks. The base offered load of the network was 1.5 Tbps, with an all-to-all (fully meshed) demand pattern. The second network, a U.S. continental network, consisted of 18 cities and 23 trunks, with each city having either two or three outgoing trunks. The research group normalized the overall traffic of the U.S. network to the same base offered load as the European network to allow easier comparison.
Figure 2. CapEx Savings as a Function of Offered Load in a European Network
Figure 3. CapEx Savings as a Function of Offered Load in a U.S. Network
As the figures show, the CapEx savings of the IP-over-SONET and IP-over-DWDM models are comparable at low network loads. The ripples in the graphs at low loads result from the fact that such networks also have low interface counts, so the addition of a single interface has a relatively large effect on overall CapEx. These variations become less significant as network loads increase.
The graphs illustrate that the cost of an IP-over-SONET architecture at low loads (encompassing tie lines, DXC, and ITU OC-192 DXC interfaces) is slightly more than that of an IP-over-DWDM architecture with 10 Gigabit Ethernet ITU router interfaces. However, both models are less expensive than an IP-over-optical architecture, which requires costly OC-192 short-reach router interfaces and transponders.
As the traffic load grows, the CapEx savings of IP-over-DWDM become more significant. The IP-over-SONET architecture needs more in-site tie lines from the DXC to the router, as well as transponders for optical traffic that bypasses a given node. These costs become much greater than the cost of additional ITU interfaces on the router. At higher network loads, the IP-over-SONET architecture achieves 5 to 10 percent savings over conventional IP-over-optical approaches. However, the IP-over-DWDM architecture consistently achieves savings of approximately 50 percent.
The difference in relative savings between the U.S. and European networks results from the fact that trunk distances are much longer in the U.S. network, requiring carriers to make significant investments in channel regeneration equipment, regardless of the architecture employed.
Operating Savings
One would assume that eliminating a significant amount of SONET or electrical transponder equipment in an IP-over-DWDM architecture would result in a smaller equipment footprint and lower power consumption and OAM&P costs. The Cisco Research Group study attempted to quantify these operating savings by comparing OAM&P overhead for various network components expressed in employee hours per week, monthly rental or building maintenance costs per equipment rack, and power consumption for all three architectures. The study derived labor costs, rack rental costs, and electrical power costs from representative industry rates. Table 2 compares these operating expenses (OpEx) for the three architectures, for the same equipment described in Table 1.
Table 2. Comparison of Operating Expenses for Different Architectures
*Not applicable
The Cisco Research Group study also measured OpEx savings as a function of changing network loads for IP-over-SONET and IP-over-DWDM architectures in U.S. and European networks, compared to an IP-over-optical approach (Figures 4 and 5). The same networks investigated for CapEx savings were used to compare OpEx savings.
Figure 4. OpEx Savings as a Function of Offered Load in a European Network
Figure 5. OpEx Savings as a Function of Offered Load in a U.S. Network
As the figures illustrate, IP-over-SONET architectures carry even greater operating costs than the conventional IP-over-optical model at low loads because of the additional equipment footprint, power, and OAM&P demands of the DXCs. The IP-over-DWDM architecture requires neither transponders nor DXCs, resulting in an immediate OpEx savings over other architectures. This savings becomes even more pronounced as the network load increases, because the IP-over-DWDM architecture requires fewer investments in new equipment as it scales. At higher network loads, the overall OpEx savings of the IP-over-DWDM approach exceed 55 percent.
This research also demonstrates that, although power consumption is comparable for all three architectures at low loads, the differences become significant at higher network loads. At network loads approaching 10 Tbps, the power needs of additional transponders or SONET DXCs result in 30 percent greater power requirements. In addition to these cost savings, the greater power efficiency of IP-over-DWDM architectures can provide a distinct competitive advantage. In many markets, it is extremely difficult for carriers to secure additional utility feeds to their equipment centers, which effectively limits their ability to expand capacity to meet new demand. IP-over-DWDM architectures support greater capacity, allowing carriers to boost capacity without requiring additional power.
Extending Capital and Operating Savings to Any Network
To help ensure that the projected savings were not the result of the specific features of the test networks used in the study, the Cisco Research Group conducted the same cost comparisons on 100 synthetic networks. These networks all had the same node count and offered load as the European network studied; however, the topology and traffic patterns varied. The group generated internode connectivity patterns randomly, subject to these constraints: each node had between two and five outbound trunks, the entire network lay within a diameter of 800 km, and no trunk was longer than two-thirds of this diameter.
Figure 6 shows the distribution of the CapEx savings for IP-over-SONET and IP-over-DWDM for the synthetic networks, relative to an IP-over-optical architecture. The arrows indicate the savings realized in the real European network. Although the figure does show some variation in the savings realized, the close clustering indicates that, in general, the CapEx advantages of the architectures do not depend on the network topology or traffic matrix.
Figure 6. Spread of CapEx Savings in 100 Synthetic Networks
The research group also investigated the projected OpEx savings on the synthetic networks to determine whether these advantages were dependent on a specific network topology or traffic matrix. Figure 7 plots the savings for the 100 synthetic networks. Again, the arrows on the graph indicate the OpEx savings realized in the European network at base load. Once again, although the study showed some variation in OpEx savings realized by various synthetic networks, the clustering of the results confirms that these savings are generally independent of the network topology and traffic matrix.
Figure 7. Spread of OpEx Savings in 100 Synthetic Networks
Extending IP-over-DWDM Benefits as Networks Scale
The Cisco Research Group study clearly demonstrates the advantages of the IP-over-DWDM model in networks using 10 Gigabit Ethernet interfaces. However, although 10 Gigabit Ethernet is the most common interface used today, carriers can expect to require even higher-bandwidth channels in the future, with the next logical step being a migration to 40-Gigabit channels. To assess whether the benefits of IP over DWDM could be sustained at higher bandwidths, the research group modeled the three core transport architectures in 40-Gigabit environments. To perform this comparison, the group extrapolated from the real-world European and U.S. networks, replacing the OC-192 line system technology with OC-768.
This comparison demonstrated that both IP-over-SONET and IP-over-DWDM architectures continue to offer CapEx savings over a conventional IP-over-optical approach at all network loads. The research indicated that the advantages of IP-over-DWDM architectures are not as significant in networks with 40-Gigabit channels that have low network loads, and that the architecture even trails the IP-over-SONET model in these environments. However, this scenario is extremely unlikely, as a network with 40-Gigabit channels with a very low offered network load would severely underutilize the available bandwidth - undermining the reasons for investing in more costly and complex 40-Gigabit technology to begin with.
Like the comparison of U.S. and European networks using 10 Gigabit Ethernet channels, the CapEx savings of IP-over-DWDM architectures become more pronounced as the offered network load increases. At higher network loads approaching 15 Tbps, the study indicated that IP-over-DWDM can deliver approximately 30 percent CapEx savings over IP-over-optical infrastructures in European networks, and nearly 20 percent savings in U.S. networks.
Operating Advantages
In addition to carrying lower basic operating costs than other core network architectures, the IP-over-DWDM model offers maintenance and management benefits. Conventional architectures rely on point-to-point circuits that require a substantial, costly manual effort to install and configure equipment at each node every time a change is made. An IP-over-DWDM architecture employs a flexible optical transmission layer that allows carriers to quickly and easily establish new trunks or modify the capacity of existing trunks. Using modern ROADM solutions and tunable DWDM router interfaces, carriers often can provision circuits remotely, using simple point-and-click interfaces, and eliminate many costly service calls. The Cisco Research Group study suggests that when interfaces are deployed in groups of three in an IP-over-DWDM architecture, nearly half the service calls required by an IP-over-optical network can be eliminated. In the future, advanced ROADM solutions will even allow the network to reconfigure itself dynamically to adapt to changing traffic requirements.
The IP-over-DWDM model also allows carriers to integrate the control planes for the network and transport layers. As a result, service providers can use a single element management system (EMS) to manage the entire network. If they choose, they even can merge the operations staff responsible for each previously separate layer - a substantial potential savings. IP-over-DWDM architectures also support segregated management of optical and IP transmission for carriers that want to maintain separate management organizations.
Improved Network Reliability
IP-over-DWDM architectures employ less electronic equipment than IP-over-optical architectures, which require OEO conversion equipment, and IP-over-SONET architectures, which require DXCs. According to most network reliability analyses, transponders and some DXCs have significant failure rates. By reducing the amount of equipment in a network node, carriers also reduce the potential for failures.
Transparent Optical Transport
Merging distinct IP and optical layers into a single, flexible, switched DWDM layer makes core networks inherently insensitive to traffic format and bit rate. Carriers can use any mix of bit rates and modulation schemes that endpoint interfaces can support. As a result, carriers can choose the most efficient, least costly transport options - instead of having these choices dictated by the capabilities of the electrical core network. In addition, eliminating the extra layer of electronic equipment also reduces transmission latency. As a result, IP-over-DWDM core networks are inherently optimized for latency-sensitive services such as IP voice, video, and wireless applications.
Proactive Traffic Protection
Many carriers have continued to use SONET, despite its inefficiencies, because of its excellent resilience, performance monitoring, and failure protection features. However, new IP protection technologies such as Fast Reroute (FRR) and fast protection triggers from the G.709 layer in the interface can perform the same or comparable functions without additional layers of equipment and their associated costs. Additionally, although SONET provides excellent traffic protection in the SONET transmission layer, it cannot protect against failures in the IP layer itself. As a result, service providers today also use an IP-based protection scheme - often leading to duplication of protection bandwidth and administrative requirements.
IP-over-optical architectures also have problems with IP-layer protection. In traditional core network models, IP routers have no way to recognize degraded optical signals and will engage IP-based protection schemes only in the event of an outright trunk failure. In IP-over-DWDM architectures that use WDM-PHY router interfaces with G.709 framing, the network can directly alert IP routers performing FRR functions of optical network problems. The IP routers can then either switch away from a deteriorating connection prior to a failure or, if a failure occurs, provide recovery speeds that are better than SONET's.
Conclusion
Carriers have several options for core networks. Many service providers employ a traditional IP-over-optical or IP-over-SONET model; these architectures have effectively met their requirements. However, these conventional approaches are not optimized for the requirements and traffic mix of today's and tomorrow's core networks.
Cisco recommends an alternative IP-over-DWDM model, which merges traditionally separate IP and optical layers into a single, flexible DWDM transmission layer. This streamlined core network architecture allows service providers to eliminate substantial OEO conversion equipment between IP routers and the optical layer. Ultimately, the model offers greater flexibility, scalability, and power efficiency than conventional core network approaches and delivers substantially lower total cost of ownership.
1For convenience, this document uses the term SONET; however, the discussion equally applies to SDH technology. In fact, it applies more widely to any subwavelength time-division multiplexing (TDM) technology, such as G.709 multiplexing - for example, 2.5G (ODU1) multiplexing.