The Cisco Routed Optical Network
Service provider traffic continues to be over-whelmingly IP services, and consequently:
Current service provider network infrastructure is not optimized for transport of IP services and is challenged with high TCO attributed to:
In order to address these pain points and allow networks to scale effectively to meet traffic projections, Cisco is advocating a new Routed Optical Networking architecture to bring about simplification in order to drive higher scale and lower TCO for IP services:
The Cisco Routed Optical Network architecture features:
Service providers are facing challenges in addressing the exponential traffic growth on their networks with flat Average Revenue Per User (ARPU). Traffic will continue to grow from video, gaming, and virtual/augmented reality as well as from the promises of mobility with 5G and future technologies. Service providers need to continue meeting the demand for this new capacity, while also rolling out new revenue generating services and lowering their costs. Looking at forecasts, a service provider will need to move 11 times more traffic per $1 of infrastructure investment in 2022 than in 2012.
Figure 1 – Cisco Visual Networking Index (VNI) vs SP Network Investment
Internet traffic has seen a compounded annual growth rate of 30 percent or higher over the last five years because more devices are connected, and more content is being consumed1.
Service providers recognize that the bulk of the services consuming this capacity are increasingly lower ARPU which is not commensurate with the cost of building the network infrastructure that can scale to support the capacity growing at an exponential rate. Technology continues to evolve following Moore’s Law, and we continue to push Shannon’s Limit. From operations we see a strong drive to leverage automation, Software-Defined Networking (SDN), telemetry, machine learning, and artificial intelligence to lower operational costs. From a network architecture perspective, we continue to see investments in individual network layers, from packet to Optical Transport Network (OTN) to Dense Wavelength Division Multiplexing (DWDM), with each layer evolving around capacity and flexibility in a siloed manner. This contributes to the cost and complexity of the overall network infrastructure.
As the tried and tested solutions to address these pain points have proven to be ineffective, a more disruptive approach is necessary to bring about the required level of network scale and efficiency to accommodate the anticipated huge exponential traffic growth. This transformative approach must be taken to:
This objective of this paper is to provide details on these transformative network architectures and solutions that form part of Cisco’s Converged SDN Transport initiative to facilitate efficient network growth for IP services. The Routed Optical Networking architecture is one component of the Converged SDN Transport Framework.
Traditional network architectures are comprised of networking layers that relied on line cards for service hand-off between the layers. This kind of layered architecture is highly inefficient as it consumes too much line card Capital Expenditure (CapEx) resources and relies on manual operations for service hand-off between the layers. Moreover, each networking layer has its own control and management planes associated with it, which operate independently from each other.
Figure 2 – Traditional Network Architecture
This creates huge complexities in service assurance, fault correlation, path optimization in terms of network utilization, as well as network planning and optimization. These complexities represent challenges to service providers’ aspirations towards achieving service driven end-to-end, closed-loop automation across the entire network infrastructure. The Total Cost of Ownership (TCO)2 associated with this network architecture is prohibitively high and will not allow service providers to scale their network to meet the capacity demands of IP services in a cost-efficient manner.
Cisco is committed to addressing these challenges by applying simplification through key technology that brings about true convergence of the IP and optical domains. Network simplification is aimed at removing complexities inherent to the infrastructure in order to allow service providers to leverage their assets more effectively through:
These initiatives are embodied in the Routing Optical Networking architecture which is a key enabler for the realization of Cisco’s vision for the Internet for the Future. This new approach transitions networks from the siloed infrastructure to a new architecture that relies on a single control plane based on IP/MPLS in a converged Hop-to-Hop (H2H) IP and optical network. This drives significant simplification and cost savings. It addresses the complexities and redundant networking layers that present bottlenecks to scalability and enables end-to-end automation in the service provider network infrastructure through:
Figure 3 – Cisco Converged SDN Transport Network Architecture
Transformative architectural changes are often disruptive to the present operational constructs of many service providers and thus involve a phased introduction. Service providers would take different paths to reach the envisioned Routing Optical Networking architecture based on the level of change they are willing to operationalize at Day 1.
Solution components of the Routed Optical Networking architecture can interwork with the ROADM-based optical network architectures that service providers have in their network infrastructure today. For example, the coherent DCO optics already supported on the current IP aggregation devices may be leveraged in ROADM-based infrastructure today. The Converged SDN Transport architecture is defined by two initiatives which address key inefficiencies in the network architectures of today:
1. Integration of 400G transponders function.
2. Service convergence via the integration of:
a. OTN aggregation and switching functions
b. Photonic switching
Figure 4 – Routed Optical Network Architecture Transformative Solution Components
Many service providers today are focused on the layer of the network to deliver a service rather than on the service itself. The focus on networking layers is largely driven by the assumption that specific Service Level Agreement (SLA) targets could only be achieved in specific networking layers where transport infrastructure could only leverage TDM services that required a distinct independent network silo/overlay. This essentially meant that:
Fundamentally, service providers were forced into a position of building layered networks rather than an ideal service-focused network. Multiple efforts are underway to shift away from this model by leveraging technological and organizational enhancements, resulting in:
Figure 5 – Routing Line Card Bandwidth Enabled by Optical Pluggable Modules
Architectural changes are required to bring about these benefits by reducing complexity, maximizing capacity, and avoiding inefficient networks. One might argue that SDN is built to address this complexity and hide from the operator; however, the complexity that SDN must overcome is directly related to the inherent complexity within the system. CDC ROADMS have delivered on networking flexibility in the photonic layer, but this flexibility comes at a cost of complexity in Operating Expenses (OpEx). Multi-layer networks add an additional dimension of complexity from an operational lifecycle perspective of planning, protecting, and managing systems..
In order to address the current pain points faced by service providers, we leverage the following principles as guidelines for building the next-generation networks:
The Routed Optical Networking architecture takes advantage of the technology advancements and new capacities being made available to deliver a network optimized around the service. It enables true network capacity optimizations while providing network simplification. This is accomplished by the collapsing of layers with reduced port-density trade-off optics and eliminating the complexity and redundant nature of layers that add unnecessary switching complexity to a network.
A fundamental impact requirement of optical bypass featured in both the Hollow Core (HC) and the Optimal Bypass (OB) architectures is that it requires leveraging ROADM switching elements in the optical network infrastructure to establish direct routes between routing devices (e.g. direct Provider Edge (PE) device to Provider (P) device) in order to avoid H2H optical connectivity between adjacent routing devices (e.g. PE to PE). This optical bypass feature leverages optical transit on ROADM elements as well as high performance coherent DSP. This ensures optical reach performance to establish feasibility for optical connections between routing devices over challenging (due to distance or quality or both) long-haul fiber infrastructure. This means the HC architecture is much more reliant on leveraging the most powerful coherent DSPs in order to bridge the gap between Moore’s Law and Shannon’s Limit4. As many service providers are aware, this trade-off leads to low fiber utilization (monetization). This is due to lower channel count and high network costs attributed to large volumes of regeneration introduced as necessary measures for maintaining performance over large distances. Having the large volumes of required regeneration defeats the primary purpose of the HC architecture, which is to minimalize the use of routing line cards as a transport function. In this case, the regeneration function is merely transferred from the routing device to the optical transponder line cards.
In contrast to the HC architecture, H2H architecture seeks to avoid the challenges posed by the trade-off between channel capacity vs. performance characterized by Shannon’s Limit. It avoids the need for optical connectivity between routing devices over long distances by advocating optical connectivity only between adjacent routing devices.
Table 1 – HC to OB to H2H Architecture Comparison
The H2H approach shows relative savings of 40% over a HC network and an approximate saving of 34% over an OB network. It is true that more interfaces are required, but the advantage of H2H architecture in a Converged SDN Transport network is the ability to leverage lower cost, with capacity optimized interfaces to offset the cost of additional interfaces while simplifying the operational lifecycle.
The Cisco Routed Optical Networking architecture takes advantage of advanced technology and features available today to build a new architecture that scales with technology, enabling a network that will:
With recent advances in packet switching silicon, Field Programmable Gate Array (FPGA) technology and stagnating innovation in the space of OTN framers, bit-transparent transport services are better delivered using emulation technology over a high capacity, scalable, and cost-effective packet network infrastructure. This solution, referred to as Private Line Emulation, allows bit-transparent point-to-point connections between a wide range of client ports which include OC-48, OC-192, 1GbE, 10GbE and 100GbE. It also allows for transparent Optical channel Data Unit (ODU-k) connections for Optical channel Transport Unit (OTU-k) client ports.
The connections are established using pseudo-wires in accordance with the Pseudo-Wire Emulation (PWE3) architecture and are leveraging an MPLS or Segment Routing (SR) underlay. This architecture can leverage SR transport with enhancements for circuit-style services to realize flexibility, efficiency gains, and simplicity of SR. This flexibility allows for a single, common SR/MPLS switching layer for private line services and any other carrier ethernet or IP service offering, reducing networking layers, complexity, and cost.
Topology Independent-Loop Free Alternatives (TI-LFA) is used to provide fast reroute protection to services for failure scenarios including link, node, and local Shared Risk Link Group (SRLG) by leveraging the post-convergence path. TI-LFA relies on SR to build a protection mechanism based on proven (IP-Fast Re-Route) IP-FRR concepts to extend sub-50ms protection to private line services, similar to the 1+1 protection schemes that are commonly reliant on being served from an OTN switching network layer.
There are two main concepts driving changes in the Routed Optical Networking architecture:
The direct integration of Digital Coherent WDM interfaces in the Router eliminates the traditional manually intensive service hand-off across the demarcation between the optical transport and packet domains. The result is a single network infrastructure that can be planned, designed, implemented, and operated as a single entity.
Network automation is a key element to plan, optimize, manage, and maintain all the network functions to enable a true SDN and drive network intelligence on an end-to-end basis. The real-time information of node state and condition is coupled with pre-determined trigger mechanisms for managing and optimizing service routes. The path computation, orchestration, and management toolkits that form the automation ecosystem are open, programmable, modular, operationally ready, and consistent with existing practices.
Figure 6 – Network Architecture Enabled Network Automation
The automation architecture includes unified capacity planning, path optimization, and element management for both IP and optical layers. It also includes topology and inventory visualization, service assurance, and closed-loop automation for proactive remediation powered by an end-to-end service orchestration and workflow engine.
In the past, direct termination of DWDM interfaces on the routing device to eliminate transponders required a coherent DSP to be implemented on the router line card, which occupies real estate on the line card. This resulted in reduced port density and capacity on the router line cards. With the new pluggable coherent optics that incorporate the coherent DSP on the optical pluggable modules instead of the host line card, the maximum router capacity can be maintained with little to no density tradeoffs.
A key pillar of the Routed Optical Network is the integration of the coherent pluggable modules. As router port densities increase, the CapEx spend transitions from the line card ports to the pluggable optics. A key enabler for the scale envisioned in the Internet of the Future is the 400GbE line rate. 400GE coherent optics leverage a multi-vendor Quad Small Form-Factor Pluggable (QSFP) with standardized specifications, which allows interoperability for easier adoption and gains of scale. Previous implementations relied on proprietary Digital Signal Processors (DSPs) on router line cards that were not interoperable or subject to standardization.
400GbE will be featured in the next generation routing line cards which can realize up to 14.4TBps on a single blade. In order to ensure full flexibility, these line cards will support both coherent and grey optical interfaces (or a combination thereof) in the form of QSFP56-DD form-factor pluggables. The use of QSFP56-DD for both coherent and grey optical interfaces can be leveraged to limit any tradeoff in terms of port densities and IP fabric capacity commonly associated with coherent optics on routing platforms (previously referred to as IPoDWDM). This means that a specific line card required to host coherent DWDM optical interfaces will no longer be required. A universal line card which can be flexibly deployed to support the termination of coherent DWDM, or grey optical line interfaces, can be realized. While the maximum number of 400G ZR/ZR+ ports (via QSFP56-DD) on a single coherent line card is limited by the optical power budget and characteristics associated with the line, the 400G ZR/ZR+ line card can still support significantly higher port densities than what is feasible on the previous generation CFP2-DCO-based coherent DWDM line cards.
QSFP form-factor has been widely leveraged in the industry and Cisco has been a key contributor in promoting QSFP-DD Multiple Supplier Agreement (MSA) through the Optical Internetworking Forum (OIF) as well as other standards organizations. QSFP56-DD is strategic in the realization of 400GbE and specifically for 400GbE ZR+. It is capable of dissipating the heat associated with the ZR+ interface as it incorporates the coherent transmission required to establish longer distance optical reach at the 400G line rate.
In the IP aggregation application space, IP/optical integration can already be served by leveraging IP aggregation devices which feature Modular Port Adapters (MPA) and Digital Coherent Optical (DCO) pluggable modules. The integration of DCO modules allows the direct termination of coherent DWDM interfaces directly into the IP aggregation devices without incurring the cost and complexity of optical transponder line cards. While the integration of CFP2-DCO modules into routing devices has already been leveraged to address key networking pain points in the IP aggregation space, the port densities associated with CFP2-based optics are too low compared with the QSFP pluggables employed in their grey optical line card counterparts to offer the similar TCO savings. This is primarily because in the IP aggregation space, typically a small number of 100G/200G line interfaces are terminated at each aggregation site which does not lead to significant deloading of the IP fabric. However, in the case of core/edge applications, the volume of traffic is significantly higher and therefore core/edge aggregation devices leveraging current DCO line cards are more prone to IP fabric deloading arising from lowered port density.
Figure 7 – Evolution of Pluggable Coherent Optics
Current port densities on core/edge routing devices are in the order of 3.6TBps per line card5. This level of port density scale on grey interface port line cards is a challenge to match with coherent line cards because current ACO/DCO implementations are limited to the CFP2 pluggables. These pluggables require a larger footprint compared with the QSFP modules leveraged in the grey interface line cards. Therefore, the use of CFP2-ACO/DCO line cards for the termination of significant volumes of traffic in core/edge applications could introduce significant levels of IP fabric deloading. This remains a temporary issue as we transition from the use of DCO interfaces from a CFP2 form-factor to one leveraging QSFP56-DD which offers a much smaller footprint that can also be leveraged for grey client interfaces.
2 Cost considerations commonly associated with the Capital Expenditures (CapEx) as well as the Operating Expenses (OpEx) of owning and operating the network
3 This is the maximum number of coherent 400Gbps channels that can be accommodated on a single line card. The actual number of coherent DWDM ports that can be accommodated is determined on the optical power budget imposed on the optical channels terminating on the line cards.
4 Moore’s Law vs. Shannon’s Limit: Moore’s Law is a representation of the service capacity growth trends in a service provider’s network whereas Shannon’s Limit defines the upper limit with regards to the capacity of the optical transport network to carry the service. An implication of Shannon’s Law is the trade-off between optical performance (reach characterized by optical signal to noise ratio) and error-free digital capacity that can be transported over the optical network; the shorter the optical reach (distance), the higher the digital channel capacity.
5 Cisco NCS 5500 36x100GbE Service Line Card