Pioneering the IP and Optical Transformation

Traffic is overwhelmingly IP

Communication service providers (CSPs) are working to balance their need to expand network capacity while controlling costs in the face of flat revenues. Traffic will continue to grow from video, gaming, and virtual/augmented reality as well as from the promises of mobility with 5G and future technologies. CSPs need to continue meeting the demand for this new capacity, while also rolling out new revenue generating services and lowering their costs. Looking at forecasts, a CSP will need to move 11 times more traffic per $1 of infrastructure investment in 2022 than in 2012.

Figure 1. Cisco Visual Networking Index (VNI) vs CSP network investment

Routed optical networking architecture

Cisco is committed to addressing these challenges by simplifying key technologies that enable convergence of the IP and optical domains. Routed optical network simplification is aimed at removing complexities inherent in the current architectural designs and multiple layers to allow CSPs to leverage their assets more effectively.

The routed optical networking architecture framework is a key enabler for the realization of the Cisco Internet for the Future vision. This new approach transitions networks from the siloed infrastructure to a new architecture that relies on a single control plane based on IP/MPLS in a converged router-to-router IP and optical network. This drives significant simplification and cost savings. It addresses the complexities and redundant networking layers that present bottlenecks to scalability and enables end-to-end automation in the CSP network infrastructure through:

• Assimilation of any OTN switching infrastructure required to address any legacy Time-Division Multiplexing (TDM) services
• Direct integration of high-capacity optical interfaces (for both grey and coherent) directly on the routing devices without the compromise of the deloading of the IP fabric that was present in Cisco IP over Dense Wavelength-Division Multiplexing (IPoDWDM)
• Full core router-to-router IP routing architecture is characterized by a single networking/switching layer in the IP domain and simple point-to-point optical infrastructure without the cost and complexity of colorless directionless contentionless (CDC) reconfigurable optical add-drop multiplexers (ROADMs)
• Single, unified transport SDN across IP/routing and optical transport infrastructure for:
• Unified capacity and network planning
• Path optimization
• Service assurance, inventory, and element management
• Closed-loop automation

Figure 3. Cisco Routed Optical Network Architecture

This architectural design provides the opportunity for significant cost savings but such a transformative architectural change could present a challenge to the operational constructs of many CSPs and thus involve a phased introduction. CSPs would take different paths to reach the envisioned routed optical networking architecture, and the Cisco Customer Experience (CX) team can help. The CX team has developed network models that can outline different levels of potential cost savings based upon the level of transformation adopted. Additionally, the CX team operates the Routed Optical Networking Architecture Transformation Advisory service that will help CSPs plan, validate, and implement the solution so they can achieve the expected savings.

Solution components of the routed optical networking architecture can interoperate with the ROADM-based optical network architectures that CSPs have in their network infrastructure today. For example, the coherent DCO optics already supported on the current IP aggregation devices may be leveraged in ROADM-based infrastructure today. The routed optical networking architecture is defined by two initiatives which address key inefficiencies in the network architectures of today:

• Integration of 400G transponders function
• Service convergence via the integration of OTN aggregation and switching functions and photonic switching

Figure 4. Routed Optical Network Architecture Transformative Solution Components

Routed optical networking architecture for service convergence

Services are key to any communication service provider’s business. Assuring that all services can be provisioned quickly for faster revenue and can ride over the IP network with no additional latency, with the assurance that those services are visible for management, are key to this new environment.

• Private-line services were previously a challenge to transport
• Bi-directional, deterministic services could only be provided via a network layer
• OAM was provided from a network layer perspective
• SLAs were based on the network layer rather than service availability

Fundamentally, CSPs were forced into a position of building layered networks rather than an ideal service-focused network. Multiple efforts are underway to shift away from this model by leveraging recent technological and organizational enhancements. Some examples are:

• CO modernizations where legacy TDM traffic is carried over the packet network with new efforts to move legacy services to private line emulation are now possible.
• Routing capacity and scale has massively increased and is being delivered in smaller footprints. Capacity that used to require a rack of gear is now available in 1 RU or 2 RU configurations. The Cisco 8000 series routers can provide up to 24.6 Tbps in a 1 RU platform.
• Universal line cards are used where it is possible to mix and match grey and colored (400G ZR/ZR+) optical interfaces by leveraging innovation in optics to realize 400GbE clients in a QSFP-DD form-factor. This greatly reduces exposure to port density trade-off commonly associated with line systems and facilitates integration on the service-routing platform.
• Increased flexibility is realized, with multiple set points and multiple degrees of freedom on the optical front. Integration of automation tools provides the operator with assurances for connectivity and performance with an ability to focus root-cause alarms on a problem, whether it be in the router (transceiver), line terminating equipment, intermediate amplifier, or in the outside plant fiber itself.
• Simplified and streamlined operational models enabled by automation and life-cycle management toolkit allow the network to be treated as one rather than as many stitched together.

Figure 5. Routing Line Card Bandwidth Enabled by Optical Pluggable Modules

Architectural changes are required to bring about these benefits by reducing complexity, maximizing capacity, and avoiding inefficient networks. Routed optical networking allows CSPs to operate a network that has a converged optical and IP layer that increases visibility and control while reducing costs. The convergence of the optical and IP layers can eliminate the dedicated OTN layer that carries private line services, or providers can leave the OTN and Muxponder network in place and grow new services on the IP layer and cap the OTN/TDM already in place. Services need to get from ingress to egress, meet SLAs, and be as cost efficient as possible while reducing carbon footprint. By focusing on the service layer, we can leverage PLE plus 400G ZR/ZR+ optics and automation enhancements together to converge layers, simplify the network, and enhance the lifecycle management of the network.

Converging private line services onto the packet layer may raise some concern around service latency. The concern arises from running critical TDM services over a packet network which traditionally meant introducing latency and reducing reliability. However, packet networks are no longer slow, nor do they incur latency, thanks to the hardware-based packet forwarding engines used in today’s networks. Latency can be controlled by using strict bandwidth accounting (RSVP-TE) and/or quality of service (QoS) mapping to strict priorities, ensuring no additional latency is added to the network.

To address the current pain points faced by CSPs, we leverage the following principles as guidelines for building the next-generation networks:

• Converge layers:
• Simplify operational lifecycle
• Reduce power and footprint
• Advanced circuit emulation for legacy services and high-speed private line and OTN services:
• Network simplification with single-layer network
• Meet existing SLAs
• Provide dedicated Bandwidth (BW) and predictable paths
• Integrate optics at significantly reduced port-density trade-off:
• 400G price points are changing the game
• Smaller form factors enabling huge reductions of up to zero density trade-offs relative to grey optical line card port density
• Path to gain even greater capacity
• Move from a ROADM-based architecture to H2H architecture:
• Gracefully transition from bypass to H2H networks
• Provide optimal capacity by shortening end point distances
• Operational simplification
• Simplify lifecycle management

Routed Optical Networking architecture takes advantage of the technology advancements and new capacities being made available to deliver a network optimized around the service. It enables true network capacity optimizations while providing network simplification. This is accomplished by the collapsing of layers with reduced port-density trade-off optics and eliminating the complexity and redundant nature of layers that add unnecessary switching complexity to a network.

Hollow core versus hop-to-hop comparison

A fundamental requirement of optical bypass featured in both hollow core (HC) and optimal bypass (OB) architectures is that it requires leveraging ROADM switching elements in the optical network infrastructure to establish direct routes between routing devices (e.g. direct provider edge (PE) device to provider (P) device) in order to avoid H2H optical connectivity between adjacent routing devices (e.g. PE to PE). This optical bypass feature leverages optical transit on ROADM elements as well as high performance coherent DSP. This ensures optical reach performance to establish feasibility for optical connections between routing devices over challenging (due to distance or quality or both) long-haul fiber infrastructure. This means the HC architecture is much more reliant on leveraging the most powerful coherent DSPs to bridge the gap between Moore’s Law and Shannon’s Limit. As many CSPs are aware, this trade-off leads to low fiber utilization (monetization). This is due to lower channel count and high network costs attributed to large volumes of regeneration introduced as necessary measures for maintaining performance over large distances. Having large volumes of required regeneration defeats the primary purpose of the HC architecture, which is to minimalize the use of routing line cards as a transport function. In this case, the regeneration function is merely transferred from the routing device to the optical transponder line cards.

In contrast to the HC architecture, routed optical networking architecture seeks to avoid the challenges posed by the trade-off between channel capacity vs. performance characterized by Shannon’s Limit. It avoids the need for optical connectivity between routing devices over long distances by advocating optical connectivity only between adjacent routing devices.

Table 1. HC to OB to H2H Architecture Comparison

The Cisco Routed Optical Networking architecture takes advantage of advanced technology and features available today to build a new architecture that scales with technology, enabling a network that will:

• Decrease costs:
• Models have shown as much as 75% savings
• Reduction in power consumption and equipment footprint via integration and elimination of redundant network layers
• Network simplification featuring:
• Single layer router-to-router network
• Simplified planning, design, activation, management, and troubleshooting
• Meet and exceed existing SLAs:
• Circuit emulation to address legacy TDM and private line services
• Reduction in components, increasing availability and mean time between failure
• Improvement on time to market of new revenue-generating services enabled by:
• Service-focused network architecture
• Integrated automation and telemetry functionality
• Optimize fiber capacity utilization via:
• Less exposure to Shannon’s Limit by decreasing the un-regenerated distance and increasing the spectral efficiency (transported bit per symbol)

OTN and private line service emulation

With recent advances in packet switching silicon, Field Programmable Gate Array (FPGA) technology and stagnating innovation in the space of OTN framers, bit-transparent transport services are better delivered using emulation technology over a high capacity, scalable, and cost-effective packet network infrastructure. This solution, referred to as Private Line Emulation, allows bit-transparent point-to-point connections between a wide range of client ports which include OC-48, OC-192, 1GbE, 10GbE and 100GbE. It also allows for transparent Optical channel Data Unit (ODU-k) connections for Optical channel Transport Unit (OTU-k) client ports.

The connections are established using pseudo-wires in accordance with the Pseudo-Wire Emulation (PWE3) architecture and are leveraging an MPLS or Segment Routing (SR) underlay. This architecture can leverage SR transport with enhancements for circuit-style services to realize flexibility, efficiency gains, and simplicity of SR. This flexibility allows for a single, common SR/MPLS switching layer for private line services and any other carrier ethernet or IP service offering, reducing networking layers, complexity, and cost.

Topology lndependent-loop free alternatives (TI-LFA) are used to provide fast reroute protection to services for failure scenarios including link, node, and local shared risk link group (SRLG) by leveraging the post-convergence path. TI-LFA relies on SR to build a protection mechanism based on proven IP-fast reroute (IP-FRR) concepts to extend sub-50ms protection to private line services, like the 1+1 protection schemes that are commonly reliant on being served from an OTN switching network layer.

There are two main concepts driving changes in the routed optical networking architecture:

• Direct integration of digital coherent WDM interfaces in the router
• Photonic infrastructure simplification by reducing reliance on ROADM

The direct integration of Digital Coherent WDM interfaces in the Router eliminates the traditional manually intensive service hand-off across the demarcation between the optical transport and packet domains. The result is a single network infrastructure that can be planned, designed, implemented, and operated as a single entity.

Multi-layer automation as part of the routed optical networking solution

Automation gives you the speed to execute operations with known repeatable quality, analytics for better decision making, and a way to synchronize business decisions with application processes. Speed gives the advantage of faster service provisioning for reduced time to revenue, faster decisions for proactive planning and problem avoidance, and increased responsiveness to protect the customer experience. Automation is truly about looking at the process; taking it to machine speed and executing it at a completely repeatable level and do it statefully, so that the outcome is exactly what is intended. The step to automate processes brings visibility, insights, and actions to the network in a fully autonomous and ideally closed-loop fashion. This is done with Cisco Crosswork Network Automation, a portfolio of tools to simplify and normalize repeatable operations.

Automation is essential to the routed optical networking solution. Reducing workloads, positively impacting network performance and the client experience, provides a competitive and differentiated network experience. Few competitors can deliver automation on a multi-layered network - controlling both IP and optical with specific domain controllers. Controllers with microservices essentially manage the specific domains. They have the topology, discovery, inventory, southbound alarming, and device profiles. And since it is a multi-domain world, there are multiple vendors in each networking domain that must be considered. Thus, you need a hierarchical controller to manage across those domains to bring visibility, insights, and actions aggregated at a higher level. The IP and optical controllers show a more detailed level of granularity of the network. The multi-layer HCO provides a service level view and service assurance as it looks across domains.

Figure 6. Architecture-enabled network automation

As these networks come together, with routed optical networking and coherent pluggables in the routers, there is the need to have a common way to operationalize this. This is partly why IPoWDM failed 20 years ago, not because of the technology, but because it’s about bringing the operations together in a way that allows optical and IP teams to work together to deliver a service, assure it, and visualize it for the customer. That’s what multilayer automation brings to the solution.

The multilayer controller (HCO) provides end-to-end optimization and visibility. It effectively delayers the network and manages the services with dedicated controllers by talking directly to both CNC (IP controller) and ONC (optical controller) and any third-party network. The multi-layer automation architecture provides visualization, connectivity, and event awareness. It is vendor agnostic and communicated through open and standard service models. It is a solution for routed optical networking as well as for routing and optical platforms outside the solution, and is an industry standard solution.

Here are the elements of Cisco’s Crosswork Automation:

• Hierarchical Controller (HCO)
• Multi-vendor IP Controller (CNC)
• Cisco Optical Networking Controller (CONC)
• Element Management (EPNM)
• Cisco Optical Network Planning tool (CONP)

Figure 7 shows how Cisco is bringing it all together.

Figure 7. HCO’s function is an abstract hierarchical controller to stitch IP and optical layers

IP/optical convergence through optical transponder integration

In the past, direct termination of DWDM interfaces on the routing device to eliminate transponders required a coherent DSP to be implemented on the router line card, which occupies real estate on the line card. This resulted in reduced port density and capacity on the router line cards. With the new pluggable coherent optics that incorporate the coherent DSP on the optical pluggable modules instead of the host line card, the maximum router capacity can be maintained with little to no density tradeoffs.

A key pillar of routed optical networking is the integration of the coherent pluggable modules. As router port densities increase, the CapEx spend transitions from the line card ports to the pluggable optics. A key enabler for the scale envisioned for the Internet for the Future is the 400G line rate. 400G coherent optics leverage a quad small form-factor pluggable (QSFP) with standardized specifications, which allows interoperability for easier adoption and gains of scale. Previous implementations relied on proprietary digital signal processors (DSPs) on router line cards that were not interoperable or subject to standardization.

800G and beyond will be featured in the next generation routing line cards which can support up to 24.6Tbps on a single blade. To ensure full flexibility, these line cards will support both coherent and grey optical interfaces (or a combination thereof) in the form of QSFP-DD form factor. The use of QSFP-DD for both coherent and grey optics can be leveraged to limit any tradeoff in terms of port densities and IP fabric capacity commonly associated with coherent optics on routing platforms (previously referred to as IPoDWDM). This means that a specific line card required to host coherent DWDM optical interfaces will no longer be required. A universal line card which can be flexibly deployed to support the termination of coherent DWDM, or grey optical line interfaces, can be realized. While the maximum number of 400G ZR/ZR+ ports on a single coherent line card is limited by the optical power budget and characteristics associated with the line, the 400G ZR/ZR+ line card can still support significantly higher port densities than what is feasible on the previous generation CFP2-DCO-based coherent DWDM line cards.

QSFP form-factor has been widely leveraged in the industry and Cisco has been a key contributor in promoting QSFP-DD multiple supplier agreement (MSA) through the Optical Internetworking Forum (OIF) as well as other standards organizations. QSFP-DD is strategic in the realization of 400GbE and specifically for 400G ZR+. It can dissipate the heat associated with the ZR+ interface as it incorporates the coherent transmission required to establish midhaul optical reach at the 400G line rate.

Evolution of Digital Coherent Optics

In the IP aggregation application space, IP/optical integration can already be served by leveraging IP aggregation devices which feature Modular Port Adapters (MPA) and Digital Coherent Optical (DCO) pluggable modules. The integration of DCO modules allows the direct termination of coherent DWDM interfaces directly into the IP aggregation devices without incurring the cost and complexity of optical transponder line cards. While the integration of CFP2-DCO modules into routing devices has already been leveraged to address key networking pain points in the IP aggregation space, the port densities associated with CFP2-based optics are too low compared with the QSFP pluggables employed in their grey optical line card counterparts to offer the similar TCO savings. This is primarily because in the IP aggregation space, typically a small number of 100G/200G line interfaces are terminated at each aggregation site which does not lead to significant deloading of the IP fabric. However, in the case of core/edge applications, the volume of traffic is significantly higher and therefore core/edge aggregation devices leveraging current DCO line cards are more prone to IP fabric deloading arising from lowered port density.

Figure 8. Evolution of pluggable coherent optics

Convergence helps save the planet

Environmental sustainability is top of mind at Cisco as we work to improve our impact on the planet. Our efforts began with designing massively scalable systems that provide more features in smaller footprints. Smaller devices allow CSPs to reduce their real estate, reducing power consumption and leading to a lower carbon footprint. Cisco is also focused on recycling and reusing components and materials from our older systems to be less reliant on precious resources, and we are working through our supply chain to reduce packaging and integrate more sustainability sourced materials.

Convergence of the IP and optical layers further reduces the number of devices, positively impacting power consumption and reducing truck rolls. Network modeling of the Cisco Routed Optical Networking solution benefits shows up to 45% power reduction and up to 70% real estate savings. The enhanced programmability of the network through automation and new packaging of optics and other platforms to use less material can help our customers lower carbon dioxide (CO2) emissions and achieve their overall sustainability goals.

Endnotes

¹ Cisco Annual Internet Report (2018–2023) White Paper

² Cost considerations commonly associated with the Capital Expenditures (CapEx) as well as the Operating Expenses (OpEx) of owning and operating the network