White Paper
TRANSITIONING METROPOLITAN-AREA NETWORKS FROM FRAME RELAY AND ATM TO ETHERNET AND MPLS USING CISCO 12000 SERIES ROUTERS
Carriers are rolling out more Ethernet networks in metropolitan areas in order to take advantage of Ethernet's price and performance and its ubiquity within their enterprise customer base. Ethernet provides an ideal foundation for new services such as Layer 3 VPNs and virtual private LAN service (VPLS). However, a large installed base of Frame Relay and ATM customers still remains, and it would be cost-prohibitive and disruptive to simultaneously move those customers to Ethernet. Carriers would rather allow individual customers to gradually migrate at their own pace, while continuing to benefit from the investment and revenue stream of ATM and Frame Relay equipment and services that are already deployed within their networks. A slower transition requires interoperability between the newer Ethernet-connected sites and the traditional ATM- and Frame Relay-connected sites. This paper describes some of the ways to achieve service interoperability and why the Cisco® 12000 Series routers are well suited for such a role.
Introduction
Increasingly, carriers are converging both their access networks and their core networks. This convergence results from the need to reduce operational expenses as well as offer new services over a single infrastructure. Within the core network, the migration toward an IP and Multiprotocol Label Switching (IP/MPLS) infrastructure has been in motion for some time. IP/MPLS is ideally suited for such migration because it offers the benefits of an IP-centric control plane while still being able to manage Layer 2 transport services such as ATM, Frame Relay, and Ethernet, as well as Layer 3 services such as VPNs based on the Border Gateway Protocol (BGP) and end-to-end IP-level quality of service (QoS) with Traffic Engineering capabilities. Furthermore, IP/MPLS offers a suite of redundancy mechanisms such as the ability to perform Fast Reroute around failed links and nodes, to maximize network availability, and minimize service disruption. More recently, carriers have also been considering the lower cost-per-bit of Ethernet to provide access to their enterprise customers. The standardization of Ethernet technology has helped provide the scalability and flexibility that carriers need, and boosted the availability of these capabilities in carrier-class platforms, commonly referred to as Carrier-Class Ethernet. This, combined with the cost and performance aspects of Ethernet and its near ubiquity in the enterprise, also make it appealing for metropolitan carrier networks, hence, making migration to Ethernet on access networks merely a matter of time.
However, despite this envisaged trend, a significant installed base of ATM- and Frame Relay-connected customers and networks, which grew in the 1990s, continues to exist within carrier networks today. Typically, these networks have been connected through point-to-point links to form a hub-and-spoke network topology. More importantly for carriers, these customers and networks also continue to provide a significant ongoing source of revenue, which carriers are reluctant to erode too aggressively, particularly while newer services and their associated business models fully mature, such as in the case of Ethernet. As a comparison, the estimated combined ATM and Frame Relay market size for carriers in North America alone was US$6 billion1 in 2004 compared with US$523 million2 for Carrier Ethernet services, despite Carrier Ethernet having a higher compound annual growth rate (CAGR) for the same period. In addition, the geographic reach of current Carrier Ethernet services may sometimes be inadequate for large enterprises that require nationwide or global coverage to meet current established ATM and Frame Relay services. Similarly, end-carrier customers that rely on these existing ATM and Frame Relay services are also very conscious of trying to balance the cost and performance benefits of moving to Ethernet-based access services, while at the same time minimizing service disruption to their existing networks and ensuring that their service-level agreements (SLAs) are maintained on the newer service.
There are many barriers to overcome when migrating access networks to Ethernet, perhaps comparable with the core convergence to IP/MPLS that has already been taking place. It is, therefore, not surprising that the widespread migration to Ethernet has not been as aggressive as initially predicted and may take the remainder of the decade or longer to be fully realized.
Therefore, carriers need to offer Ethernet services that fully interoperate with existing ATM and Frame Relay services so that customers can migrate gradually without severely impacting their existing infrastructure and, hence, business. This needs to occur while the characteristics of the existing ATM and Frame Relay network are maintained over an Ethernet access network and IP/MPLS core. This need has given rise to the multiservice edge (MSE) type of device that can perform this capability in an optimal manner. An MSE device can cost-effectively manage Frame Relay, ATM, and Ethernet services at carrier-class scalability and resiliency and transport these services over a common IP/MPLS core. The Cisco 12000 Series routers uniquely allow carriers to bridge this gap between Frame Relay/ATM services and emerging Ethernet services through a comprehensive set of interoperating capabilities that can be used over an existing IP/MPLS infrastructure in a cost-effective manner.
Multiservice Edge Challenge
One of the defining characteristics of an MSE-type device is its capability to handle high-density Ethernet aggregation while still offering competitive ATM and Frame Relay port densities and a well-developed feature set. Historically, traditional ATM and Frame Relay aggregation devices have lacked the overall architectural capacity to scale from 100-Mbps to 10-Gbps Ethernet densities cost-effectively, while traditional Ethernet switches have lacked the flexible feature set and interface-level support for ATM and Frame Relay services. These limitations required carriers to deploy multiple devices for each service type that they offered. This was further complicated by the need to consolidate and condition both types of traffic toward an emerging IP/MPLS-enabled core. Hence, an optimal MSE device needs to perform all of these functions and ideally bring the feature sets of both device types into a single, scalable, unified platform that can be managed effectively by the carrier.
Service Transparency
One of the first aspects of infrastructure convergence is for a carrier to consolidate all traffic types, that being primarily ATM, Frame Relay, Point-to-Point Protocol (PPP), High-Level Data Link Control (HDLC), and Ethernet/VLANs under a unified backbone technology and provide transport services for these at Layer 2 to maintain service transparency to its end customers. Increasingly, IP/MPLS is becoming the core transport medium of choice for this, although other Layer 2 tunneling technologies such as Layer 2 Tunneling Protocol Version 3 (L2TPv3) can also be used. IP/MPLS allows the carrier to use some of the strengths of MPLS, such as the connection-oriented infrastructure, Traffic Engineering capabilities, and network-resiliency features such as Fast Reroute (FRR) for transport of Layer 2 traffic, while maintaining the existing hub-and-spoke topologies of enterprise WANs. This transport is achieved by building point-to-point tunnels across the core IP/MPLS network between the provider-edge devices that interlink the customer-edge devices that need to communicate through the Layer 2 access protocols. This Layer 2 tunnel is commonly referred to as a pseudowire and is a specific instantiation of a Layer 2 VPN type referred to as Virtual Private Wire Service (VPWS). The corresponding provider-edge-to-customer-edge link or the attachment circuit and the Layer 2 protocol type that is carried over the pseudowire is generically labeled as the Pseudowire Emulated Service (PWES). Figure 1 illustrates this concept. Hence, it is imperative that an MSE device can map the full suite of Layer 2 access protocols in order to take full advantage of the core IP/MPLS network; this capability is generally referred to as Any Transport over MPLS (AToM). It is also beneficial to map the Layer 2 QoS characteristics to IP/MPLS Experimental Bit (EXP) fields by the MSE device to provide differentiated treatment where necessary. It is typically only with these types of capabilities that true service transparency can be maintained for the end customer.
Figure 1
Virtual Private Wire Service
Service Interoperability
As customers start to migrate toward Ethernet-based access services, some sites will continue to use ATM and Frame Relay connections. In order for a carrier to allow service transparency between these interconnected sites at Layer 2, full service interoperability between the various transport protocols needs to be available in an MSE device. Service interworking in IP/MPLS has evolved from similar earlier mechanisms such as FRF.8 (Frame Relay Forum) for FR to ATM service interworking, but in the generalized context refers to interworking of ATM, Frame Relay, Ethernet, PPP, and HDLC services. An effective MSE device needs a full set of interworking capabilities for these protocols in both bridged and routed encapsulation types between attachment circuits on the local provider-edge node, as well as with remote provider-edge nodes across the IP/MPLS core.
Reliability
An important consideration of network edge (or core) convergence for the carrier is reliability and availability. Specifically, customers that may have had services connected to multiple network elements or even networks on the carrier side will now be connected through a consolidated MSE device into one IP/MPLS-based network. The reliability and availability of the end-to-end service is an overriding concern because a failure of any intermediate link or node in the network has the potential to cause a much greater degree of impact than previously. Therefore, an MSE device needs to have well-developed service protection and restoration capabilities to effectively mitigate such failure scenarios. Although there are many individual capabilities that contribute to overall reliability and availability of a service, in the realm of IP/MPLS, a few that stand out are control-plane protocol convergence time, FRR performance, and Bidirectional-Forwarding Detection (BFD) implementation, which will be discussed in further detail.
Control-Plane Convergence
Control-plane convergence refers to the process and the time it takes for a protocol such as Border Gateway Protocol (BGP), Label Distribution Protocol (LDP), or any other Interior Gateway Protocol (IGP) to recalculate after any network topology change such as a node or link failure. It can be a critical factor in the overall service-restoration time from an end-to-end perspective. In the case of BGP, after an update has been received, the time taken for the best path operation is a direct consequence of the BGP table size based on the layout of the network. This can potentially add many minutes to the overall service-restoration time where it is based on BGP convergence. This dependency on BGP table size can be alleviated almost completely by the use of techniques such as prefix-independent rewrites within the MSE device's BGP implementation. Prefix-independent rewrite uses a separate table to store common adjacency information shared by multiple prefixes that are linked to it, as opposed to storing this information within every single entry in the forwarding table. By using this approach, when an adjacency change is necessary, it is a simple matter of updating the common adjacency information once, rather than the size of the BGP table, thus keeping the convergence time a constant, regardless of the size of the network topology involved.
In the case of IGP (Open Shortest Path First [OSPF], Intermediate System-to-Intermediate System [IS-IS], etc.) within an Autonomous System (AS), although there is no direct way around the Shortest Path First (SPF) compute time, certain optimizations in the implementation, such as prefix prioritization and fine-tuning of update generation and flooding timers, can also have a measurable impact on the overall service-restoration time. Because LDP typically relies on IGP convergence, this can impact Label Switched Path (LSP) setup where no prior backup paths may have been established. In most cases, however, it is prudent to have a restoration mechanism based on FRR available to manage link or node failures.
Fast Reroute Performance
FRR capability enables a protection mechanism for Traffic-Engineered tunnels in an MPLS network by provisioning backup tunnels that protect specific links and nodes along the path(s) taken by the primary tunnels. The primary advantage of this approach is that because the backup tunnels are established ahead of time, they are independent of IGP convergence when a failure occurs. However, the FRR database still needs to be recomputed based on the backup information and a rewrite would typically be performed on a per-protected-prefix basis. This approach has the disadvantage of linking the recompute time with the size of the network topology. A better approach is to write this information in a prefix-independent manner, precomputing the FRR database to make the rewrite time and the service restoration predictable. Such a mechanism would be necessary for an MSE device to provide optimum fault management through FRR in the case of a link or node failure in the downstream MPLS network regardless of its overall size. Figure 2 illustrates this mechanism and its operation.
Figure 2
Prefix-Independent versus Prefix-Dependent Rewrite Operation for BGP and FRR (changes highlighted)
Bidirectional-Forwarding Detection
BFD is a simple hello mechanism that detects network failures at the forwarding-plane level between two adjacent systems. It protects against control-plane failure of a node, which may unnecessarily impact the data-forwarding component, and also provides faster response times (less than one second) exceeding typical control protocol behavior such as OSPF hellos through the use of adaptive timers. BFD is independent of the underlying data link protocol and media levels. A robust BFD implementation, therefore, must be distributed to run on the actual line cards of an MSE device rather than on the route processor where other control protocols would typically reside. This helps ensure that the desired characteristics of BFD are met independent of load conditions of the route processor. BFD can be used to provide subsecond restoration times for IGP protocols, MPLS LSPs, etc. and significantly helps to reduce the overall service-restoration time when used in combination with some of the other methods discussed so far.
VPLS and BGP/MPLS Migration
Thus far, this paper has discussed ways to migrate from ATM and Frame Relay to Ethernet, and how interworking and transport technologies such as AToM can significantly help this migration. Because ATM and Frame Relay are point-to-point technologies, this paper has focused on point-to-point services. However, Ethernet is inherently a broadcast technology that is designed to provide multipoint-to-multipoint connectivity. In large network deployments, hub-and-spoke topologies tend to result in suboptimal traffic routing due to the increased reliance on the hub site, and when combined with increased peer-to-peer traffic from newer applications, may result in increased network latency and congestion, as well as an exposure should a failure occurs at the hub site. For this reason, as ATM/Frame Relay-to-Ethernet migration accelerates, more enterprise customers will want multipoint-to-multipoint connectivity similar to their own enterprise LANs. Two leading scalable technologies are available today for carriers to provide such multipoint service connectivity beyond the metropolitan-area network (metro): Virtual Private LAN Service (VPLS) for Layer 2 VPN services and BGP/MPLS for Layer 3 VPN services. VPLS alleviates many of the previous limitations of similar technologies, such as the limited number of Virtual LAN IDs, susceptibility to Spanning Tree Protocol scalability issues, broadcast storms, etc. while using an IP/MPLS infrastructure at the network core. This provides the carrier a way to extend the metro's reach by extending multipoint Layer 2 VPNs beyond the metro into the WAN. The other leading multipoint VPN solution is BGP/MPLS, which provides a scalable, flexible, and extensible Layer 3 solution. A detailed discussion of these two VPN types is beyond the scope of this document, but there is a clear advantage to having an MSE device that fully supports these services because it helps carriers to use the same infrastructure for further service offerings to their customers.
The Cisco 12000 Series Router Advantage
To effectively support migration from ATM and Frame Relay services to Ethernet, carriers need a platform with the following characteristics:
• Feature-rich in terms of all three access technologies, with high-density Ethernet aggregation capacity
• Capable of transporting Layer 2 protocols over an IP/MPLS core though point-to-point connections such as AToM
• Interoperable to gradually support migration to Ethernet-based services while continuing to support existing ATM and Frame Relay customers
• Supports well-developed service protection and restoration capabilities, such as fast control-plane protocol convergence mechanisms, superior FRR performance capabilities, and BFD for subsecond failover
• Support scalable VPLS and BGP/MPLS VPN services to allow flexible migration to multipoint VPN connectivity when required
The Cisco 12000 Series routers, with their broad Cisco IOS® Software feature set, edge-optimized line cards, and enhanced route processors, are the ideal MSE devices to support Frame Relay and ATM migration to Ethernet services. The Cisco 12000 Series routers currently offer:
• Two-fold increase in Ethernet port density with the recent announcement of the 10-Gbps modular IP Services Engine (ISE) SPA Interface Processor (SIP) with intelligent oversubscription and queuing that can support both high-density, low-speed ATM/Frame Relay and high-speed Ethernet interfaces at cost points comparable to most Ethernet switches in the market. The SIP supports a full set of features such as Layer 2 and Layer 3 QoS, IPv6, Multicast, Netflow v9 accounting, and IP/MPLS at 10-Gbps line rates.
• Extensive support for Layer 2 protocol transport over IP/MPLS through Any Transport over MPLS (AToM). Support for protocols and encapsulations including Ethernet, ATM, Frame Relay, ATM/AAL5, ATM/Cell Relay, PPP, and HDLC.
• Flexible interoperability between protocols with support for the following: Ethernet to Frame Relay - bridged interworking, Ethernet to Frame Relay - routed inter-working, Ethernet to ATM - bridged interworking, Ethernet to ATM - routed interworking, Frame Relay to ATM - FRF.8-like services, Frame Relay to PPP, Frame Relay to HDLC.
• Extensive support for service protection and accelerated restoration through the support for prefix-independent rewrites to help ensure constant convergence times for Border Gateway Protocol (BGP) and Traffic Engineering Fast Reroute, prefix prioritization and fine-tuning of update-generation and flooding timers in the case of Interior Gateway Protocol (IGP), and distributed Bidirectional-Forwarding Detection (BFD) support on the line cards.
• Well-developed VPLS feature set with support for multipoint-to-multipoint connectivity in either transparent or nontransparent operation, circuit multiplexing, IEEE 802.1Q tunneling (Q-in-Q), MAC address-based forwarding, Jumbo Frames, MAC address-based rate limiting, dual homing, and failure-recovery mechanisms among many others.
• Feature-rich and industry-leading scalable BGP/MPLS VPN support suite.
Conclusion
The multiservice edge (MSE) devices in carrier networks are fast becoming critical components to enable the profitable transition to cost-effective Ethernet services for the carrier while facilitating migration of the large installed base of ATM and Frame Relay customers. MSE devices provide a means to consolidate the access aggregation layer, thereby reducing ongoing operational expenses for the carrier while offering a set of new services such as VPLS and BGP/MPLS that take advantage of an IP/MPLS core infrastructure. Furthermore, MSE devices provide a robust set of service protection and restoration capabilities to help carriers maintain existing SLAs with their end customers. The industry-leading Cisco 12000 Series routers with Cisco IOS Software deliver a powerful solution for carriers to address these needs today and well into the future.
1Source: Ovum, World Wide Frame & ATM Service Connections and Revenue (2004)
2Source: Infonetics Research, Ethernet Services - Annual Worldwide Market Size and Forecasts (April 2005)