Cisco Port Adapters

Scaling the Routed WAN

White Paper

Scaling the Routed WAN

New Solutions for Enterprise Central Site WAN Connectivity


With the rapid growth of IP and the Internet, enterprise WAN networking is undergoing rapid change. New service offerings, such as Frame Relay and Asynchronous Transfer Mode (ATM), provide new options to complement existing leased-line private networks. The increasing volume of data traffic, increasingly based on IP, has shifted the WAN from traditional time-division multiplexers to a routed architecture. Concurrently, voice over IP, voice over Frame Relay, and voice over ATM enable data/voice integration on these new services. However, recurring service cost is still the dominant issue in enterprise WAN budgets, with labor costs to operate and manage the WAN also a significant issue.

As the leader in routed WAN and IP-based networking for enterprise networks, Cisco brings new capabilities to enterprise routed WAN networks to support new services, reduce costs, and enable enterprise WAN networks to grow with the demands of enterprise network applications. This white paper briefly discusses the evolution of enterprise WAN networking and describes how enterprise network managers can utilize the new multichannel networking capabilities, advanced routed software services, and new features on the Cisco 7500 and 7200 routers.

The Evolution of WAN Networking

The WAN services market is large and is growing rapidly. With the rapid growth of the Internet and IP/intranet applications in enterprise LANs, IP-based WAN services are growing at a Compound Annual Growth Rate of 73 percent. This scenario is placing an increasing spotlight on the routed WAN infrastructure in enterprise WANs and a focus on IP-based networking services and software features. In addition, the performance increases in the LAN are driving up the speeds (and costs) of WAN connections for enterprises. Figure 1 illustrates this rapid growth in the WAN services market.

Figure 1: Growth in US WAN Services Market

It is expected that leased-line services will continue to make up a very large part of the market, with Frame Relay also growing at a rapid rate. (See Figure 1.) Many different service offerings will continue to coexist, as each has advantages in different applications with different cost versus control tradeoffs.

Leased Lines

Leased lines are the largest WAN service category today. These services provide dedicated, private bandwidth with low and fixed delay characteristics, with speeds from 56 kbps through 155 Mbps Synchronous Optical Network/Synchronous Digital Hierarchy (SONET/SDH) fiber. Leased-line services provide a good choice for networks that require a proven service with a high degree of control over the network, a common requirement in large enterprise WAN networks. This service type is also commonly used as the "last-mile" technology to access other services (for example, Internet or Frame Relay). It is the most expensive WAN service alternative in dollars per bits per second, but it also provides the greatest network control.

Frame Relay

Frame Relay provides virtual circuit connectivity for enterprise networks that require 56 kbps up to T1/E1 speeds. It costs less than leased lines because it uses statistical multiplexing of packets to gain efficiencies within the network, at the cost of a less-stringent bandwidth and latency guarantee. Frame Relay is being widely deployed in enterprise networks to connect regional and branch offices into the enterprise backbone.


Like Frame Relay, ATM provides virtual circuit connectivity but at higher speeds (DS3/E3 up to OC-3/Synchronous Transport Module level 1 (STM-1) today, with support for T1/E1 in the future). It is capable of higher quality of service (QoS) than Frame Relay and is used today to connect enterprise WAN and MAN backbone sites.


This networking technology is still applicable in low-speed environments where circuit noise is a problem. It is typically used in international networks in places where the network infrastructure cannot yet support Frame Relay or ATM.

IP Access

IP Access is the fastest growing WAN service and today is primarily used for connections to the public Internet. Increasingly, however, enterprise network managers are looking to IP access as a way to outsource some of the management of the enterprise WAN intranet. New IP protocol technologies for security, QoS, and virtual private networking will enable IP access to become a viable option for leveraging public data networks for private enterprise WANs.

Enterprise networks have traditionally combined multiple applications on time-division multiplexing (TDM)-based leased-line WAN backbones. For the same cost-saving reasons, they want to achieve this same integration across newer packet- and cell-based WAN services. The key to supporting this data/voice/video integration is support for QoS in the service offering and in the enterprise WAN router that connects to the WAN service. This has been a characteristic of Cisco routers for T1/E1 speeds (for example, support for priority queuing and custom queuing), but is increasingly needed at higher speeds.

Finally, it is worth noting that recurring bandwidth costs still dominate the budgets of enterprise WAN networks. Leased lines are still a dominant technology for WAN networking today, providing control and guaranteed bandwidth to enterprise customers. As enterprise WAN networks grow, more cost-effective leased-line termination solutions are required for the enterprise WAN router. In addition, increasingly sophisticated software and services in WAN routers that can manage bandwidth and measure, analyze, and account for traffic is necessary to improve the operational efficiency of the WAN, thereby lowering costs.

These trends lead to the following requirements for enterprise WAN equipment:

  • Multiple WAN services: WAN devices need to provide support for a broad range of WAN services, enabling the enterprise to pick the best price and capabilities match to their requirements, whether it is leased-lines, IP access, Frame Relay, ATM, X.25, or a combination of services.

  • Speeds greater than T1/E1: WAN equipment needs to scale with the increasing performance demands of enterprise networks and the capabilities of WAN services. This scenario is obviously important for ATM networks, supporting up to OC-3/STM-1 speeds, but is important for leased-line backbones as well.

  • Bandwidth Control: In order to combine multiple applications with differing requirements on the same WAN service, the service and the enterprise WAN equipment must provide integrated and compatible priority and QoS capabilities. This QoS capability must also support high-speed WANs as well as a range of WAN services to meet the increasing performance requirements of enterprise intranets.

  • Reduce equipment cost: In many enterprise WAN solutions today, several pieces of equipment are required to deliver the solution. WAN equipment must integrate more functions into a single platform and provide higher-density WAN interfaces to save costs.

  • Reduce management cost: WAN equipment must reduce the complexity of building enterprise WAN networks and provide improved management visibility into the network and its operation.

Today, the Cisco 7500 router is the strategic high-end router for both collapsed backbone LAN and enterprise WAN applications in most of the world's enterprise networks. Over the next few years, multilayer switching technologies and products, such as those developed by Cisco for the Catalyst family of LAN switches, will provide increased Layer 3 performance in campus LANs to address the needs of new Fast Ethernet and Gigabit Ethernet infrastructures. The Cisco 7500 provides the evolving Enterprise LAN, with its shift to fewer media types, with seamless integration of existing multiprotocol and multiple media networks such as Fiber Distributed Data Interface (FDDI), Token Ring, and ATM. It also provides integration into the data center with channel-attached Enterprise System Connection (ESCON) and bus and tag connections to the mainframe. In addition, the Cisco 7500 remains an outstanding LAN solution for those networks that do not yet need the higher performance of multilayer switches, or that have a conservative posture regarding the adoption of new technologies.

As this LAN shift occurs, the Cisco 7500 and 7200 routers will retain their strategic WAN role in enterprise networks. To meet the new requirements of service flexibility, high-speed connectivity, bandwidth management, and cost reduction in the enterprise route WAN, Cisco is significantly enhancing the capabilities of the Cisco 7500 and 7200 routers. The next sections describe how these new enhancements meet the emerging requirements of the new enterprise routed WAN network.

Cisco Solutions for the Routed WAN

To meet the new requirements of service flexibility, high-speed connectivity, bandwidth management, and cost reduction in the enterprise route WAN, Cisco significantly enhanced the capabilities of the Cisco 7500 and 7200 routers. These new capabilities enable the Cisco 7500 and 7200 to provide large-site WAN connectivity for enterprise networks, and are part of an ongoing plan to deliver industry-leading routed WAN solutions. These capabilities are in four areas:

  • Multichannel networking---A new technology for leased-line and IP access networks that reduces cost and complexity, and adds flexibility and even greater port density to the WAN capabilities of the Cisco 7500 and 7200

  • ATM---A major enhancement to the traffic shaping and QoS capabilities of the Cisco 7500/7200 ATM interfaces

  • Advanced services---High-performance QoS and bandwidth management capabilities that give enterprises enhanced abilities to control and measure use of scarce WAN bandwidth and resources

  • Platform enhancements---Increases in the performance, scalability, and flexibility of the Cisco 7500 and 7200 routers

Multichannel Networking

Leased-line WAN networks provide a dedicated private line connection between pairs of sites, where control of the bandwidth is a key requirement. There are typically large numbers of connections to a central router to many remote sites. At the central site, each connection to a remote site has required a separate router port and a separate data service unit/channel server unit (DSU/CSU), or other external line termination device, resulting in high equipment cost and management complexity.

Cisco simplified this problem with the introduction of the Channelized T3 Interface Processor (CT3IP) in 1997. With the CT3IP, 28 individual 1.544-Mbps DS1 connections to remote sites are multiplexed onto a single DS3 channel connected directly to a Cisco 7500 router at the central site. With this solution, no external CSU/DSUs are required to terminate either the individual DS1 circuits or the DS3 line. The high level of integration provided by this card allows enterprise customers to vastly reduce operational complexity because of the removal of duplicated equipment and high cable count.

More importantly, because the pricing break-even crossover from DS1 to DS3 connectivity occurs at between six and eight DS1 lines, using Channelized DS3 to aggregate 28 DS1 customers can result in saving as much as 60 percent off DS1 transmission bills. With the CT3IP, no external M13 multiplexer is required to achieve this significant cost saving. Figure 2 illustrates leased-line WAN networking with and without channelization.

Figure 2: Networking with T3 Channelization

In the new multichannel networking interfaces, Cisco has built on the capabilities found in the CT3IP by significantly increasing port density and functional integration. These single-wide port adapters for the Cisco 7500 and 7200 provide channelization all the way down to the DS0 (64 kbps) channel level, T1/E1 support, Integrated Services Digital Network Primary Rate Interface (ISDN PRI) capability, and fully integrated CSU/DSUs. This high level of integration reduces equipment costs and management complexity, shrinks required rack space, simplifies sparing, and enables users to better manage WAN costs.

By supporting up to 128 total connections at speeds ranging from a single 64-kbps channel all the way up to multiple bundled T1/E1 links, multichannel interfaces provide a high degree of flexibility. These interfaces support individual DS0 channels (64 kbps), multiple DS0s as a fractional T1/E1 connection (64 kbps up to 2 Mbps), and full T1 or E1 connections (1.544 or 2 Mbps), all on the same interface card. With the multichannel bundling features provided by Cisco Express Forwarding and Multilink PPP, multiple T1/E1 connections can be treated as a single, higher-speed link for even greater flexibility.

The multichannel networking interfaces provide the flexibility to connect a central site to remote sites with a wide range of speed requirements, all over a single interface as shown in Figure 3. As remote site performance needs change, or as tariffs alter the cost equation, it is simple to change the configuration of the multichannel networking interfaces via software to accommodate the new requirements. Multichannel networking is supported for T3, E3, T1, and E1 physical interfaces on the Cisco 7500 and 7200 routers.

Figure 3: Multichannel Networking

For even more flexibility in supporting cost-effective enterprise WAN networking, the Cisco 7500 and 7200 also support direct termination with clear-channel T3 and E3 interfaces. These interfaces eliminate the need for expensive, external, separately managed DSUs for T3 and E3 leased-line applications. As with multichannel networking, using direct-termination interfaces reduces equipment costs, shrinks rack space, and simplifies management. Furthermore, the T3 and E3 direct-termination interfaces are compatible with DSUs from Kentrox, Digital Link, and Larscom, supporting their proprietary subrate and scrambling modes for investment protection and flexibility at remote sites.

Enterprise Example of Multichannel Networking

The simplest application of multichannel networking is aggregating multiple leased line-connected remote sites at different speeds onto a single T3 line into the central site as shown in Figure 4. This application concentrates up to 128 sites into the router without an M13 multiplexer or external T1 CSU/DSUs at the central site and supports sites connected at 64 kbps up to multiple T1s. The carrier grooms the remote site leased lines into a single T3 for connection to the multichannel networking interface on the Cisco 7500 or 7200 at a central site.

Figure 4: Multichannel Networking Diagram, and the Network Architecture for Aggregation of up to 128 Remote Sites

A different example is shown in Figure 5. Here, remote sites are connected to the central site at multiple T1 speeds, all without the use of external inverse multiplexers. This provides a cost-effective approach to connecting sites that require more than a single T1/E1, but less than a full T3/E3. It also provides a solution to high bandwidth requirements where T3/E3 connectivity is not available.

Figure 5: Multichannel Bundling Diagram

ATM Connectivity

Where ATM WAN connectivity is the right choice, the Cisco 7500 and 7200 routers now support a family of WAN-capable ATM port adapters. With a choice of DS3, E3, OC-3/STM-1 single-mode (intermediate or long reach) or multimode interfaces, the enhanced ATM port adapter provides key ATM traffic shaping features needed to efficiently use ATM WAN services. This traffic shaping capability will also enable the new enhanced ATM port adapters to support ATM QoS across an ATM backbone, and Tag Switching QoS across a Tag Switching ATM backbone. Furthermore, the enhanced ATM port adapter hardware is available bit rate (ABR)-ready, and will support this new ATM technology in a future Cisco IOS software release. This scenario will enable network managers to take advantage of cost-effective high-throughput ABR services as they are offered by service providers.

Advanced Services

The third key to meeting the requirements of new enterprise routed WANs is high-speed support for advanced services. Supporting capabilities such as QoS and multicast, and providing the ability to measure and report on the network, will enable enterprise network managers to combine multiple applications onto the enterprise WAN to make efficient use of new WAN services and to manage the expense of scarce WAN resources.

High-Speed Quality of Service

Cisco IOS software today supports a wealth of QoS capabilities for differentiated network services such as priority queuing and custom queuing. These services are widely deployed in enterprise networks today, particularly in support of mission-critical applications. Historically, they have supported line speeds of up to T1/E1 rates and were not applicable at higher speeds. In 1997, Cisco announced Internet QoS features for Cisco routers. These innovative features enabled Internet service providers to offer IP QoS services to their customers, with support in their backbones at high speed.

These high-speed IP QoS capabilities are now available for enterprise WAN networks that utilize DS3, E3, or
OC-3/STM-1 connections. By classifying traffic as to its importance, the enterprise routed WAN can prioritize packets properly on high-speed WAN links when congestion occurs. This setup ensures that mission-critical application packets, or time-sensitive real-time traffic, are sped to their destinations. The new enterprise QoS capabilities include:

Committed Access Rate (CAR)

This feature performs both packet classification and bandwidth management functionality. The packet classification features let users partition network traffic into multiple priority levels or classes of service (CoSs). The network manager can define up to six CoSs using the three precedence bits in the type-of-service (ToS) field in the standard IP packet header. The manager can then use other QoS features to assign appropriate traffic-handling policies, including congestion management, bandwidth allocation, and delay bounds for each traffic class.

Weighted Random Early Detection (WRED)

This feature provides network managers with powerful congestion-control capabilities designed to provide preferential treatment for premium-class traffic under congestion situations while concurrently maximizing network throughput and capacity utilization and minimizing packet loss and delay.

Weighted Fair Queuing (WFQ)

This feature provides bandwidth allocations and delay bounds to specified IP traffic sources by segregating the traffic into flows or classes and then servicing the various queues according to their assigned weights. WFQ classes can be defined by IP precedence, application ports, or incoming interface.

But fully utilizing QoS in a network requires much more than just queuing capabilities in the network nodes. CiscoAssure Policy Networking enables business users and applications to utilize the intelligence that is embedded in a network. Simply put, CiscoAssure Policy Networking makes it easier for a network manager to take advantage of distributed network intelligence features.

To set up a QoS policy, the network manager uses the CiscoAssure Policy Administration graphical user interface (GUI) to specify a policy based on business rules. A QoS policy binding is then created and activated by QoS policy servers and network-based enforcement in Cisco IOS devices. The Common Open Policy Service (COPS) Protocol provides policy exchange between the policy servers and the Cisco IOS software embedded in the intelligent network elements. Cisco IOS software translates the policy binding into local QoS enforcement mechanisms such WFQ or WRED.

NetFlow Traffic Measurement and Accounting

Enterprise network managers need detailed visibility into the traffic in their networks for planning, monitoring, and accounting purposes. Cisco's suite of NetFlow tools enable enterprise network managers to collect and process this information, delivering the following key benefits:

Accounting/billing: NetFlow data provides fine-grained metering (for example, flow data includes details such as IP addresses, packet and byte counts, time stamps, type of service, application ports, and so on) for highly flexible and detailed resource utilization accounting. Enterprise customers may utilize the information for departmental charge back or cost allocation for resource utilization.

Network planning and analysis: NetFlow data provides key information for sophisticated tools such as NetSys to optimize both strategic network planning (for example, backbone upgrade planning, routing policy planning) as well as tactical network engineering decisions (for example, adding additional Versatile Interface Processors [VIPs] to routers, upgrading link capacity). This scenario enables network managers to minimize the total cost of network operations while maximizing network performance, capacity, and reliability.

Network monitoring: NetFlow data enables extensive near-real time network monitoring capabilities. Flow-based analysis techniques may be utilized to visualize traffic patterns associated with individual routers as well as on a network-wide basis (providing aggregate traffic or application-based views) and to provide proactive problem detection as well as efficient troubleshooting and rapid problem resolution.

Monitoring and profiling: NetFlow data enables network managers to gain a detailed, time-based view of user and application usage over the network. Network managers may utilize this information to plan and allocate network and application resources (for example, intranet Web server sizing and location) to responsively meet enterprise business demands.

NetFlow on Cisco routers and NetFlow-enabled switches generates granular Layer 3 measurements on a per-flow basis, including source and destination IP address, source and destination application ports, packet and byte counts, flow-starting and flow-ending time stamps, and so on. Cisco also provides an extensive NetFlow infrastructure (see Figure 6), including the following utilities and applications:

  • FlowCollector---Provides scalable reception of NetFlow data export from multiple routers, performs data volume reduction via flexible filtering and aggregation policies, periodically stores collected data in flat files, and makes the collected data available to application consumers via a set of well-documented interface formats, including flow detail records

  • FlowAnalyzer---Provides NetFlow export data visualization via a Java-based GUI to analyze network traffic patterns as well as to identify, diagnose, troubleshoot, and resolve network problems

  • NetSys---Imports aggregated NetFlow data from the FlowCollector(s) and the data is then used for link and router utilization and performance analysis calculations.

Figure 6: NetFlow Metering/Measurement Infrastructure

Cisco Enterprise Accounting

New in CEA Version 2.0, CEA for NetFlow delivers crucial, end-to-end accounting to view network usage, including performance analysis, bandwidth allocation (including QoS levels), and application usage. The reports and data from CEA for NetFlow can be used to proactively monitor the network, resolve trouble spots, and provide an excellent tool for outbound marketing and customer service via intimate knowledge of customer network and application usage patterns.

As part of Cisco's Assured Network Services strategy, CEA for NetFlow delivers key infrastructure management and service-level agreement functionality that helps users account for and control the most expensive cost of owning a network: transmission expenses. CEA for NetFlow is ideal for both enterprise networks and network service providers, enabling internal and customer network reports, charge backs, and resource allocation for such advanced network services as QoS and application-based CoS.

Platform Extensions

Extending and enhancing the Cisco 7500 and 7200 platforms is the third major element in meeting the evolving requirements of enterprise routed WAN networks.

Advanced Router System for Cisco 7500

The advanced router system (ARS) for Cisco 7500 family routers is a systematic approach and development strategy that addresses the expanding needs for performance, density, and intelligent services for the routed WAN. It targets three critical functions within the router to enable application of advanced WAN services without the performance degradation normally experienced in high-service/high-performance networks.

1. First, a method is needed to ensure high packet forwarding performance with increasing WAN density requirements. The VIP creates a distributed architecture that moves the packet forwarding decisions from the central routing engine out to the network line cards as illustrated in Figure 7. VIPs also integrate part of the network operating system (Cisco IOS software) to enable flexibility in how the router manages packets with respect to security and QoS. This intelligence also enables new, high-density/high-function WAN port adapters necessary for wide and complex network infrastructures.

2. ARS ensures high performance of the distributed architecture through new software technology that scales both the control and forwarding processes in the router. For example, Cisco Express Forwarding proactively updates routing information within VIPs. This scenario ensures that the main route processor is free from packet forwarding chores and therefore improves router control activities. This ability to remove the main route processor from all forwarding activity, even in the presence of extensive network topology changes, provides more centralized resources to the tasks of route management (control).

Figure 7: Advanced Router System for Cisco 7500

3. Finally, to scale the application of services, Cisco 7500 ARS leverages numerous technologies such as specialized application-specific integrated circuits (ASICs) to accelerate data movement in Versatile Interface Processors. ARS also takes advantage of the distributed Cisco IOS within each VIP to accelerate a variety of network services such as QoS, security, data compression, and accounting.

Together, the elements of the advanced router system provide a new-generation high-end router where high-density and high-throughput capabilities are coupled with extensive Layer 3 services to enable new network architectures. Current ARS features of the Cisco 7500 include:

Current Cisco 7500 Advanced Router System Features
Intelligent Line Cards Scalable Router Control Advanced Services



Distributed Switching

Fast EtherChannel® Technology


Cisco Express Forwarding

Tag Switching

Internet QoS


Distributed Encryption/compression

Hardware-based accelerated service adapters

Distributed Weighted Fair Queuing

Cisco 7200 Multifunction Capabilities

As enterprise and service provider customers continue to drive the need for decreased operational and management costs, simplified network management, and increased revenue opportunities, the Cisco 7200 addresses these requirements by collapsing functions previously performed by separate devices into a single, cost-effective platform. Through functional integration, customers enjoy significant cost and management savings through a simplified network architecture that requires fewer platforms. Key markets for the Cisco 7200 include high-end multiservice (voice/video/data) edge applications (enterprise backbone, service provider customer premises equipment [CPE]), service provider points of presence (PoPs), and the IBM space.

Figure 8: Cisco 7200 Multifunction Flexibility


As enterprise WAN networking requirements evolve, Cisco continues to enhance the Cisco 7500 and 7200 routers, providing solutions that meet these requirements. New capabilities, such as multichannel networking, enhanced ATM interfaces, advanced software services, and platform enhancements enable enterprise network managers to efficiently utilize new and existing WAN services, reduce costs, and grow their WAN networks with the ever-increasing demands of enterprise network applications.