Designing and Deploying Multilink PPP over Frame Relay and ATM

Introduction

Multilink PPP over ATM and Multilink PPP over Frame Relay (MLPoATM and MLPoFR) was introduced in Cisco IOS® Software Release 12.1(5)T. This feature is targeted at customers with Frame Relay/ATM interworking (FR/ATM IW) that deploy Voice over IP (VoIP) across medium to low speed WAN links. Prior to this feature, there was no common Layer 2 fragmentation scheme that was supported by Cisco IOS on both ATM and Frame Relay customers with FR/ATM IW were forced to do Layer 3 fragmentation.

Prerequisites

Requirements

This document is intended for networking personnel involved in the design and deployment of VoIP networks that involve MLPoATM and Frame Relay networks. Cisco recommends that you have knowledge of these topics:

• Frame Relay

• ATM

• PPP

• MLP

• Frame Relay/ATM interworking

• Voice Quality of Service (QoS) Configuration

This document is not intended to provide technology training on these subjects. A list of reference materials is included at the end of this document. Cisco recommends that you review and understand these documents prior to reading this document:

• VoIP over Frame Relay with Quality of Service (Fragmentation, Traffic Shaping, LLQ / IP RTP Priority)

• VoIP QoS for Frame Relay to ATM Interworking with LLQ, PPP LFI and cRTP

Components Used

The information in this document is based on these software and hardware versions:

• Cisco IOS Software Release 12.1(5)T or later for MLP over FR/ATM IW

• Cisco IOS Software Release 12.2(2)T or later for Compressed Real Time Transport Protocol (cRTP) over ATM

• Cisco IOS Software Release 12.0(7)T for Low Latency Queueing (LLQ) over Frame Relay and ATM on the Physical Interface

• Cisco IOS Software Release 12.1(2)T for LLQ over Frame Relay and ATM per permanent virtual circuit (PVC)

The case study included in this document is based on a production network that uses these software and hardware versions:

• The core Cisco 3660 routers run Cisco IOS Software Release 12.2(5.8)T. The requirement for cRTP over ATM dictates the use of Cisco IOS Software Release 12.2T. This known issue is resolved in Cisco IOS Software Release 12.2(8)T1:

  Cisco bug ID CSCdw44216 (registered customers only) - cRTP causes high CPU when the class-based weighted fair queueing (CBWFQ)/LLQ link reaches congestion.

• The branch Cisco 2620 routers are in the process of being upgraded from Cisco IOS Software Release 12.2(3) to one 2.2(6a). The Cisco 2620s also act as branch H.323 gateways. The upgrade is triggered by a gateway-related issue. As far as the WAN and QoS features are concerned, Cisco IOS Software Release 12.2(3) does not exhibit any significant issues.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Design

This section discusses several design concepts related to the design and deployment of Multilink PPP over Frame Relay and ATM.

Data Link Overhead

When you design ATM and Frame Relay networks with MLP, you must understand the data link overhead. Overhead influences the amount of bandwidth consumed by each VoIP call. It also helps determine the optimum MLP fragment size. It is critical to optimize the fragment size to fit an integral number of ATM cells, especially on slow speed PVCs where a significant amount of bandwidth is wasted on cell padding. The data link overhead on MLP over Frame Relay and ATM PVCs depends on these factors:

• The mode of operation of the FRF.8 IW device (transparent or translational).

• The direction of the traffic (ATM to Frame Relay or Frame Relay to ATM).

• The PVC leg. Overhead is different on the ATM and Frame Relay legs of the PVC.

• The traffic type. Data packets have an MLP header; VoIP packets do not.

This table shows the data link overhead for a data packet. It details the number of bytes in the various Frame Relay, ATM, LLC, PPP, and MLP headers for all permutations of the FRF.8 mode of operation, traffic direction, and PVC leg.

¹DSAP/SSAP—Destination Service Access Point/Source Service Access Point.

²NLPID—Network Layer Protocol Identification.

The translational PVC is the easiest to comprehend as the overhead is the same in both directions. This is because the FRF.8 device translates between the MLPoATM and MLPoFR formats. As a result, the frame format is MLPoFR on the Frame Relay leg in both directions. The format on the ATM leg is MLPoATM in both directions.

The transparent PVC is slightly messier because the overhead differs in the two directions. This complexity arises because the Frame Relay router sends packets in the MLPoFR format. This format is carried across by the IW device onto the ATM side. Similarly, the ATM router sends packets in the MLPoATM format. This format is carried across by the IW device onto the Frame Relay side. Therefore, the result is different frame formats in the two directions on each leg.

In comparison, the overhead on an end-to-end Frame Relay PVC that uses FRF.12 is 11 bytes. Therefore, on an end-to-end Frame Relay link, FRF.12 is a more efficient choice for link fragmentation and interleaving (LFI) than MLP. On end-to-end ATM virtual circuits (VCs), MLP is the only choice since there is no standards-based fragmentation available. However, end-to-end ATM VCs are medium to high speed. Therefore, LFI is not required. The exception to this rule is slow speed ATM VCs over digital subscriber line (DSL).

The PPP ID is present in the first MLP fragment only. Therefore, the overhead in the first fragment is always two bytes more than in subsequent fragments.

The table here shows the data link overhead for a VoIP packet. It details the number of bytes in the various Frame Relay, ATM, LLC, and PPP headers for all permutations of the FRF.8 mode of operation, traffic direction, and PVC leg. The main difference between a data and a VoIP packet is that VoIP packets are sent as PPP packets and not as MLP packets. All other aspects are identical to the data scenario.

In comparison, the data link overhead for a VoIP packet on an end-to-end Frame Relay PVC is shown in the far right column.

VoIP Bandwidth Requirements

When you provision bandwidth for VoIP, the data link overhead must be included in the bandwidth calculations. This table shows the per call bandwidth requirements for the various flavors of VoIP. It is based on the characteristics of the PVC. The calculations in this table assume a best-case scenario for cRTP (for example, no UDP checksum and no transmission errors.) Headers are then consistently compressed from 40 bytes to 2 bytes.

Because the overhead varies on the different legs of the PVC, it is necessary to design for the worst-case scenario. For example, consider G.729 with 20 millisecond (msec) sampling and cRTP across a transparent PVC. The bandwidth requirements for this scenario on the Frame Relay leg is 12.4 Kbps in one direction and 13.2 Kbps in the other. Provisioning needs to be done with the assumption of 13.2 Kbps per call.

In comparison, the VoIP bandwidth requirement on an end-to-end Frame Relay PVC is also shown in the far right column of the above table. The additional overhead of PPP compared to native Frame Relay encapsulation results in an extra bandwidth consumption between 0.5 Kbps and 0.8 Kbps per call. Therefore, Frame Relay encapsulation with FRF.12 makes more sense than MLP on an end-to-end Frame Relay VC.

Note: cRTP over ATM requires Cisco IOS Software Release 12.2(2)T or later.

Optimize Fragmentation Size

The reason to use MLP on a Frame Relay/ATM PVC is to allow for large data packets to be fragmented into smaller chunks. The router then expedites VoIP packets by interleaving them in between the data fragments. In Cisco IOS, the fragmentation size is not configured directly. Instead, the desired delay is configured with the help of the ppp multilink fragment-delay command. Cisco IOS then uses this formula to calculate the appropriate fragment size:

fragment size = delay x bandwidth/8

When you do MLP across ATM, the fragment size needs to be optimized so that it fits into an integral number of cells. If this optimization is not done, the bandwidth required can almost double due to padding. For example, if each fragment is 49 bytes long, two ATM cells are required to carry each fragment. Therefore, 106 bytes are used to carry a payload of only 49 bytes.

Cisco IOS automatically optimizes the fragment size to use an integral number of ATM cells when it performs MLPoATM and MLPoFR. Cisco IOS automatically rounds up the calculated fragment size to the next integral number of ATM cells. No new CLI commands are added. Cisco IOS performs this optimization on both the Frame Relay and ATM ends of a MLPoFR/ATM PVC. It is possible that an MLP PVC can be end-to-end Frame Relay. In this case, optimizing it for ATM is not required. However, Cisco IOS optimizes this scenario for ATM since it has no way to detect whether the remote end is ATM or Frame Relay.

Note: Due to rounding, the delay that results can be slightly higher than that configured. This rounding error is more significant on low-speed PVCs.

You can also configure optimization manually. Since the fragment size cannot be specified directly in Cisco IOS, calculate the optimal fragment size and convert it into a delay. When you calculate the fragment size, adjust for the data link overhead, as the MLP code assumes that every link is High-Level Data Link Control (HDLC) and has 2 bytes of data link overhead. In addition to the HDLC data link overhead, the MLP code includes in its calculations the 8 bytes made up of the MLP ID, the MLP sequence number, and the PPP ID as outlined in the first table above.

Use this procedure in order to calculate the delay to be configured in Cisco IOS:

1. Calculate the theoretical fragment size based on the desired delay and the bandwidth of the PVC:

   fragment = bandwidth * delay / 8

2. Make sure that the fragment is a multiple of 48 bytes, so that it fits into an integral number of ATM cells.

   The formula to calculate the cell aligned fragment size is:

   fragment_aligned = (int(fragment/48)+1)*48

3. Adjust for the data link overhead that is not taken into account by MLP.

   As seen earlier, this overhead differs based on the PVC characteristics. Consider the ATM side of the PVC as this is the side for which you optimize. This table shows the number of bytes of data link overhead on the ATM side.

   FRF.8 Mode	Transparent		Translation
   Traffic Direction	Frame Relay to ATM	ATM to Frame Relay	Frame Relay to ATM	ATM to Frame Relay
   Frame Relay or ATM leg of PVC	ATM	ATM	ATM	ATM
   LLC DSAP/SSAP (0xfefe)	0	2	2	2
   LLC Control (0x03)	1	1	1	1
   NLPID (0xcf for PPP)	1	1	1	1
   AAL5	8	8	8	8
   Non MLP overhead (bytes)	10	12	12	12

   To arrive at the fragment size on which MLP bases its calculations, subtract the data link overhead from the desired cell-aligned fragment size. Add back 2 bytes in order to compensate for the HDLC encapsulation that MLP always assumes.

   The formula to calculate the fragment size that the MLP code targets is:

   fragment_mlp = fragment_aligned - datalink_overhead + 2

4. Convert the fragment size that results into the corresponding delay with this formula:

   delay = fragment_mlp/bandwidth x 8bits/byte

5. In most cases, the delay calculated is not an integral number of milliseconds. Since Cisco IOS only accepts an integer value, you must round down. Therefore, the delay value you configure in Cisco IOS with the help of the ppp multilink fragment-delay command is:

   fragment_delay = int(fragment_mlp/bandwidth x 8bits/byte)

6. In order to compensate for the above rounding error, fudge the bandwidth used by MLP to convert from delay to fragment. The fudged bandwidth that you configure in the Cisco IOS with the help of the bandwidth command is:

   bandwidth = fragment_mlp/fragment_delay * 8

This table shows the optimal values of ppp multilink fragment-delay and bandwidth for the most common PVC speeds. A target delay of 10 msec is assumed. In order to simplify the table, the calculations do not differentiate between transparent and translational PVC, or between traffic directions. The maximum difference in data link overhead is only 2 bytes. Therefore, the penalty for designing for the worst case of 12 bytes is small. Also shown in the table is the "real" delay that is encountered due to the fact that you increase the fragment size so that fragments fit into an integral number of cells.

Traffic Shaping and Policing Considerations

Special consideration is given to traffic shaping and policing in a Frame Relay/ATM IW environment. The issues in the Frame Relay to ATM direction are different to the issues in the ATM to Frame Relay direction. Therefore, each topic is described separately.

The main issue in the Frame Relay to ATM direction is the potential expansion in bandwidth consumption when going from frame to cell. For example, a 49 byte frame on the Frame Relay side consumes two cells, or 106 bytes, on the ATM side. Therefore, it cannot be assumed that the sustainable cell rate (SCR) equals the committed information rate (CIR). The worst case scenario occurs when each Frame Relay frame contains 1 byte of payload. Each of these bytes consumes a full 53 byte cell on the ATM side. As an example of this concept, this extreme and unrealistic scenario dictates an SCR that is 53 times the CIR. Two more realistic examples are:

• A G.729 VoIP packet is 60 bytes long, or 69 bytes (when MLP and Frame Relay overhead are included). On the ATM leg, it consumes two cells or 106 bytes. Therefore, if all traffic carried is VoIP, then an appropriate mapping is SCR = 106/69 = 1.5 x CIR.

• A Telnet packet that carries a single keystroke contains 40 bytes of TCP/IP header and 1 byte of data. On the Frame Relay side, this totals 56 bytes, including overhead. However, on the ATM side this packet expands to two cells. In this case, SCR = 106/56 = 1.9 x CIR.

Appendix A of the ATM Forum standard, BISDN Inter Carrier Interface (B-ICI) Specification Version 2.0, describes two methods of mapping between SCR and CIR. While both provide a scientific way to derive SCR from CIR, neither one is any more accurate than the data to which they are applied. One of the numbers required by the formulas is the typical frame size, a number that is hard to determine in a real network. Also, a number that can potentially change as new applications are rolled out on an existing ATM/Frame Relay network. The best recommendation to solve this problem is to work closely with your carrier since their policing policy will be of critical importance. With the assistance of the carrier, this fail-safe approach is possible:

• Frame Relay to ATM Direction - In the Frame Relay to ATM direction, the carrier needs to police traffic inbound at the Frame Relay ingress point only. For instance, on the Frame Relay switch connected to the Frame Relay attached customer premises equipment (CPE) device, the carrier polices traffic according to the contracted CIR. This policing policy is illustrated in the figure here. No further policing should happen once traffic is allowed into the network at the ingress point. The implication of this policing policy is that data received on the Frame Relay side is allowed to expand and consume whatever bandwidth is required to allow for cell tax, AAL overhead, and padding in order to carry it across the ATM leg of the network. In most cases, the ATM bandwidth required is less than twice the Frame Relay bandwidth used.

  

• ATM to Frame Relay Direction - In the ATM to Frame Relay direction, the opposite is experienced. Bandwidth usage is reduced when going from ATM to frame as cell tax, AAL overhead, and when padding is removed. However, because the potential transmit rate of the ATM side is much higher than that of the Frame Relay link, setting up the traffic shaping correctly on the ATM router is critical for the integrity of voice. If the shaping is too loose, then the ATM router feeds data at a rate faster than the physical speed of the Frame Relay link at the other end. As a result, packets start queuing up on the FRF.8 switch. At some point, packets start to drop . and since the ATM/Frame Relay networks do not distinguish between voice and data, VoIP packets are also dropped.

  At the same time, if shaping is too restrictive, then throughput suffers. Because of ATM cell tax, AAL overhead and padding is removed by the FRF.8 device. It does not consume bandwidth on the Frame Relay link. Therefore, you can oversubscribe the ATM side of the PVC slightly. The amount of padding and AAL overhead varies depending on average frame sizes and how aggressive the fragmentation is. Because you cannot accurately qualify this overhead, you are better off not trying to optimize for it. On the other hand, cell tax is completely deterministic. It is 5 bytes for every 48 bytes of payload. You can optimize for cell tax by setting the shaping target on the ATM router to 53/48 x SCR. The policing on the carrier side must be set to allow for this slight oversubscription.

Hints and Caveats

• MLPoATM/Frame Relay is currently only tested and supported with a service-policy attached to either the virtual-template or dialer-interfaces. Omitting the service-policy can cause the feature to not work. One example of this is documented in Cisco bug ID CSCdu19313 (registered customers only) .

• MLPoATM/Frame Relay clones two virtual-access interfaces for each PVC. One of these represents the PPP link. The other represents the MLP bundle. The show ppp multilink command is used to tell which is which. Multiple PPPoFR links per bundle is not supported. Putting two PPPOFR circuits into one bundle traffic will not be load balanced well across the circuits, since the PPPOFR driver does not provide the flow control signaling that real serial drivers do. MLPPP load balancing over ATM or frame relay might show noticeably less effectiveness than the same load balancing over physical interface.

• Each PVC is associated with four different interfaces, namely the physical interface, the sub-interface, and two virtual-access interfaces. Only the MLP virtual-access interface has fancy queuing enabled. The other three interfaces must have first in, first out (FIFO) queuing.

• When you apply a service-policy command to a virtual-template, Cisco IOS displays this normal warning message:

  cr7200(config)#interface virtual-template 1
  cr7200(config-if)#service-policy output Gromit
  Class Based Weighted Fair Queueing (CBWFQ) not supported on interface Virtual-Access1
  Note CBWFQ supported on MLP bundle interface only.
  

  These messages are normal. The first message advises that a service-policy is not supported on the PPP virtual-access interface. The second message confirms that the service-policy has taken effect on the MLP bundle virtual-access interface. This fact is verified with the help of the show interfaces virtual-access , show queue and show policy-map interface commands to check the queuing mechanism.

• PPP authentication is not strictly required since a PVC is like a leased-line. However, PPP authentication is handy as the show ppp multilink command is then used to determine the name of the router at the other end of the PVC.

• Frame Relay traffic shaping must be enabled for MLPoFR PVCs on the Frame Relay router.

• Cisco IOS Software Release 12.2 initially supported a maximum of twenty-five virtual templates per router. This limitation can impact the scale of an ATM distribution router if every PVC is required to have a unique IP address. The workaround for affected versions is to use IP unnumbered or to use dialer interfaces instead of virtual templates. In Cisco IOS Software Release 12.2(8)T, the support is increased to 200 virtual templates.

• Some service providers do not yet support PPP translation on their FRF.8 devices. Whenever this limitation is in place, the providers must configure their PVCs for transparent mode.

• Most examples in the Cisco IOS documentation show configurations that include a Frame Relay or ATM sub-interface. This sub-interface serves no purpose. The virtual template should just be attached to the physical interface. The result is a more compact and manageable configuration. This is especially important if there are a large number of PVCs.

• Use the show ppp multilink command as a fool-proof way to determine if there are any cell/frame drops on the carrier side. The only acceptable fragment loss is one caused by cyclic redundancy check (CRC) errors.

• Although MLPoATM/Frame Relay was introduced in Cisco IOS Software Release 12.1(5)T, bugs in this and subsequent releases dictate that care be taken when you select the Cisco IOS Software Release. Cisco recommends to use the latest maintenance release of the Cisco IOS 12.2 mainstream. However, other VoIP feature requirements can dictate the use of a different Cisco IOS Software Release, such as 12.2(2)XT if Survivable Remote Site Telephony (SRST) is required. This table lists some of the known issues. When you select Cisco IOS, each Cisco bug ID should be evaluated against the chosen IOS.

Case Study

Introduction

This section describes one of the first customer deployments of the MLPoATM/Frame Relay feature. The customer is referred to by the fictitious name XYZ Ltd. In 2001, XYZ Ltd replaced their PBXs with an IP Telephony solution. As part of this project, a new IP network was built. and Frame Relay/ATM interworking was chosen for the WAN. The carrier that provides the WAN service uses Newbridge switches to deliver the ATM and Frame Relay services.

Network Overview

The XYZ Ltd network is a hub and spoke network that connects twenty-six branches with two core sites. Each of the core sites is served by an E3 ATM attached Cisco 3660 router. Eighteen of the twenty-six branches are medium sized. They have dual Frame Relay PVCs that connect back to the 3660s at the two core sites through ATM/Frame Relay IW. The remaining eight branches are very small. They connect back to the closest medium sized branch through a single Frame Relay PVC. All branch routers are Cisco 2620. An end-to-end ATM PVC connects the two 3660 routers at the two hub sites. This figure illustrates the topology.

This table shows the Frame Relay access speeds and CIR. The CIR varies from 32 kbps to 256 kbps. Also shown in the table is the maximum number of simultaneous VoIP calls carried by each PVC.

Frame Relay traffic shaping is performed by the branch routers. Bursting beyond CIR is permitted. Cisco IOS traffic shaping is set to shape to the access speed, with mincir equal to CIR. If backward explicit congestion notifications (BECNs) are received from the carrier, then the router throttles back to mincir. This approach is against Cisco recommendations when doing VoIP over Frame Relay. Voice is already in trouble by the time the router responded to congestion notifications. However, in this case the carrier believes that its network is sufficiently over provisioned to never drop any frames or cells, and hence, BECNs should never be seen.

ATM traffic shaping is set to shape at the frame access speed at the remote end, plus cell tax. For instance, if the access speed is 512 Kbps, then SCR is set to 512x53/48 = 565 Kbps. This shaping rate is used to maximize throughput. This is safe because cell tax is stripped at the IW point. The policing on the carrier side is configured generously so that the slight oversubscription is allowed.

cRTP is used across the WAN. The hot spot as far as CPU is concerned is the Cisco 3660 distribution router at core site 1. By adding the numbers in the above table, it is determined that the theoretical maximum number of VoIP calls that traverse this router is 102 calls. Performance numbers from this graph indicate that the Cisco 3660 CPU load for 100 cRTP sessions (which are fast switched) is approximately 50 percent.

Virtual templates are used with IP numbered WAN links. One virtual template is required per PVC which is possible since the total number of PVCs that terminate on each Cisco 3660 is less than twenty-five.

Configurations

This document uses these configurations:

• Core ATM Router

• Branch Frame Relay Router

!--- Note: This section shows the parts of a core Cisco 3660 router !--- configuration that is relevant to MLPoATM.

        
class-map match-all Voice_Stream
  match access-group 100
class-map match-all Voice_Control
  match access-group 101

policy-map toortr01
  class Voice_Stream
    priority 175
  class Voice_Control
   bandwidth 18
   random-detect

interface loopback0
 ip address 10.16.0.105 255.255.255.252

interface ATM2/0
 description To Carrier E3 ATM Service
 no ip address

interface ATM2/0.15 point-to-point
 pvc toortr01 0/58 
  vbr-nrt 406 406
  tx-ring-limit 8
  protocol ppp Virtual-Template15

interface Virtual-Template15
 bandwidth 320
 ip unnumbered loopback0
 ip tcp header-compression iphc-format
 service-policy output toortr01
 ppp multilink
 ppp multilink fragment-delay 14
 ppp multilink interleave
 ip rtp header-compression iphc-format


!--- Note: Do not configure !--- IP addresses directly on any configuration source, !--- such as a virtual template, whenever the possibility !--- exists that this information is cloned to multiple !--- active interfaces. The exception to this rule is the !--- rare case where the intent is to define multiple parallel !--- IP routes and have IP do load balancing between them. !--- If an IP address is present on the configuration source, !--- this IP address is copied to all the cloned interfaces. !--- IP installs a route to each of these interfaces.
 

!--- Note: This section shows the parts of a branch Cisco 2600 router !--- configurations that is relevant to MLPoFR.


class-map match-all Voice_Stream
  match access-group 100
class-map match-all Voice_Control
  match access-group 101

policy-map dhartr21
  class Voice_Stream
    priority 240
  class Voice_Control
   bandwidth 18
   random-detect

interface loopback0
 ip address 10.16.0.106 255.255.255.252

interface Serial0/0
 description To Carrier Frame Relay Service
 encapsulation frame-relay IETF
 frame-relay traffic-shaping

interface Serial0/0.1 point-to-point
 frame-relay interface-dlci 38 ppp Virtual-Template1
  class dhartr21

interface Virtual-Template1
 bandwidth 320
 ip unnumbered loopback0
 max-reserved-bandwidth 85
 service-policy output dhartr21
 ppp multilink
 ppp multilink fragment-delay 10
 ppp multilink interleave

map-class frame-relay dhartr21
 frame-relay adaptive-shaping becn
 frame-relay cir 320000
 frame-relay bc 3200
 frame-relay mincir 320000

Related Information

• VoIP over Frame Relay with Quality of Service (Fragmentation, Traffic Shaping, LLQ / IP RTP Priority)
• VoIP QoS for Frame Relay to ATM Interworking with LLQ, PPP LFI and cRTP
• Configuring Link Fragmentation and Interleaving for Frame Relay and ATM Virtual Circuits
• PPP Over AAL5, RFC 2364, July 1998
• PPP in Frame Relay, RFC1973, June 1996
• The PPP Multilink Protocol (MP), RFC 1717, Nov. 1994
• Compressing IP/UDP/RTP Headers for Low-Speed Serial Links, RFC 2508, Feb. 1999
• Frame Relay / ATM PVC Service Interworking Implementation Agreement FRF.8.2
• Frame Relay Fragmentation Implementation Agreement FRF.12
• Voice Technology Support
• Voice and IP Communications Product Support
• Troubleshooting Cisco IP Telephony
• Technical Support & Documentation - Cisco Systems