Cisco on Cisco
Tail End Hop Off Case Study: How Cisco Cuts the Cost of Voice Calls
Cisco reduces long distance calling expense with creative routing
(PDF - 306 KB)
Take a quick survey
Every business day, Cisco generates an enormous number of voice calls—local and long-distance, in-country and international. These calls are made by more than 65,000 Cisco employees who communicate internally with each other from more than 450 company sites in more than 165 countries worldwide, as well as externally with customers, partners, vendors, and colleagues.
As part of the company's increased efforts to reduce expenses during the 2009 global economic recession, a cross-function team of Cisco employees looked for new ways to capture more cost savings for long-distance voice calls, especially international calls.
Previously, only about 15 percent of external long-distance calls were carried on the Cisco network for all or part of the call's route from origin to destination. The remaining calls are considered off-net—that is, they terminate off of the Cisco network and use the networks of one or more voice service providers to reach their destination.
In a few locations, Cisco IT had implemented a network feature called tail-end hop off (TEHO) to reduce costs by transporting these calls for most of the distance on the Cisco network. However, the process to identify the best places to implement TEHO had never been formalized, nor had TEHO been deployed extensively within Cisco.
The team's mandate was to:
- Make a comprehensive examination of Cisco's long-distance costs, call volumes, and traffic patterns.
- Identify more locations for implementing TEHO or other cost-saving routing designs.
- Deploy TEHO where it made sense.
- Track the long-distance costs avoided through the call-routing changes.
- Reduce expenses for long-distance calling, especially internationally
- Use TEHO routing at more Cisco sites
- Initial cost savings of $900,000 for the new TEHO sites
- Future cost avoidance of at least $4 million annually
- Improved ability to identify additional cost savings and routing efficiencies
- Identify regulatory restrictions
- Create a cross-function project team
- Understand true cost savings
- Use a comprehensive CDR tool
- Verify local network capacity
- Consider impact on call routing and dial plan designs
- Train finance staff on call-routing concepts
- Identify more sites for TEHO and regional routing as justified by calling patterns and costs
TEHO extends the concept of least-cost routing designs, traditionally used for data networking, to voice calls. The TEHO implementation is based on intelligent routing technology in the Cisco Unified Communications Manager (CUCM) and the Cisco IOS® Gatekeeper
Cisco IT business analyst
When using TEHO for a long-distance call to an external phone number, an intelligent IP network can determine if a Cisco office is located near the call’s destination. The gateway routes the call across the Cisco global WAN to the gateway at that local site (the “tail end”), where the call “hops off” to the Public Switched Telephone Network (PSTN) and completes as a local call. (Figure 1)
Because TEHO can yield significant expense reductions, the Cisco team wanted to identify more locations where it could be implemented cost-effectively. As of mid-2009, Cisco had implemented TEHO in 24 new countries, for a total of 55 locations worldwide.
The first step toward identifying possible TEHO opportunities is building the right cross-function team. The Cisco IT team consisted of representatives from the following groups:
- Capacity management and optimization, which sees Cisco’s telecom invoices and the opportunities to reduce costs.
- Unified communications design team, which defines routing and dial plan standards.
- Voice operations engineers, who build the TEHO routing plans into the Cisco Unified Communications Managers and Cisco IOS Gatekeepers and also identify which WAN routes have sufficient capacity to support added voice traffic.
Figure 1. Routing of a long distance voice call in TEHO and non-TEHO implementations
The Cisco team identified potential sites for new TEHO implementations by first examining the locations with the highest international call volumes and costs, based on data gathered from service provider invoices and call detail records (CDRs). For each site, the team followed defined processes to:
- Determine whether TEHO is permissible under local regulations and identify any regulatory restrictions and issues.
- Assess the current capacity and projected impact on the site’s WAN bandwidth and local PSTN voice trunks.
- Establish a baseline for tracking bandwidth utilization and calling costs before and after TEHO implementation.
- Identify any impact to operational support for the site.
- Conduct latency tests for the proposed configuration to determine whether TEHO would impact voice-call quality.
- Create new “least-cost routing” TEHO route-pattern configurations within each affected CUCM cluster worldwide. These configurations direct each call along the right path based on the dialed number.
- Add new dialpeers to the associated Cisco IOS Gatekeepers.
"Previously, we were looking at only part of the expense picture by implementing TEHO in locations like the United Kingdom where we knew we had a large volume of calls,” explains Bill Lowers, Cisco IT network engineer. “By bringing staff from both the capacity management and optimization group and the voice operations group to the team, we gained the insights we needed to identify where TEHO and regional routing could make a difference in our costs.”
TEHO may not be feasible for many locations because of regulatory restrictions, low call volumes, or the costs of adding the necessary WAN bandwidth or local PSTN trunks to support more voice traffic. In those cases, the Cisco team determined it was still possible for the company to reduce its calling costs by implementing voice-routing hubs in selected world regions.
Figure 2. A regional routing hub takes advantage of differences in service provider prices for long-distance calls.
This routing may be appropriate because long-distance costs relate to historical issues and partnership relationships among service providers, rather than the call’s distance travelled. For example, it may be less expensive to route a call for part of the distance over the company’s internal network and via another country than to route it directly for the entire distance over the PSTN.
This routing design may also yield lower costs than could be achieved even by implementing TEHO in some destination countries, due to more competitive long-distance rates based on higher traffic volumes than could be generated by an individual remote office.
The Cisco team identified Amsterdam as the location for the first regional routing hub for long-distance voice calls. “This site is centrally located for much of Cisco’s international voice traffic, which gives us the ability to choose among service providers based on the potential cost savings available in each region and the unique pricing arrangements among different countries,” says Anita Mallol, Cisco IT business analyst. (Figure 2)
Table 1 summarizes Cisco’s practices for routing voice calls made by employees. The path taken by a particular call depends on a combination of considerations for network routing and cost. In addition, if long-distance calls cannot be completed over the Cisco network for any reason, they are automatically routed to the PSTN as a failover practice.
|Table 1. Examples of Cisco IT practices for voice call routing.|
|Call Origin and Destination||Call Classification||Network Path|
|One Cisco office to another, local or long-distance (internal call)||Typically on-net||Carried entirely on the Cisco network if sufficient WAN capacity is available. Calls may be routed over the PSTN for very small or new Cisco offices, or when sufficient WAN capacity is not available across the entire path.|
|Cisco office to an external local telephone number||Off-net||Sent over local voice trunks to the PSTN nearest the originating office, which means that most of the local call is carried over the PSTN.|
|Cisco office to an external long-distance number||Full off-net or partial on-net and off-net||Unless TEHO is activated for that destination, the call is sent over local voice trunks to the PSTN nearest the originating office. This design means the call is carried by the selected long distance provider over the entire distance.
If TEHO is activated, the call may be carried for some or all of the distance over the Cisco WAN for significant cost avoidance (TEHO or regional routing).
The TEHO and regional routing implementations have produced several benefits for Cisco’s costs and operations.
Reduced expense for voice calls. “TEHO is a great example of how Cisco IP telephony capabilities can help a company reduce its business costs,” says Mallol. Between March and July 2009, Cisco produced direct cost savings of more than US$900,000 (nearly 30 percent of previous average spending) for long-distance calls since increasing TEHO traffic to existing sites and implementing TEHO in 24 new countries earlier in the year.
In the future, Cisco expects to avoid costs of at least $4 million annually for these sites, after including the overall charges for the associated WAN circuits. As TEHO and regional routing are implemented for other locations, Cisco expects to realize additional expense reductions, even accounting for the cost of WAN capacity increases.
Call routing efficiencies. The TEHO and regional routing designs allow Cisco IT to optimize voice-call routing based on local and long-distance costs for each country. Network managers can change that routing based on dynamic factors such as new traffic patterns, changes in service provider prices, and Cisco business activity.
Opportunities for potential future cost savings. When a cross-function team analyzes calling data, it is possible to discover potential cost savings from other sources. For example, the Cisco team’s detailed analysis of called numbers indicated that many long-distance calls are made to mobile phones. This insight allows Cisco to consider consolidating mobile-phone service with fewer providers to receive discounted pricing.
Identify regulatory restrictions. Government regulations determine how long-distance voice calls can be routed:
- In some countries, all voice calls—internal or external, local or long-distance—must be sent entirely over the PSTN.
- Some countries do not permit TEHO or routing of external voice calls over an IP telephony network.
- A country may not allow TEHO, but permit long-distance internal voice calls to be sent over the corporate WAN instead of the PSTN.
These restrictions may mean that cost savings from TEHO cannot be realized in every country. However, regional call routing to those countries may still result in lower charges because of lower long-distance rates at the regional hub than at the call’s origination point.
Important Note: Because each network is unique and comprised of many combinations of private, enterprise, and public connections, Cisco’s implementation of VoIP in any particular area does not imply third-party authorization to implement similar services without first consulting local authorities. Accordingly, companies that propose to implement advanced VoIP telephony on an enterprise basis should review applicable laws and regulations in the countries in which that VoIP telephone service will be deployed, in order to determine the current regulatory requirements. They may also wish to enlist legal counsel in these countries.
Cisco IT network engineer
Create a cross-function project team. The Cisco IT team that evaluates potential sites for TEHO and regional routing includes representatives from multiple groups concerned with voice traffic and costs. This membership gives the team a comprehensive view of long-distance calling costs, network facilities, and design; insight on detecting potential toll fraud; and coordinated changes to dial plans and call routing to avoid negative impacts on users.
“The team helps to overcome the challenges of identifying calling costs in a very large enterprise like Cisco,” says Troy Wylie-Hill, Cisco IT network engineer. “For example, the people who paid the service provider invoices couldn’t verify the validity of the charges and the people who work on the network don’t usually see the invoices, so they don’t know all of the costs associated with the network design.”
Understand true cost savings. “Overall cost savings due to TEHO should be evaluated against the higher costs created by added WAN bandwidth, more local voice trunks, and increased routing complexity,” says Lisa Lockhart, Cisco IT service manager. “In addition, the cost savings that might be gained by transferring more traffic to one service provider’s network may be more than offset by the loss of discounted pricing that occurs when that traffic is transferred away from the previous service provider.”
Making these cost tradeoff decisions requires a good understanding of call volumes and costs. The Cisco team calculates calling expense by using CDRs and service provider invoices. This provides a more accurate result compared to simply calculating an average cost per minute, because call charges can vary according to factors such as distance, time of day, and service provider discounts.
Use a comprehensive CDR tool. A good tool for analyzing call detail records is essential for activities such as:
- Identifying sites that can potentially benefit from TEHO or regional call routing.
- Defining thresholds and receiving reports and alerts to determine whether unexpected traffic increases are due to normal business activity, potential fraud or asset misuse, or an indication of additional network capacity needs.
Verify local network capacity. Because TEHO sends more voice calls through a local Cisco office, the local network capacity must be adequate in two areas:
- The WAN bandwidth to the site must be able to handle a higher level of voice traffic without introducing latency that would impact call quality.
- The site must have enough voice trunks to the PSTN to avoid creating an “all circuits busy” condition that would block further inbound and outbound calls for that office. This condition would also cause any new TEHO call to overflow back to its origination point and complete as a long-distance PSTN call, with the associated cost.
By tracking voice-quality metrics and the number of overflow calls, Cisco IT can identify when a WAN circuit upgrade or additional local voice trunks are needed at a local site.
Consider the impact on dial-plan and call-routing designs. TEHO may not be practical in some locations, even where it is legal and where sufficient WAN and voice trunk capacity is available. The reason: TEHO can add complexity to call-routing designs and troubleshooting.
TEHO and regional routing add more route patterns to the CUCM clusters and the gatekeepers, which adds complexity for diagnosing the source of general voice problems. More possible entry and exit points for a call can also make it more difficult to detect and track the source of toll fraud.
Train finance staff on the basics of IP telephony call routing. The people who first notice potential issues and opportunities for new TEHO are often the finance staff and analysts who review service provider bills on a regular basis. When they understand the basic processes of dial plan, IP telephony call routing, and TEHO, they are far more likely to identify possible new opportunities for cost savings. “Understanding how calls to certain destinations should be routing helps us identify anomalies that could indicate fraud or a problem in the network,” says Mallol.
Cisco expects to implement TEHO in more locations as they are identified by the team's ongoing analysis of long-distance calling costs.