Guest

Cisco Unified Communications Manager (CallManager)

Case Study IP Telephony Deployment: National Bulletin Singapore

Cisco - Case Study IP Telephony Deployment: National Bulletin Singapore

Document ID: 13968

Updated: Feb 02, 2006

   Print

Introduction

This document makes available to Cisco customer, partners, and employees the experiences and lessons learned from the IP Telephony deployment at National Bulletin (NB) in Singapore. This document attempts to:

  • Describe and critique the design of the deployed solution.

  • Identify possible improvements to the design.

  • Highlight tradeoffs in the design.

NB is a global publishing company. The Singapore operation consists of approximately 6,000 sales, printing, and writing staff. The NB staff resides in a number of office buildings located within the same vicinity. In late 2000, NB added another building, the DBS building, to their campus. This additional building houses 750 employees. Rather than deploy a private branch exchange (PBX) in the new building, NB decided to deploy an IP Telephony solution. As such, the greenfield deployment includes the network component.

The NB IP Telephony solution is a single-site design. All IP Telephony users are located in the DBS building, and are distributed across five floors. Cisco CallManagers, Public Switched Telephone Network (PSTN) gateways, and voice mail are also physically located in the DBS building.

A wide-area network (WAN) link connects the DBS building to the NBAP building less than 1 Km away. This WAN link carries Voice over Internet Protocol (VoIP) traffic across to NBAP, where a gateway connects into the worldwide NB PBX network. This diagram shows the DBS and NBAP buildings.

65363.gif

Campus LAN Infrastructure

The DBS LAN infrastructure consists of one Catalyst 6509 switch in the core and nine Catalyst 4006 switches in the wiring closets. This table shows how the Catalyst 6509 switch is populated.

Slot Module Description
1 WS-X6K-SUP1A-MSFC Supervisor with Multilayer Switch Feature Card (MSFC)
2 WS-X6K-S1A-MSFC2/2 Supervisor with MSFC
3 WS-X6416-GBIC 16-port GE module
4 WS-X6408A-GBIC 8-port GE module
5 WS-X6348-RJ45V 48-port 10/100 module with inline power
6 WS-X6608-E1 8-port E1 gateway
7 WS-X6624-FXS 24-port Foreign Exchange Station (FXS) gateway
8 WS-X6624-FXS 24-port FXS gateway
9 Empty

This table shows how the Catalyst 4006 switches are populated.

Slot Module Description
1 WS-X4013 Supervisor 2 with two Gigabit Ethernet (GE) ports
2 WS-X4148-RJ45V 48-port 10/100 module with inline power
3 WS-X4148-RJ45V 48-port 10/100 module with inline power
4 WS-X4148-RJ45V 48-port 10/100 module with inline power
5 WS-X4148-RJ45V 48-port 10/100 module with inline power
6 WS-X4148-RJ45V 48-port 10/100 module with inline power

The total capacity of the LAN infrastructure is to connect and power 2,160 IP phones.

The Catalyst 4006 switches connect back to the Catalyst 6509 by way of one of the GE ports on the supervisor, in a pure hub-and-spoke fashion. Four of the five floors have two Catalyst 4006 switches while the fifth floor has one Catalyst 4006 switch. This diagram illustrates how the switches are spread across the floors and how they connect back to the Catalyst 6509 switch.

65364.gif

The Catalyst 6509 constitutes a serious single point of failure. A significant improvement in availability can be achieved by adding a second Catalyst 6509, and dual-homing the Catalyst 4006 switches to both core switches using the spare GE port on the Catalyst 4006 supervisors. With this design, there is little justification for the duplication all of the modules in the Catalyst 6509. Rather, the modules that exist (the supervisors, GE, and FXS modules) can be split across the two chassis. However, an additional eight-port E1 module should be added so that the PSTN connectivity can also be split across the two chassis. This design also allows for the two Cisco CallManagers to be connected to separate switches. This ensures that a Catalyst 6509 failure does not completely isolate the Cisco CallManagers.

A second Catalyst 6509 switch was part of the initial proposal. However, due to cost considerations, NB decided on a single Catalyst 6509.

NB follows the Cisco design recommendations and has IP phones and data devices in separate Virtual LANs (VLANs). Each Catalyst 4006 has its own voice VLAN. Therefore, there are two voice VLANs per floor, for a total of nine voice VLANs. Each Catalyst 4006 has 240 ports. Therefore, each voice VLAN is potentially home to 240 IP phones. This is a conservative design, but does have the advantage that it limits the impact if a malfunctioning device floods the VLAN with broadcasts. When the Catalyst 6000 family MSFC routes between VLANs, Layer 3 forwarding performance is not an issue.

65365.gif

All data devices reside in a single, large VLAN. This does not comply with Cisco design recommendations. However this was the design preferred by NB due to their internal operational and maintenance requirements. Because this single data VLAN spans all switches, a Layer 2 broadcast storm in this VLAN has the potential to affect all IP phones. This makes quality of service (QoS) on the Catalyst switches even more critical. QoS is discussed later in this document.

This example shows a typical VLAN configuration for a Catalyst 4006 port. This example places all 48 ports in slot 5 in voice VLAN 110 and data VLAN 11.

set port auxiliaryvlan 5/4-48 110 
set vlan 11 type ethernet state active 
set vlan 11 5/4-48

The NB network has these three distinct QoS trust boundaries:

  • Catalyst 4006 10/100 port.

  • Catalyst 6509 10/100 port connecting to Cisco CallManager.

  • Catalyst 6509 10/100 port connecting to the Cisco 7200 router.

The Catalyst 4000 10/100 modules in use have a single receive (RX) queue (1q1t) and two transmit (TX) queues (2q1t). All ports are configured with the commands in this example to enable the second TX queue, and to put frames with a class of service (CoS) value between 2 and 7 in the second queue. As a result, all Real-time Transport Protocol (RTP) packets (CoS=5) and all Skinny packets (CoS=3) go into the second queue, while all other traffic goes into the first queue.

set qos enable
set qos map 2q1t 1 1 cos 0-1
set qos map 2q1t 2 1 cos 2-3
set qos map 2q1t 2 1 cos 4-5
set qos map 2q1t 2 1 cos 6-7

Note that the Catalyst 4006 does not support any kind of policing. It trusts the CoS of any frame received on its ports. This is not an issue as long as an IP phone is connected since the default behavior of the IP phone is to not trust traffic received on the PC port, and to rewrite it with a CoS of 0. However, a PC connected directly to a Catalyst 4006 port can potentially exploit the QoS if it sends data as 802.1p frames. This requires a somewhat sophisticated user. However, Windows 2000 and standard Ethernet network interface cards (NICs) do support 802.1pq.

The QoS configuration on the Catalyst 6509 is slightly more involved since ports on the Catalyst 6509 have a variety of queue structures, as shown in this table.

Module Receive Queue Transmit Queue
WS-X6K-SUP1A-MSFC 1p1q4t 1p2q2t
WS-X6416-GBIC 1q4t 2q1t
WS-X6408A-GBIC 1p1q4t 1p2q2t
WS-X6348-RJ45V 1p1q4t 1p2q2t

Any port that has a strict priority queue puts all frames with CoS=5 in that queue by default. However, it is preferred to have all VoIP signaling traffic (frames with CoS=3) in the second non-priority queue. This configuration enables this behavior.

set qos map 1p2q2t tx 2 1 cos 3
set qos map 2q2t tx 2 1 cos 3

The GE ports connecting to the Catalyst 4006 switches are inside of our trust boundary. Normally the system would trust the CoS of the received frames. Since the Catalyst 4006 trust boundary can be compromised by connecting a PC directly to a switch port, traffic from the Catalyst 4006 switches is treated as untrusted, and is policed by the Catalyst 6509. The policing is done by an access control list (ACL) that looks for RTP, Skinny, H.225, and H.245 packets. RTP packet headers are rewritten with DSCP=46 while all VoIP signaling packet headers are rewritten with DSCP=26. The ACL for this, as shown in this example, is mapped to all GE ports.

set qos acl ip ACL_VOIP dscp 46 udp any any range 16384 32767
set qos acl ip ACL_VOIP dscp 46 udp any range 16384 32767 any
set qos acl ip ACL_VOIP dscp 26 tcp any any range 2000 2002
set qos acl ip ACL_VOIP dscp 26 tcp any range 2000 2002 any
set qos acl ip ACL_VOIP dscp 26 tcp any any eq 1720
set qos acl ip ACL_VOIP dscp 26 tcp any eq 1720 any
set qos acl ip ACL_VOIP dscp 26 tcp any any range 11000 11999
set qos acl ip ACL_VOIP dscp 26 tcp any range 11000 11999 any
set qos acl map ACL_VOIP 3/1-16,4/1-8,

Two of the 10/100 ports on the Catalyst 6509 are used to connect to the two Cisco CallManagers. These are essentially trusted ports, but in the NB network they are treated as untrusted, and the Catalyst 6509 forces CoS=3 on received frames. This example shows the port configuration.

set vlan 110 5/2-3
set port qos 5/2-3 cos 3

An alternative, and cleaner approach, is to configure Cisco CallManager to set the IP differentiated services code point (DSCP) value in all VoIP signaling packets. To do this, set the service parameters IpTosCm2Cm and IpTosCm2Dvce to 0x26 on Cisco CallManager. The Catalyst 6509 can then be configured to trust DSCP for frames received on that port, as shown in this example.

set port qos 5/2-3 trust trust-dscp

This approach has the advantage that only VoIP-controlled frames, and not every frame from Cisco CallManager, receive good QoS. This is important if a Cisco CallManager upgrade image is uploaded to the CallManager server, or if large amounts of call detail records (CDRs) are routinely pulled off the server. Currently, this kind of traffic also receives a high QoS.

Finally, one of the 10/100 ports on the Catalyst 6509 is used to connect to the Cisco 7200 Series WAN router. This is also a trusted port, but the current Cisco IOS® in use on the Cisco 7200 router does not copy the DSCP value to the CoS field. To overcome this limitation, the switch port is treated similarly to the GE ports (classify incoming traffic using the same ACL) and selectively provide QoS based on this. Therefore, the configuration for the router switch port is shown in this example.

set vlan 10 5/1
set qos acl map ACL_VOIP 5/1

WAN Infrastructure

The WAN component of the NB IP Telephony network is small. The Cisco 7200 Series router in the DBS building has WAN links to both NBAP and Capital Towers. However, only the link to NBAP carries voice. Even then there are separate links between DBS and NBAP for voice and data. Voice quality problems were discovered during the early stages of the deployment, and it was decided to change codec from G.729 to G.711. This required extra bandwidth, and voice and data on the WAN was therefore separated. The cause of these problems was later found to be with the IP phone load in use at the time. As a conservative measure, NB decided to stay with G.711 and separate WAN links for voice and data in the short term.

Currently the voice WAN link consists of three physical E1 links that are bundled together by Multilink PPP (MLP). Because of the relatively high speed of the link, no Link Fragmentation and Interleaving (LFI) is required. The only required QoS feature is queuing. The preferred queuing mechanism is Low Latency Queuing (LLQ). However, this did not work due to a Cisco IOS issue with LLQ and MLP, where the service-policy command disappeared from the configuration if the link went down. As an interim workaround, Priority Queuing has been in use. This example shows the current WAN configuration.

interface Multilink88
 ip address 10.104.209.73 255.255.255.248
 priority-group 1
 ppp multilink
 ppp multilink fragment-delay 10
 ppp multilink interleave
 multilink-group 88

interface Serial4/0
 bandwidth 2000
 encapsulation ppp
 ppp multilink
 multilink-group 88

interface Serial4/1
 bandwidth 2000
 encapsulation ppp
 ppp multilink
 multilink-group 88

interface Serial4/2
 bandwidth 2000
 encapsulation ppp
 ppp multilink
 multilink-group 88

priority-list 1 protocol ip high list 121
priority-list 1 protocol ip medium list 122
priority-list 1 default low
priority-list 1 queue-limit 500 40 60 80

access-list 121 permit udp any any range 16384 32767
access-list 121 permit udp any range 16384 32767 any
access-list 122 permit tcp any any range 2000 2002
access-list 122 permit tcp any range 2000 2002 any
access-list 122 permit tcp any any eq 1720
access-list 122 permit tcp any eq 1720 any
access-list 122 permit tcp any any range 11000 11999
access-list 122 permit tcp any range 11000 11999 any

The current WAN configuration is a compromise and is not recommended for use in other deployments. The medium term plan is to consolidate voice and data on to a single WAN link, and to replace priority queuing with LLQ. Separate links for voice and data require static routing or policy-based routing, and the advantages of using a dynamic routing protocol are lost. Priority queuing, even with voice packets being assigned the high queue, does not guarantee that strict priority is given to voice packets. System traffic, such as routing updates, keepalives, and so on still takes preference over voice packets in the high queue.

It has been verified that LLQ works correctly in Cisco IOS Software Release 12.2. This example shows the router QoS, after the move to LLQ. The bandwidths are based on 60 simultaneous G.729 calls (RTP: 60 x 24 kbps = 1440 kbps and Signaling: 60 x 0.5 kbps = 30 kbps).

interface Multilink88
 service-policy output VoIP

class-map VoIP-RTP
 match access-group 121
class-map VoIP-Sig
 match access-group 122

policy-map VoIP
 class VoIP-RTP
  priority 1440
 class skinny
  bandwidth 30

access-list 121 permit udp any any range 16384 32767
access-list 121 permit udp any range 16384 32767 any
access-list 122 permit tcp any any range 2000 2002
access-list 122 permit tcp any range 2000 2002 any
access-list 122 permit tcp any any eq 1720
access-list 122 permit tcp any eq 1720 any
access-list 122 permit tcp any any range 11000 11999
access-list 122 permit tcp any range 11000 11999 any

IP Telephony

IP Phones and Their Connection to the Switch

The DBS building has approximately 750 IP Phone 7960s. IP phones connect to 10/100 ports on the Catalyst 4006, and receive inline power from the switch. PCs connect to switch ports in the back of the IP phone, as depicted in this diagram.

65373.gif

IP phones and PCs are on separate VLANs and IP subnets.

Site Planning

All IP phones receive inline power from the Catalyst 4006 line cards. The switches themselves are powered by three in-chassis AC power supplies. The inline power, however, is sourced externally from the Catalyst 4006 auxiliary power shelf (WS-P4603). The power shelf has three power supplies. Each supplies 1050W at -52V DC. This is sufficient to power a fully populated Catalyst 4006 switch with Cisco IP Phone 7960s connected to all 240 ports.

65366.gif

All Catalyst 4006 switches run off an uninterrupted power supply (UPS). This allows them to continue operation for two hours in the event of a power failure. The Cisco CallManagers connect to a four-hour UPS.

Cisco CallManager

The NB CallManager deployment model is single site with centralized call processing. One can argue that the model is in fact multi-site because of the WAN link to NBAP and the associated gateway located there. But this fact can (for the most part) be ignored as no Call Admission Control (CAC) is required across the WAN. This is because the number of calls across the WAN is implicitly limited by the number of trunks connecting the gateway to the PBX.

The NB Cisco CallManager cluster consists of two Cisco Media Convergence Server 7835s (MCS-7835). One CallManager performs the database publishing function, and the other subscribes to the database. All IP phones register with the subscriber as the primary Cisco CallManager, and use the publisher as the secondary Cisco CallManager.

Two regions are configured: NBAP and DBS. The gateway at NBAP is the only device in the NBAP region, all other devices are in the DBS region. The intended design is to use G.711 for all calls within the DBS building, and use G.729 only for calls across the WAN. Currently, however, calls across the WAN link are also G.711.

The IP phones at the DBS region have a five-digit extension in the range of 17000 to 17999. There are five other NB sites in Singapore that have a mix of four- and five-digit extensions. This table shows the NB Singapore sites.

Site Name Site Code Digits
Department of Binding Singapore DBS 5
NB Acme Printing NBAP 5
Acme Building A ABA 4
Acme Building B ABB 5
Douglas Printing DP 4
Grant Printing GP 4

Extensions are assigned so that the first digit uniquely determines if the extension is four or five digits. IP phone users can dial any NB Singapore PBX extension by dialing the four- or five-digit extension.

As discussed later in the Gateway Integration section, there are three types of gateways:

  • One Cisco 7200 H.323 gateway that connects to a legacy NB PBX network.

  • Three Catalyst 6509 E1 gateways that connect to PSTN.

  • Two Catalyst 6509 24-port FXS gateways that connect to voice mail.

This is reflected in the Cisco CallManager route group configuration. One route group exists for each of the three gateway types. This table outlines the characteristics of each route group.

Route Group Platform Slot/Port Port Type Priority
DBS VM Catalyst 6509 Switch 6/1-24 7/1-24 FXS FXS 1 2
DBS PSTN Catalyst 6509 Switch 8/1 8/2 8/3 PRI PRI PRI 1 2 3
NBAP Legacy Cisco 7200 Router 5/1-2 PRI 1

Calls to the various destinations are routed as follows:

  • Calls to PSTN use the DBS PSTN route group. There is no backup.

  • Calls to voice mail use the DBS VM route group. There is no backup.

  • Calls to the NB Telnet service use the NBAP Legacy route group.

  • Calls to an NB Singapore PBX extension use the NBAP Legacy route group as the primary, and the DBS PSTN route group as the secondary route group.

All that remains now is to define appropriate route patterns and link them to the route groups. This is straightforward for the first three items listed above, as only a single route list is required. Things get slightly more involved with the last item because of the backup route group. The preferred path for a call from an IP phone to a PBX extension is through the NBAP Legacy route group. If this gateway is unavailable, calls are routed through the PSTN through the DBS PSTN route group. When this happens, digits have to be prefixed to the dialed extension in order to create the full PSTN phone number. The digits prefixed depend on the site being called, hence, each site must have a different route list. Because there are five NB sites with PBXs, and several PSTN prefixes per site, they end up with ten route lists.

This diagram shows all NB route lists. The two- or three-digit number included in the name of most route lists reflects the digits that are prefixed to the called extension, whenever calls are sent to the DBS PSTN route group.

65367.gif

Certain NB IP phone users are restricted in the numbers they can call. This is controlled by placing the route patterns and IP phone extensions in a number of partitions. Partitions are then grouped together into a calling search space. IP phones belong to a calling search space and can only call the numbers contained in the partitions in that search space. This table lists the NB Cisco CallManager search spaces and partitions.

Search Space Partitions Description
Guest DBS User DBS Guest NB SGP DBS Call Attendant Normal DBS user phone Lobby and other public area phone NB Singapore PBX extension Receptionist
User DBS User NB GSDN NB Telnet SGP PSTN DBS Guest NB SGP IDD DBS Call Attendant Normal DBS user phone — International calls by way of worldwide NB network Domestic calls in Singapore Lobby and other public area phones NB Singapore PBX extension International calls Receptionist

Voice Mail Integration

DBS uses the Octel voice mail system. This was chosen because it is a worldwide NB standard, and voice mail networking between the various systems was desired.

Cisco CallManager connects to the Octel voice mail system by means of two 24-port FXS cards in the Catalyst 6509 switch. Only 30 of the available 48 ports are used. A 9600 bps Simplified Message Desk Interface (SMDI) link connects the primary Cisco CallManager to the Octel device.

The Octel system is also connected to the corporate NB Octel network. This is done by means of four foreign exchange office (FXO) ports on the Octel device that connect to an old Lucent Definity PBX. This diagram shows how the DBS Octel system connects to both the new and old world.

65368.gif

Note: The initial design did not include the PBX. Rather, the idea was to network the voice mail across through the 24-port FXS card and the VoIP network. But during the pilot, issues were encountered in the way the FXS card handled the Dual Tone Multi-Frequency (DTMF) tones from the Octel device. As a workaround, the 24-port FXS card was replaced with a Cisco IOS gateway. This worked fine, but NB preferred the PBX-based solution.

It is worth noting the resilience characteristics of the voice mail solution. In addition to the voice mail system, there are other single points of failure:

  • The SMDI link is not redundant: A failure of the primary Cisco CallManager will take the voice mail system out of service. Should this situation occur, the NB strategy is to manually move the SMDI cable to the backup Cisco CallManager. Alternatively, an SMDI splitter would allow both CallManagers to be connected at the same time, and allow for automatic failover.

  • Currently both 24-port FXS cards reside in the same Catalyst 6509 chassis. A Catalyst 6509 failure will take the voice mail system out of service. As discussed earlier in this document, there is much to be gained in terms of resilience by adding a second Catalyst 6509.

Gateway Integration

The following table lists the three different types of voice gateways in the NB network.

Platform Interface Type Number of Ports Protocol Connecting To
Catalyst 6509 PRI/E1 8 x 30 channels Skinny PSTN
Catalyst 6509 Analog FXS 2 x 24 Skinny Voice mail
7206VXR PRI/CAS/E1 2 x 30 channels H.323 Legacy PBX

The Catalyst 6509 holds a single eight-port E1 card. Three of these eight ports connect to the PSTN by means of PRI. Each E1 port has its own IP address and its port functions as independent gateways. Call routing to and from the gateways is controlled by Cisco CallManager by means of the Skinny protocol. Users dial a PSTN number by dialing 0, followed by the PSTN number. Incoming calls have the leading digits stripped, and the call is routed to the called IP phone based on the last five digits.

Users dial '9' to select an outside line, and Cisco CallManager removes this digit before presenting the call to the PSTN. NB tried two methods of removing the digit. The first method includes a 'dot' in the route pattern on Cisco CallManager, and all pre-dot digits are discarded before presenting it to the PSTN. The second, and preferred method, is to configure the discard as part of the gateway setup on Cisco CallManager. This is done by setting the Number of digits to strip field to one. This second method is better because it preserves the leading '9' when the number is stored in the Placed calls directory on the IP phone. This means the user can later redial the number from the directory without having to press editdial and add the leading '9'.

The FXS gateways are used solely for voice mail. These gateways are controlled by Cisco CallManager by means of Skinny.

The Cisco 7200 gateway provides connectivity between the IP Telephony network at DBS and the worldwide NB PBX network. This gateway is the only VoIP device not physically located in the DBS building: It is located at the NBAP building, and reached by means of a WAN link.

The Cisco 7200 gateway is fitted with a single two-port E1 port adapter, and uses H.323 to talk to CallManager. Both E1 ports connect to a Nortel Meridian located at NBAP. One port uses QSIG and the other uses Channel Associated Signaling (CAS) to talk to the PBX. Calls to Singapore PBX extensions are routed down the QSIG port, while international calls, using the private voice network (Telnet), are routed through the CAS port.

The mix of CAS and QSIG complicates the setup. CAS is required to provide access to personal identification number (PIN) code-protected international dialing by way of the worldwide Telnet voice network. When a user dials this service, they dial 313xxxxxx, where xxxxxx is a six-digit PIN code. The Meridian application that authenticates this PIN code does not appear to be supported by way of PRI. Hence, the need to use CAS on one of the two trunks for this purpose.

That said, the CAS signaling proved easier to get going than the QSIG trunk. Among the issues encountered were time slots locking up on the Meridian and disagreement on the B-channel numbering. Almost all issues had to do with a mismatch of configuration parameters on the Cisco 7200 and Meridian side. These issues were resolved after Cisco IOS provided a working Meridian configuration.

A copy of the Meridian configuration is included below. This configuration is from a Meridian Option 11C, running software release 24.24. The following software packages are required in order to activate this configuration.

QSIG      263
QSIGGF    305
MASTER    309
QSIG-SS   316
ETSI-SS   323

The following configuration is from the Config Route Data Block.

TYPE  RDB     (Route Data Block)
CUST  00      (Customer Number)
DMOD
ROUT  xx      (Route Number)
DES           (Trunk Description)
TKTP  TIE     (Trunk type - Tie Line or DID)
ESN   NO 
RPA   NO
CNVT  NO
SAT   NO
RCLS  INT     (Route Class - Internal or External)
DTRK  YES     (Digital Trunk - YES)
BRIP  NO
DGTP  PRI2    (Digital Group Type - PRI 30B + D)
ISDN  YES     (YES when DGTP is PRI or PRI2)
MODE  PRA     (ISDN/PRA Route)
IFC   ESGF    (ESGF req QSIG & QSIF GF Pkgs)
SBN   NO
PNI   00000
NCNA  NO
NCRD  NO
CTYP  UKWN
INAC  NO
ISAR  NO
CPFXS YES
DAPC  NO
INTC  NO
DSEL  VOD
PTYP  DTT
AUTO  NO
DNIS  NO
DCDR  NO
ICOG  IAO     (Bothway Trunk In and Out (IAO))
SRCH  RRB
TRMB  YES
STEP
ACOD  xxxx    (Trunk access code)
TCPP  NO
TARG  03
BILN  NO
OABS
INST
ANTK
SIGO  STD
MFC   NO
ICIS  YES
OGIS  YES
PTUT  0
TIMR  ICF     512
OGF   512
EOD   13952
NRD   10112
DDL   70
ODT   4096
RGV   640
GTO   896
GTI   896
SFB   3
NBS   2048
NBL   4096
IENB  5
TFD   0
VSS   0
VGD   6
DTD   NO
SCDT  NO
2 DT  NO
DRNG  NO
CDR   NO
NATL  YES
SSL
CFWR  NO
IDOP  NO
VRAT  NO
MUS   NO
PANS  YES
FRL   0 x
FRL   1 x
FRL   2 x
FRL   3 x
FRL   4 x
FRL   5 x
FRL   6 x
FRL   7 x
OHQ   NO
OHQT  00
CBQ   NO
AUTH  NO
TTBL  0
PLEV  2
OPR   NO
ALRM  NO
ART   0
PECL  NO
DCTI  0
TIDY  xx
SGRP  0
AACR

This configuration is from the D-channel.

ADAN  DCH 12      (D-Channel Number assigned by programmer)
CTYP  MSDL        (D-Channel Card type) 
CARD  02          (Card Location)
PORT  1           (Port Number) 
DES   CISCO_5300  (Description)
USR   PRI         (User type - PRI for ISDN PRA only)
DCHL  2           (Loop the D-Channel will be associated with)
OTBF  32          (Output request buffer)
PARM  RS422 DTE   (Default)
DRAT  64KC        (64kb/s Clear - Don't change) 
CLOK  EXT         (Clock Source - External)
NASA  NO          (Default NO)
IFC   ESIG        (ETSI Q Reference Signaling - MSDL D-Channel ONLY) 
ISDN_MCNT  300    (Default)
CLID  OPT0        (Opt0 is default for ESIG and ISIG interfaces)
CO_TYPE  STD      (100% Compatible)
SIDE  NET         (Network or User Side)
CNEG  2           (2 = Channel is indicated - alternative is accepted)
RLS   ID  **      (Default)
RCAP  COLP        (COLP is default for ESIG, ISIG interfaces)
MBGA  NO          (Default)
OVLR  NO          (Default)
OVLS  NO          (Default)
T310  120         (Default)
T200  3           (Default)
T203  10          (Default)
N200  3           (Default)
N201  260         (Default)
K     7           (Default)

This is a configuration for the loop timers for the PRI tie-line interface.

LOOP 2

MFF    AFF
ACRC   NO
ALRM   REG
RAIE   NO
G1OS   YES
SLP    5       24 H    30    1 H
BPV    128    122
CRC    201     97
FAP    28       1
RATS   10
GP2    20     100 S    12 S    12 S    4 S
MNG1   60 S
NCG1   60 S
OSG1   60 S
MNG2   15 S
NCG2   15 S
OSG2   15 S
PERS   50
CLRS   50
OOSC   0

This configuration is from the B-channel.

TN     00x 0x      (LEN, TN (Terminal Number))
TYPE   TIE         (Trunk Type - Tie Line)
CDEN   SD          (Single Density Card
CUST   0           (Customer 0)
TRK    PRI2        (Trunk Type)
PDCA   x           (Pad Category Table)
PCML   A           (A-law or Mu-Law)
NCOS   0           (Network Class of Service)
RTMB   x y         (Route Member- x is router no., y is member no.)
B-CHANNEL SIGNALING
TGAR   0           (TGAR Restricted Dialing Leave as 0)
AST    NO          (Default)
IAPG   0           (Default)
CLS    CTD DIP WTA LPR APN THFD XREP BARD
P10    VNL         (Class of services)
TKID

The gateway configuration for the two E1 trunks is shown here. Notice that the gateway acts as the network side, while the Meridian is the user side.

controller E1 5/0
 pri-group timeslots 1-31

!-–– Defines PRI trunk.


controller E1 5/1
 framing NO-CRC4
 ds0-group 0 timeslots 1-15,17-31 type e&m-wink-start

!-–– CAS trunk.


interface Serial5/0:15
 isdn switch-type primary-qsig

!-–– Defines Q.SIG signaling.

 isdn protocol-emulate network

!-–– The network side.

 isdn incoming-voice voice
 isdn send-alerting
 isdn sending-complete

The gateway has 15 POTS dial peers. Most of the dial peers point out the QSIG trunk, reflecting the various PBX extensions that are reachable on the PBX side.

The remaining dial peers point out the CAS trunk, directing calls to the Telnet voice network. Also notice that the QSIG trunk is configured to include a Progress Indicator with a value of 8 when it sends an Alerting message back to the PBX. This tells the PBX that the gateway is providing in-band ringback tone, and the PBX opens up the audio path even before a call is answered by the IP phone.

dial-peer voice 33 pots
 preference 1
 destination-pattern 33.......
 direct-inward-dial
 port 5/1:0

!-–– Calls routed out of the CAS trunk.

 forward-digits all
!
dial-peer voice 313 pots
 destination-pattern 313......
 direct-inward-dial
 port 5/1:0
 forward-digits all
!
dial-peer voice 40000 pots
 destination-pattern 4....
 progress_ind alert enable 8

!-–– Gateway to provide ringback.

 direct-inward-dial
 port 5/0:15

!-–– Calls routed out of the QSIG trunk.

 forward-digits all
!
dial-peer voice 8 pots
 destination-pattern 8T
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 7 pots
 destination-pattern 7T
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 6 pots
 destination-pattern 6T
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 5 pots
 destination-pattern 5T
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 16 pots
 destination-pattern 16...
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 13 pots
 destination-pattern 13...
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 2 pots
 destination-pattern 2T
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 1000 pots
 destination-pattern 1...
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 10 pots
 destination-pattern 0...
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 3 pots
 destination-pattern 3...
 progress_ind alert enable 8
 direct-inward-dial
 port 5/0:15
 forward-digits all
!
dial-peer voice 508200 pots
 destination-pattern 5082..
 progress_ind alert enable 8
 port 5/0:15
 forward-digits all

There are only two VoIP dial peers, one pointing to each Cisco CallManager. The dial peers are identical except for the preference, which ensures that calls are routed to the primary Cisco CallManager when available. The progress_ind setup enable 3 tells the gateway to signal to the PBX, by way of a Progress Indicator in the setup message, that the calling party is non-ISDN. Finally, the h225 timeout tcp establish 3 tells the gateway to wait a maximum of three seconds when establishing an H.225 session with the Cisco CallManager. If the Cisco CallManager does not respond within three seconds, the gateway tries the secondary Cisco CallManager.

dial-peer voice 17000 voip
 preference 1

!-–– Route to primary Cisco CallManager is preferred.

destination-pattern 17...
 progress_ind setup enable 3

!-–– Progress indicator = non-ISDN.

 voice-class h323 10
 session target ipv4:10.66.184.13
 dtmf-relay cisco-rtp h245-signal h245-alphanumeric
 codec g711ulaw
 ip precedence 5
 no vad
!
dial-peer voice 17001 voip
 preference 2
 destination-pattern 17...
 progress_ind setup enable 3
 voice-class h323 10
 session target ipv4:10.66.184.14
 dtmf-relay cisco-rtp h245-signal h245-alphanumeric
 codec g711ulaw
 ip precedence 5
 no vad

voice class h323 10
 h225 timeout tcp establish 3

Some of the most persistent issues encountered during the rollout of the NB IP Telephony project pertained to echo. The main echo issues were experienced by the IP phone users when they called certain numbers through the Cisco 7200 gateway. The effort that went into trying to reduce echo was extensive, and the following information attempts to capture the parts of this experience that may be useful to other deployments.

The initial expectation by the IP Telephony design team was for the Cisco 7200 gateway to provide connectivity between the VoIP side and a single PBX at NBAP. As it turned out, what was in fact being provided was connectivity between the VoIP side and the very large, worldwide NB voice network. The NB voice network has a legacy of its own, including a number of echo problems. In the past, NB attempted to tune the power levels of the various nodes in this network to minimize the amount of echo. The network to which the Cisco 7200 gateway was connecting was a network with existing echo problems. It also had varying levels of signal power, depending on the destination of the call. This was a difficult integration.

By introducing a packetized voice solution, with its additional delays, the echo problems were exacerbated. In an effort to handle this, the following adjustments were made.

  • The Cisco 7200 echo cancelers were adjusted to their most aggressive setting.

  • The input gain was lowered.

  • The output attenuation increased on the two E1 trunks.

While this reduced the echo, it had the undesirable side effect that volume levels, when calling certain destinations in the NB voice network, were too low and users were complaining. Due to the mismatch of signal levels on the legacy side, there was no one combination of gain and attenuation that suited calls to and from all destinations. What worked well for calls to Hong Kong created echo on calls to Korea. What worked for Korea resulted in low volume problems for Hong Kong. The configuration below shows the current compromise configuration for the voice ports on the Cisco 7200 gateway.

voice-port 5/0:15
 input gain 0
 output attenuation 3
 echo-cancel coverage 32
 compand-type u-law
 cptone SG

voice-port 5/1:0
 input gain -2
 echo-cancel coverage 32
 compand-type u-law
 cptone SG
 timeouts interdigit 5
 timeouts wait-release infinity
 timing percentbreak 60

Development engineering is currently working on improving the echo canceling capabilities of some Cisco products. NB is awaiting these improvements in order to further reduce the echo.

Alternative solutions were proposed to NB, but the customer has decided to wait for the Cisco improvements. Two proposed workarounds are discussed below in the hope that other projects can benefit from them. The general lesson learned from NB is that a red flag alert should be raised early if the proposed IP Telephony solution connects to a large private legacy voice network. By doing this, these workarounds can be designed and costed into the solution from the beginning.

  • Workaround 1—Insert third party echo cancelers between the Cisco gateway and PBX. Cisco echo canceling technology can currently cancel echo tails that are delayed less then 32 ms. The echoed signal must be subject to an Echo Return Loss (ERL) of at least 6 dB, that is, the received echo signal must be at least 6 dB lower than the originally transmitted signal. In order for it to be worthwhile, the performance of the third party canceler should exceed the above values.

  • Workaround 2—Increase the number of trunks between the Cisco gateway and the PBX. This allows each trunk to be configured with a different gain/attenuation setting. Calls can then be routed across the trunk with the most suitable echo characteristics. For example, calls to Hong Kong may go through trunk 1 while calls to Korea go through trunk 2. The PBX also needs to be able to route calls across the correct trunk, based on where the call originates.

DSP Provisioning for Conferencing and Transcoding

Although G.711 is currently used throughout the network, the intention is to use G.729 across the WAN link between DBS and NBAP. The design has taken this into account by allocating hardware digital signal processor (DSP) resources for conferencing. The hardware resources reside on the Catalyst 6509. Recall that only three of the eight E1 ports are in use for PSTN connectivity. Three of the remaining five ports are used for conferencing.

There are two unused ports on the Catalyst 6509 E1 module that are set up as transcoding resources. There is not currently a need for transcoding, but the need will arise if NB decides to deploy an IP interactive voice response (IVR) server.

Software Versions

The following table lists the software versions used in the NB network at the time this document was written.

Device Version
Catalyst 6509 5.5(3)
Catalyst 4006 6.1(1)
Cisco 7260VXR 12.1(3a)XI5
Cisco CallManager 3.0(8)
IP Phone 7960 P003Q301
WS-X6608-E1 C001W300
WS-X6624-FXS A002S300

Network Management

Network management tools are not currently used to manage the NB IP Telephony network.

Lessons Learned

Discovered Anomalies, Caveats, and Resolutions

The following table summarizes the major issues encountered during the deployment. Details of these issues are discussed earlier in this document.

Caveat Resolution
Echo when interfacing the packet voice network to a large legacy voice network. Commission additional trunks and vary the gain/attenuation so that calls can be routed across a trunk with a suitable setting. Deploy third party echo cancelers. Await improved Cisco echo canceling technology.
Mismatched QSIG parameters on the gateway and PBX side. Obtain working PBX configurations from an existing site using a similar setup.
Digit to select outside line not stored in the placed-calls directory. Discard the leading digit in a gateway configuration and do not use a pre-dot discard action in the route pattern.

TAC Cases

The following table lists all issues that resulted in a TAC case. Also included are other significant issues that were resolved locally by the deployment team.

Case # Description Status and Resolution
B124306 QSIG channel lockout. Initially resolved by reconfiguring the LD17 Channel Negotiation (CNEG) parameter on the Nortel PBX from Option(2) to Option(1). However, problem symptoms reappeared after some time. Subsequently, the PBX vendor changed the PRI QSIG setting on the PBX from ESIG (GF QSIG setting) to ESGF (European QSIG setting). After the modification, channel lockout ceased to occur but only the upper 15 channels were functional. To rectify the problem, the ISDN contiguous-bchan command was removed from the VoIP router.
A612818 B-channel mismatch where Nortel PBX uses channel 31 as the control channel while Cisco voice gateway PRI uses channel 16. Resolved by configuring the ISDN contiguous-bchan command on the PRI QSIG interface. This command is used to specify contiguous bearer channel handling so that B-channels 1 through 30 (skipping channel 16) map to time slots 1 through 31. This is available for E1 PRI interfaces only when the primary-qsig switch type option is configured by using the isdn switch-type command.
B124306 Echo. There are two scenarios in which the 7200 echo cancelers may not be able to cancel out an echo. Scenario 1: The echo is too loud for the cancelers to cancel out. Scenario 2: The echo is delayed by more than 32 ms, which is beyond the coverage of the 7200 cancelers. Scenario 1: Making adjustments to the output attenuation and the input gain on the voice gateway and the paddings on the PBX greatly improved the echo situation. The final settings are:
  • Cisco 7200 PRI Input Gain = 0
  • Ciisco 7200 PRI Output Attenuation = 3
  • Cisco 7200 CAS Input Gain = -2
  • Cisco 7200 CAS Output Attenuation = 0
  • PBX PRI TX Attenuation = 2
  • PBX PRI RX Attenuation = 4
  • PBX CAS TX Attenuation = 0
  • PBX CAS RX Attenuation = 5
The residual echoes occur as static noise in the first 1 to 2 seconds of the RTP stream. The duration is inevitable for the adaptive algorithm to train on the audio stream and profile out a effective cancellation. Scenario 2: An echo tail of more than 32 ms is unlikely in an analog voice network. However, it can occur in a packet voice network. As the echo cancellation coverage is currently at a maximum of 32 ms, development work is under way to produce a code that will integrate a third party G.168 compliant echo canceler (with tail length of at least 64 ms).
Cracker noise, poor voice quality (phone load). Loaded P003Q301 firmware into the IP phone. The Q load is more robust toward jitter and delay.
No ringback to IP phone when calling external number by way of QSIG. The gateway does not generate ringback tone unless the Setup message contains a Progress Indicator (PI) of 3 (origination address is non-ISDN). This is because the gateway assumes that without a PI of 3, the originating switch is ISDN and expects the switch to generate the ringback tone instead. With no PI of 3 setting, the gateway is expecting the ISDN switch to generate the ring, but the ISDN switch is not generating ring. This may be due to an ISDN inter-networking issue. To enable the gateway to generate the ringback tone, configure progress_ind on the VoIP dial peer.
dial-peer voice 1 voip
 destination-pattern 8...
  progress_ind setup enable 3
  session target ipv4:192.168.2.10
  dtmf-relay h245-alphanumeric
  codec g711ulaw
  ip precedence 5
The above will then force the gateway to provide ringback tone for calls coming in the ISDN that tandem out that VoIP dial peer.
A695422 There is a difference in DTMF durations. This may be caused by the fact that the catalyst is not correctly recognizing the DTMF tones sent by the voice mail system. When coming from a Catalyst card, the DTMF duration is 300 ms by default. When coming from voice mail, the duration is 130 ms. The Octel specification requires that at least five digits per second are needed for the handshaking to work properly. The Cisco CallManager currently sends H245-SIGNALLING with a duration of 300 ms, making a total of 300*5 = 1.5 sec. At this duration, the Octel voice mail will have timed out before the voice mail networking headers are completely received. Bypassed the 24-port FXS module (Skinny device) with a Cisco 3640 gateway (H.323 device) loaded with an FXS card.
One-way audio for calls across the voice gateway. The problem is caused by the gateway choosing an IP address other than that of the loopback interface. The loopback is the interface from which CallManager traffic leaves the router. Put h323-gateway voip bind srcaddr ip address on this interface to force the router to use the specified IP address as the RTP source address.
interface Loopback0
 description ::: Loopback for BGP peering
 ip address 192.170.94.34 255.255.255.255
 h323-gateway voip interface
 h323-gateway voip bind srcaddr 192.170.94.34
On the Cisco CallManager, change the H.323 gateway device from a name to an IP address. This also prevents any undesirable issues with reverse DNS/hosts lookups.
Cisco CallManager has a known problem where its services slowly deteriorate over time due to memory leakage. A temporary fix was to reboot the Cisco CallManagers on a weekly basis. Installed Windows 2000 Service Pack 1 and a virtual memory fix to the CallManager servers. As an extra precaution, NB was advised to reboot the CallManagers weekly for the subsequent months to ensure maximum stability. NB should decide to stop the weekly reboots when deemed appropriate.
A944914 WFQ does not work over Multi-PPP over multiple 2 Mbps links. The Service-policy command for LLQ implementation disappears after a while giving indeterminate WAN behavior. Three options are available for handling this issue:
  • For all interfaces in the MPPP bundle, set the bandwidth on the serial interfaces to at least 4800 (4/3 X 3600).
  • Upgrade to 12.2.018 which is a prerelease version of 12.2. This is DE (not TAC) supported. The issue is fixed in this version.
  • Implement priority queuing for another flavor of WAN QoS.
Message button on the phone was not working initially. The following settings are required to enable the Message button:
  • In Service Parameters, enter the voice mail extension number as the value for the VoiceMailDN field.
  • In Enterprise Parameters, enter the voice mail extension number as the value for the MessageDirectoryNumber field.
High level of broadcast traffic from E1 card (WS-X6608) due to a bug in the address resolution protocol (ARP) code on the Catalyst 6509 Skinny gateway module. As a workaround, the E1 cards (WS-X6608) are configured in a separate VLAN. This reduces the size of the ARP cache to a maximum of three entries. At the same time, a new Skinny gateway firmware upgrade (D004R300) has been loaded into the modules to fix the problem.
Loss of CoS across the WAN link. One of the new features in Cisco IOS Software Release 12.1(5)T and later is ToS-to-CoS mapping. With this, the router can set the CoS=ToS before sending anything to the Catalyst 6509. The Catalyst 6509 is then configured for trust-cos on the router port, and picks up the internal DSCP value from there. Cisco QoS maintains the ToS and CoS values using the DSCP.
ARP table was corrupted, resulting in a loss of audio stream when accessing the voice mail system by way of the WS-X6624 card. As a workaround, the FXS cards (WS-X6624) are configured in a separate VLAN. This reduces the size of the ARP cache to a maximum of three entries. At the same time, a new Skinny gateway firmware upgrade (A002S300) has been loaded into the modules.
Frequent hanging of FXS ports connected to the Octel voice mail device. This problem seems common with the Octel voice mail system with analog (FXO) connectivity. The number of hanging ports may vary when the voice mail device is interfaced with different PBX systems. Traditional PBXs normally have the ability to reset individual hanged ports without impacting the overall voice mail operation. Cisco has facilitated a DickTracy utility that enables users to reset and monitor individual ports on the FXS card. The DickTracy utility can be installed on any PC on the network. Run it, and connect to the IP address of your WS-X6624. Once connected, click Option. Start logging, then enter the following commands in the command line field: To get port status, enter 5 show states (this provides a status of each port. With DickTracy, port numbers are 0 based: Port 1 on the blade is port 0 in DickTracy). To reset a port, enter the following:
4 set kill port[x] 

!--- Where x is the 0-based port number.

5 disable port x
5 enable port x
No monitoring available for Multilink setup. Individual serial links cannot be IP enabled. The recommendation is to use Simple Network Management Protocol (SNMP) to poll the interface status. There are several ways to do this. The tested option is to create a UNIX script to utilize the snmpget command to collect the interface status the same way as the ping script command does. Another option is to extend the function of the MRTG (this is a freeware SNMP tool to collect interface utilization) to collect the interface status. The option is to use an SNMP-based network management application.
B300309 Router periodically reboots due to bus error. Anomalous packet handling behavior was detected in early revisions of the PA-2FEISL hardware as described in Cisco bug ID issue CSCdm74172 (registered customers only) . The issues have been addressed through a hardware change to 2-port Fast Ethernet/ISL port adapters. PA-2FEISL-xx cards with hardware revision numbers equal to or later than the hardware revisions listed below are not affected. PA-2FEISL-TX and PA-2FEISL-FX Hardware revision 1.2 Hardware revision 2.0 NB was using hardware revision 1.1.
A953213 Unable to access the Cisco CallManager user page. This issue was resolved using the following procedure.
  1. Go to the CCMAdmin page. Click System on the main menu and then click Enterprise Parameters.
  2. From the Enterprise Parameter page, check the LDAP : Cisco Base and LDAP : User Base parameters. (LDAP : Cisco Base should be o=cisco.com and LDAP : User Base should be ou=Users, o=cisco.com.)
After updating these parameters, all users were able to log in from the user page. The problem was found in Cisco CallManager 3.0.4 where the above mentioned parameters were NULL.

Related Information

Updated: Feb 02, 2006
Document ID: 13968