The fault-tolerant architecture employed by Unified CCE requires two independent communication networks. The private network (using a separate path) carries traffic necessary to maintain and restore synchronization between the systems and to allow clients of the Message Delivery Subsystem (MDS) to communicate. The public network carries traffic between each side of the synchronized system and foreign systems. The public network is also used as an alternate network by the fault-tolerance software to distinguish between node failures and network failures.

Note	The terms public network and visible network are used interchangeably throughout this document.

A third network, the signaling access network, may be deployed in Unified CCE systems that also interface directly with the carrier network (PSTN) and that deploy the Unified CCE architecture. The signaling access network is not addressed in this chapter.

Note	Cisco Unified CCH is deprecated. Use Cisco HCS for Contact Center instead.

The figure below illustrates the fundamental network segments for a Unified CCE system with a duplexed PG and a duplexed Central Controller (with Sides A and B geographically separated).

Figure 1. Example of Public and Private Network Segments for a Unified CCE System

The following notes apply to the figure above:

• The private network carries Unified CCE traffic between duplexed sides of the Central Controller or a Peripheral Gateway. This traffic consists primarily of synchronized data and control messages, and it also conveys the state transfer necessary to re-synchronize duplexed sides when recovering from an isolated state. When deployed over a WAN, the private network is critical to the overall responsiveness of Cisco Unified CCE. It must meet aggressive latency requirements and, therefore, either IP-based priority queuing or QoS must be used on the private network links.

• The public network carries traffic between the Central Controller and call centers (PGs and Administration & Data Servers). The public network can also serve as a Central Controller alternate path, used to determine which side of the Central Controller retains control if the two sides become isolated from one another. The public network is never used to carry synchronization control traffic. Public network WAN links must also have adequate bandwidth to support the PGs and Administration & Data Servers at the call center. The IP routers in the public network must use either IP-based priority queuing or QoS to ensure that Unified CCE traffic classes are processed within acceptable tolerances for both latency and jitter.

• Call centers (PGs and Administration & Data Servers) local to one side of the Central Controller connect to the local Central Controller side through the public Ethernet and to the remote Central Controller side over public WAN links. This arrangement requires that the public WAN network must provide connectivity between Side A and Side B. Bridges may optionally be deployed to isolate PGs and Administration & Data Servers from the Central Controller LAN segment to enhance protection against LAN outages.

• To achieve the required fault tolerance, the private WAN link must be fully independent from the public WAN links (separate IP routers, network segments or paths, and so forth). Independent WAN links ensure that a single point of failure is truly isolated between the public and the private networks. Deploy public network WAN segments that traverse a routed network so that you maintain PG-to-CC (Central Controller) route diversity throughout the network. Avoid routes that result in common path selection (and, thus, a common point of failure) for the multiple PG-to-CC sessions.

For each of the WAN links in the public and private network segments of a Unified CCE deployment, a prioritization scheme is required. Two such prioritization schemes are supported: IP-based prioritization and QoS. Traffic prioritization is needed because it is possible for large amounts of low-priority traffic to get in front of high-priority traffic, thereby delaying delivery of high-priority packets to the receiving end. In a slow network flow, the amount of time a single large (for example, 1500-byte) packet consumes on the network (and delays subsequent packets) can exceed 100 ms. This delay would cause the apparent loss of one or more heartbeats. To avoid this situation, a smaller Maximum Transmission Unit (MTU) size is used by the application for low-priority traffic, thereby allowing a high-priority packet to get on the wire sooner. (MTU size for a circuit is calculated from within the application as a function of the circuit bandwidth, as configured at PG setup.)

A network that is not prioritized correctly almost always leads to call time-outs and problems from loss of heartbeats as the application load increases or (worse) as shared traffic is placed on the network. A secondary effect often seen is application buffer pool exhaustion on the sending side, due to extreme latency conditions.

Unified CCE applications use three priorities: high, medium, and low. However, prior to QoS, the network effectively recognized only two priorities identified by source and destination IP address (high-priority traffic was sent to a separate IP destination address) and, in the case of UDP heartbeats, by specific UDP port range in the network. The approach with IP-based prioritization is to configure IP routers with priority queuing in a way that gives preference to TCP packets with a high-priority IP address and to UDP heartbeats over the other traffic. When using this prioritization scheme, 90% of the total available bandwidth is granted to the high-priority queue.

A QoS-enabled network applies prioritized processing (queuing, scheduling, and policing) to packets based on QoS markings as opposed to IP addresses. Unified CCE provides a marking capability of Layer-3 DSCP for private and public network traffic. Traffic marking in Unified CCE implies that configuring dual IP addresses on each Network Interface Controller (NIC) is no longer necessary because the network is QoS-aware. However, if the traffic is marked at the network edge instead, dual-IP configuration is still required to differentiate packets by using access control lists based on IP addresses.

Note

Note

Layer-2 802.1p marking is also possible if Microsoft Windows Packet Scheduler is enabled (for PG/Central Controller traffic only). However, this is not supported. Microsoft Windows Packet Scheduler is not well supported or suited to Unified CCE, and support is removed in future versions. 802.1p markings are not widely used, nor are they required when DSCP markings are available.

The primary purpose of the UDP heartbeat design is to detect if a circuit has failed. Detection can be made from either end of the connection, based on the direction of heartbeat loss. Both ends of a connection send heartbeats at periodic intervals (typically every 100 or 400 milliseconds) to the opposite end. Each end looks for analogous heartbeats from the other. If either end does not receive a heartbeat after five times the heartbeat period, that end assumes that something is wrong and the application closes the socket connection. At that point, a TCP Reset message is typically generated from the closing side. Various factors can cause loss of heartbeats, such as:

• The network failed.

• The process sending the heartbeats failed.

• The VM with the sending process is shut down.

• The UDP packets are not properly prioritized.

There are several parameters associated with heartbeats. In general, leave these parameters set to their system default values. Some of these values are specified when a connection is established, while others can be specified by setting values in the Microsoft Windows 2008 registry. The two values of most interest are:

• The amount of time between heartbeats

• The number of missed heartbeats (currently hard-coded as 5) that the system uses to determine whether a circuit has apparently failed

The default value for the heartbeat interval between redundant components is 100 milliseconds. One side can detect the failure of the circuit or the other side within 500 ms. The default heartbeat interval between a central site and a peripheral gateway is 400 ms. In this case, it takes 2 seconds to reach the circuit failure threshold.

As part of the Unified CCE QoS implementation, a TCP keep-alive message in the public network connecting a Central Controller to a Peripheral Gateway replaces the UDP heartbeat. A consistent heartbeat or keep-alive mechanism is enforced on public network interface whereas keep-alive mechanism is enforced on private network interface. When QoS is enabled on the network interface, a TCP keep-alive message is sent; otherwise UDP heartbeats are retained.

The TCP keep-alive feature, provided in the TCP stack, detects inactivity and then causes the server or client side to terminate. The TCP keep-alive feature sends probe packets (namely, keep-alive packets) across a connection after the connection has been idle for a certain period. The connection is considered down if a keep-alive response from the other side is not heard. Microsoft Windows 2012 allow you to specify keep-alive parameters on a per-connection basis. For Unified CCE public connections, the keep-alive timeout is set to 5 * 400 ms, matching the failure detection time of 2 seconds with the UDP heartbeat.

The reasons for moving to TCP keep-alive with QoS enabled are as follows:

• In a converged network, algorithms used by routers to handle network congestion conditions can have different effects on TCP and UDP. As a result, delays and congestion experienced by UDP heartbeat traffic can result in connection failures from timeouts.

• The use of UDP heartbeats creates deployment complexities in a firewall environment. With the dynamic port allocation for heartbeat communications, you open a large range of port numbers which weakens the security of your firewall.

In a network where Hot Standby Router Protocol (HSRP) is deployed on the default gateways that are configured on the Unified CCE servers, follow these requirements:

• Set the HSRP hold time and its associated processing delay lower than five times the heartbeat interval (100 ms on the private network and 400 ms on the public network). This level avoids Unified CCE private network communication outage during HSRP active router switch-over.

  Note

  With convergence delays that exceed private or public network outage notification, HSRP fail-over times can exceed the threshold for network outage detection which results in a fail-over. If the HSRP configuration has primary and secondary designations and the primary path router fails over, HSRP reinstates the primary path when possible. That reinstatement can lead to a second private network outage detection.

  For this reason, do not use primary and secondary designations with HSRP convergence delays that approach 500 ms for the private network and 2 seconds for the public network. On the other hand, convergence delays below the detected threshold (which result in HSRP fail-overs that are transparent to the application) do not mandate a preferred path configuration. This approach is preferable. Keep enabled routers symmetrical if path values and costs are identical. However, if available bandwidth and cost favor one path (and the path transition is transparent), then designation of a primary path and router is advised.

• The Unified CCE fault-tolerant design requires the private network to be physically separate from the public network. Therefore, do not configure HSRP to fail-over one type of network traffic to the other network link.

• The bandwidth requirement for Unified CCE must be guaranteed at all times with HSRP, otherwise the system behavior is unpredictable. For example, if HSRP is initially configured for load sharing, ensure that sufficient bandwidth for Unified CCE remains on the surviving links in the worst-case failure situations.

Cisco Unified Communications Manager provides support for Resource Reservation Protocol (RSVP) between endpoints within a cluster. As a protocol for call admission control, RSVP is used by the routers in the network to reserve bandwidth for calls.

RSVP traces the path between two RSVP agents that reside on the same LAN as the phones. The RSVP agent is a software media termination point (MTP) that runs on Cisco IOS routers. The RSVP agents are controlled by Unified Communications Manager and are inserted into the media stream between the two phones when a call is made. The RSVP agent of the originating phone will traverse the network to the RSVP agent of the destination phone, and reserve bandwidth. Since the network routers keep track of bandwidth usage instead of Unified Communications Manager, multiple phone calls can traverse the same RSVP controlled link even if the calls are controlled by multiple clusters.

You might use RSVP in a scenario in which two different clusters provide service to phones at the same remote site. This may occur if a cluster is assigned to handle an IP call center. In the scenario, two users at the same office are serviced by different clusters. RSVP offloads the bandwidth calculation responsibilities of Unified Communications Manager to the network routers.

For more information about Unified Communications Manager RSVP, see the Cisco Collaboration System Solution Reference Network Designs at http:/​/​cisco.com/​go/​ucsrnd.

This section briefly describes the traffic flows for the public and private networks.

In planning QoS, a question often arises about whether to mark traffic in Unified CCE or at the network edge. Each option has its pros and cons. Marking traffic in Unified CCE saves the access lists for classifying traffic in IP routers and switches.

Note	While Cisco allows Microsoft Packet Scheduler with Unified CCE 8.5, it is not supported and future releases will remove this option.

There are several disadvantages to marking traffic in Unified CCE. First, it is hard to make changes. For instance, to change the marking values for the public network traffic, you have to make changes on all the PGs. For a system with more than 30 PGs, for example, all those changes would require quite a lot of work. Second, QoS trust has to be enabled on access-layer routers and switches, which could open the network to malicious packets with inflated marking levels.

Note

In Windows, you can use the Group Policy Editor to apply a QoS policy to apply DSCP Level 3 markings to packets. You can also administer these policies through the Active Directory Domain Controller. This may simplify the administration issue. For more information, see appropriate Microsoft documentation.

In contrast, marking traffic at the network edge allows for centralized and secured marking policy management, and there is no need to enable trust on access-layer devices. A little overhead is needed to define access lists to recognize Unified CCE packets. For access-list definition criteria on edge routers or switches, see Table 1, Table 2, and Table 3. Do not use port numbers in the access lists for recognizing Unified CCE traffic (although they are provided in the tables for reference purposes) because port numbers make the access lists extremely complex and you would have to modify the access lists every time a new customer instance is added to the system.

Note

A typical Unified CCE deployment has three IP addresses configured on each virtual NIC, and the Unified CCE application uses two of them. For remote monitoring using PCAnywhere or VNC, because the port numbers are not used in the access lists, use the third IP address to prevent the remote monitoring traffic from being marked as the real Unified CCE traffic.

The default Unified CCE QoS markings can be overwritten if necessary. The tables below show the default markings, latency requirement, IP address, and port associated with each priority flow for the public and private network traffic respectively, where i# stands for the customer instance number. Notice that in the public network the medium-priority traffic is sent with the high-priority public IP address and marked the same as the high-priority traffic, while in the private network it is sent with the non-high-priority private IP address and marked the same as the low-priority traffic.

For details about Cisco Unified Communications packet classifications, see the Cisco Collaboration System Solution Reference Network Designs at http:/​/​www.cisco.com/​en/​US/​docs/​voice_ip_comm/​uc_system/​design/​guides/​UCgoList.html.

Note	Cisco has begun to change the marking of voice control protocols from DSCP 26 (PHB AF31) to DSCP 24 (PHB CS3). However, many products still mark signaling traffic as DSCP 26 (PHB AF31). Therefore, in the interim, reserve both AF31 and CS3 for call signaling.

This section presents some QoS configuration examples for the various devices in a Unified CCE system.

QoS is enabled by default on private network traffic.

• Disable QoS for the Visible (public) network traffic. For most deployments, disabling QoS for the Visible network traffic ensures timely failover handling.

  You can add QoS markings outside the contact center applications with a Windows Group Policy or by enabling marking on the IP edge routers.

For information about enabling QoS on the router during install, see the Cisco Unified Contact Center Enterprise Installation and Upgrade Guide at http:/​/​www.cisco.com/​c/​en/​us/​support/​customer-collaboration/​unified-contact-center-enterprise/​products-installation-guides-list.html.

This section presents some representative QoS configuration examples. For details about campus network design, switch selection, and QoS configuration commands, see the Enterprise QoS Solution Reference Network Design (SRND).

Note	The marking value, bandwidth data, and queuing policy in the examples below are provided for demonstration purpose only. Do not copy and paste the examples without making corresponding changes in the real working system.

switchport mode trunk 
switchport trunk encapsulation dot1q 
switchport trunk native vlan [data/native VLAN #] 
switchport voice vlan [voice VLAN #] 
switchport priority-extend trust 
spanning-tree portfast

mls qos 
    interface mod/port 
        mls qos trust dscp 

class-map match-all Unified ICM_Public_High
    match ip dscp af31
class-map match-all ICM_Public_Low
    match ip dscp af11

policy-map ICM_Public_Queuing
    class ICM_Public_High
        priority 500
    class ICM_Public_Low
        bandwidth 250

policy-map Converged_Link_Queuing
    class RTP
      priority 400
    class ICCS
        bandwidth 600
    class ICM_Public_High
        bandwidth 500
    class ICM_Public_Low
        bandwidth 250
    class ICM_Private_High
        bandwidth 200
    class ICM_Private_Low
        bandwidth 100

interface mod/port 
    service-policy output ICM_Public_Queuing

access-list 100 permit tcp host Public_High_IP any
access-list 100 permit tcp any host Public_High_IP
access-list 101 permit tcp host Public_NonHigh_IP any
access-list 101 permit tcp any host Public_NonHigh_IP

class-map match-all ICM_Public_High
    match access-group 100
class-map match-all ICM_Public_Low
    match access-group 101

policy-map ICM_Public_Marking
    class ICM_Public_High
        set ip dscp af31
    class ICM_Public_Low
        set ip dscp af11

interface mod/port 
    service-policy input ICM_Public_Marking

Once the QoS-enabled processes are up and running, the Microsoft Windows Performance Monitor (PerfMon) can be used to track the performance counters associated with the underlying links. For details on using PerfMon, see the Microsoft documentation.

This section briefly describes bandwidth sizing for the public (visible) and private networks.

Bandwidth must be guaranteed across the highly available (HA) WAN for all Unified CCE private, public, CTI, and Unified Communications Manager intracluster communication signaling (ICCS). Moreover, bandwidth must be guaranteed for any calls going across the highly available WAN. Minimum total bandwidth required across the highly available WAN for all Unified CCE signaling is 2 Mbps.

This section covers the bandwidth requirements for the private and public networks. This section also discusses bandwidth analysis for the connections from Unified IP IVR or Unified CVP PG to Unified IP IVR or Unified CVP, CTI Server to CTI OS, and Unified Communications Manager intracluster communication signaling (ICCS).

If you use auto configuration, the child PG can send the entire agent, skill group, and route-point configuration to the parent PG. If your deployment does not have much bandwidth available, that much data can take considerable time.

This table lists the approximate number of bytes (worst case) that are sent for each of the data entities. If you know the size of the child PG configuration, you can calculate the total number of bytes of configuration data that is sent. These values are worse-case estimates for sending only one item per record, with each field at its maximum size.

The total amount of data (approximate maximum) sent for this configuration is 61,925 bytes.

To mitigate the bandwidth demands, use any combination of the following options:

• Use fewer call and ECC variables on the child PG.

  Certain messages send call data from the child Unified CCE system to the parent. Using fewer and smaller variables reduces the data sent for these events.

  Note

  Call variables that are used on the child PG are sent to the parent PG, regardless of their use or the MAPVAR parameter value. For example, assume that the child PG uses call variables 1 to 8. These variables are never referenced on the parent PG, so MAPVAR is set to EEEEEEEEEE (Export all but Import nothing). Because the variables are sent to the PG where the filtering takes place, you still require bandwidth. For the reverse situation, you do not use bandwidth. For example, if the map setting is MAPVAR = IIIIIIIIII (Import all but Export nothing), then bandwidth is spared. Call variable data is not sent to the child PG on a ROUTE_SELECT response.

• Use the MAPVAR = IIIIIIIIII and MAPECC = IIIIIIIIII peripheral configuration parameters, rather than the default settings of BBBBBBBBBB for these parameters.

  Without these settings, for every ROUTE_SELECT sent to the child, all Call and ECC variables that are used on the parent are sent to the child. If you use the I (Import) or N (None) option on these parameters, then the Gateway PG does not send these variables to the child system. If you use many call variables and ECC variables on the parent, these parameter settings save bandwidth.

Note	Eliminating Import (I or B setting) of data on the ROUTE_REQUEST from the child to the Gateway PG does not save bandwidth. Even though the Gateway PG does not import the data, the child Unified CCE system still sends it.

This section addresses the traffic and bandwidth requirements between CTI OS Agent Desktop and the CTI OS server. These requirements are important in provisioning the network bandwidth and QoS required between the agents and the CTI OS server, especially when the agents are remote over a WAN link. Even if the agents are local over Layer 2, it is important to account for the bursty traffic that occurs periodically because this traffic presents a challenge to bandwidth and QoS allocation schemes and can impact other mission-critical traffic traversing the network.