Cisco on Cisco

Unified Communications Case Study: How Cisco IT Developed Migration Strategy for Global Voicemail System

Comprehensive global voicemail migration strategy yields costs savings and reduces management burden.
BACKGROUND
About Program Unity

Program Unity is a joint project between the Cisco® IT organization and the Cisco Enterprise Communications Software Business Unit (ECSBU) to replace the existing Avaya Octel voice messaging system with Cisco Unity technology. This program implements a Cisco Unity voice and desktop messaging solution across the global Cisco network—the largest Cisco Unity deployment ever undertaken-to provide all users with a standard phone user interface, a Web client interface, and a host of voice messaging improvements. Three other Cisco IT programs also fall under this project's scope:

  • Adoption of a standard global dialing plan of "8"+7 digits (that is, the number "8" followed by seven digits)
  • Consolidation of more than 250 site-specific Cisco CallManager servers to a global centralized call processing model in about 15 select data centers
  • Migration of Cisco IP Contact Center (IPCC) agents and call-routing applications to a limited number of Cisco CallManager clusters to establish dedicated Cisco Unity Voice Messaging servers for the Cisco IPCC user population

The deployment of Cisco Unity Voice Messaging is the first step toward unified messaging (the integration of voice, e-mail, and fax on the network) and a critical element of the Cisco AVVID (Architecture for Voice, Video and Integrated Data) program. Cisco Unity software comprises a suite of IP-based messaging services that enable unified messaging capabilities on the Microsoft Exchange platform and with Lotus Domino. The implementation of Cisco Unity Voice Messaging reflects Cisco's long-term commitment to converge the network delivery of voice, video, and data with IP telephony to form a complete IP Communications solution.

Program Unity Team

Program Unity consists of a global cross-functional team with members chosen for their expertise in various disciplines and functions. The team was divided into eight tracks, each responsible for a major program deliverable: architecture, design, support, implementation, automation, communications, training, and Cisco IPCC applications. In addition to the program manager, the team included a lead for each track (except for the implementation track, which had a lead for each global region) and representatives from the Cisco Enterprise Communications Software Business Unit, which is responsible for the research and development of Cisco Unity software.

Figure 1. Cisco Voice Messaging Environment-184 Octel Systems (March 2003)

Click on Image to Enlarge popup

Specific site migrations were coordinated by an implementation lead assigned to each of the Cisco global regions-San Jose (Corporate), Americas, Asia-Pacific, and Europe, Middle East, and Africa (EMEA). These leads, in turn, were managed by the global program manager, who ensured that any lessons learned or work developments from the regional migrations were adopted in the global program. Development of the migration solution was a team effort. The architecture, design, and automation teams developed the technical migration process while the communications, support, and training teams produced the necessary solutions to help ensure a successful user experience.

In addition to the Cisco project team, a team from Avaya was responsible for making configuration changes on the Octel systems and for maintenance and operational support of the Octel systems. This team played an integral role to the migration process, and it is important to note that for any Cisco Unity migration, enterprise customers should expect that the team supporting their legacy system will have some role in the system migration.

Cisco Voice Messaging Environment: Before and After Cisco Unity Migration

At Cisco, the voice-messaging environment in early 2003 consisted of 184 Octel voice messaging systems distributed worldwide. These systems were a mix of two types of Avaya Octel systems-the Aria and the Serenade-as well as a few standalone Cisco Unity systems as shown in Figure 1.

In June 2004, the project team began the deployment of Cisco Unity Voice Messaging across all geographic regions to replace the Octel systems with a single Cisco Unity Voice Messaging solution based on 55 Cisco Unity servers. Figure 2 illustrates the solution.

CHALLENGES

The following sections describe the challenges, issues, and constraints that influenced the Cisco Unity migration solution.

Figure 2. Cisco Voice Messaging Environment-55 Cisco Unity Systems (Post-Deployment)

Click on Image to Enlarge popup

Planning the Migration Strategy

The particular challenges of a user migration can be successfully overcome if they are identified and planned for in advance. As part of the planning process, before developing any technical migration solution, the project team first determined the business goals, objectives, preliminary expectations, and policies for migrating to Cisco Unity software.

"In the initial architecture and design program meetings, we analyzed the current Octel network and discussed the business objectives and our desired outcomes for this project before starting to plan the migration details," says Bernadino Caro, program manager for the Cisco Unity deployment within Cisco Systems. "With so many Octel systems distributed globally, we knew that it would take time to migrate users from the Octel systems in the distributed locations to the Cisco Unity systems in the centralized data centers. However, we also knew that we had to have a migration strategy that would allow messages to be sent between the Cisco Unity and Octel subscribers that minimized disruption to our senior staff and other employees who frequently use networked voice messaging."

This section addresses some of the decisions behind the Cisco Unity migration; however these factors will vary depending on your organization's own voice messaging environment and business goals and objectives. Refer to the following documents for more detailed information on planning a Cisco Unity migration:

Business Objectives

The corporate business objectives for migrating to Cisco Unity software affect the selected migration strategy. For example, if an organization's primary goal is to reduce the number of legacy system mailboxes, the migration strategy may be to migrate larger sites first in order to move many subscriber mailboxes at one time. Table 1 outlines some other common business objectives and possible strategies to meet them.

Table 1. Possible Business Objectives for a Cisco Unity Migration
Business Objective Migration Strategy
Reduce mailbox count Migrate larger sites first to move many subscribers (and mailboxes) at one time.
Minimize risk due to unknown factors or inexperience with migration technology Address areas of unfamiliarity or inexperience by migrating a series of smaller sites first to gain experience and to limit users' exposure to potential problems.
Give special consideration to specific users or user roles in the organization Migrate these users (for example, customer contact groups) last, so they are least likely to be affected by potential problems.
Migrate to Cisco Unity Voice Messaging as rapidly as possible Migrate all sites in a region at once. This approach is usually suitable only for small numbers of users or Octel systems, where the nature of the work required for the migration is less complex.
Migrate to Cisco Unity system rapidly, while minimizing risk Migrate one or two smaller sites to test processes, then move the largest sites so that the bulk of the users are moved to Cisco Unity software quickly. Then migrate the majority of the legacy voice messaging systems at smaller sites in a second phase.
Utilize resources as much as possible If multiple regional implementation teams are available, have these teams migrate sites in each region independently and concurrently to make the most of available resources.

The primary business goal for the Cisco project team was to eliminate the legacy systems and begin capitalizing on the cost savings of Cisco Unity software by migrating to Cisco Unity software as rapidly as possible. However, a global "flash cut" approach where all sites worldwide would be transferred to the Cisco Unity solution simultaneously was not a practical strategy for the Cisco enterprise environment for several reasons:

  • This approach would cause an increase in support calls from new users unfamiliar with the new interface, and it would also require a significant amount of Cisco CallManager programming by Cisco IT support staff.
  • A primary program requirement was that at all times during the migration, all users must be able to send and receive networked voicemail and be able to receive recorded voice name confirmation when addressing voice messages. The flash cut approach could not guarantee that networked voicemail service would not be disrupted.
  • Cisco IT support staff would be unable to provide training and service to so many users at once without an unreasonable, although temporary, personnel increase.

The migration strategy would need to maintain both the Cisco Unity and Octel voice messaging systems during the program build-out, testing, and transition. The team determined that a realistic goal would be to convert 75 percent of the users as soon as possible and the remaining 25 percent when feasible, while still maintaining all networked voice-mail functions. This meant that the team would focus on migrating a small number of Octel servers that hosted large numbers of users. The best way to do this was to create a "hybrid" environment using the Cisco Unity Bridge, which provides Octel-to-Cisco Unity messaging interoperability and directory synchronization.

The team developed a phased migration strategy where a few small and medium-sized sites would be cut over before simultaneously cutting over approximately 18,000 users at the campuses in San Jose, California, and Research Triangle Park, North Carolina. This approach would enable the team to migrate the majority of users as quickly as possible and still test and refine implementation processes, validate the Cisco Unity design and architecture, assess the training and communications, and give support teams time to prepare to support approximately 35,000 users.

Legacy Voice System User Information

The program team determined that carrying over some of the existing user information to the new Cisco Unity system would add unnecessary expenses and additional challenges to the migration process. It is important to decide how to handle this data during the planning process, to help develop the migration strategy as well as to guide communications planning for proactively setting user expectations for how this information will be managed. Table 2 lists the policy choices the team made.

Table 2. Legacy Voice System Information Policies
Legacy Information Policy
Distribution lists Distribution lists from the Octel system were not carried over to the Cisco Unity system. Instead, the automation team developed a tool for users to create, maintain, and share distribution lists. Users and system administrators preferred this solution.
Legacy voice messages Voice messages were not carried over from the Octel to the Cisco Unity system. Instead, the old mailboxes were available for 30 days after the cutover. Any response to old messages had to be created in Cisco Unity Voice Messaging as a new message.
Voice names To enable the voice confirmation feature (a feature requested by experienced voice messaging users), all subscribers were required to record their voice names during the first-time enrollment process in Cisco Unity Voice Messaging.

With regard to allowing users to forward legacy voice messages to the new Cisco Unity system, there are costs and benefits of enabling this functionality. If the costs are worth the value provided to experienced voicemail users, it is worth the investment to plan the additional migration activities.

Migration Site Order

To help ensure that the site migrations ran consistently and smoothly, the Program team considered the order in which the sites would migrate to the Cisco Unity system. For example, sites could be cut over by location (for example, migrate all North American sites first) or by user role or title (for example, migrate all executive staff first). The team opted to cut over sites based on the site code number; all users associated with a given site code were migrated at the same time. The team believed that this approach would make it easier for users to address messages to each other and would ultimately help promote acceptance and adoption of the Cisco Unity system.

When deciding the site migration order, the Program team considered the following factors:

  • Experience migrating from the legacy system to Cisco Unity software. A team from Avaya had supported the legacy Octel systems, rather than Cisco IT staff, so the Program Unity team wanted to build in enough time to gain experience both with performing site migrations and with supporting Cisco Unity software internally. The team planned to cut over several small sites first, to gain experience with migrating from both types of Octel systems and with testing the newly developed support processes.
  • Experience migrating from a decentralized legacy system to a single, centralized Cisco Unity system. A major goal of the Cisco Unity deployment was to centralize and reduce the number of voice messaging servers. Early in the deployment schedule, the team planned a small-scale site migration from several decentralized Octel systems to a single Cisco Unity system to learn how to support this type of migration.
  • Validation of the solution architecture and solution infrastructure. In addition to testing new support and migration processes, the team wanted time to validate the solution architecture and to test new user communications and training. By cutting over a series of small sites first and by allowing several weeks between site migrations, the team could assess feedback and lessons learned from one site migration and incorporate this knowledge into subsequent ones. This process also helped the team prepare for migrations at larger sites with more than 1000 users.
  • Training for Cisco IT regional teams. The team planned individual site migrations in each of the Cisco IT global regions to give the implementation teams experience before all regions began to cut over sites concurrently.
  • Voicemail network topology issues. Because of the Cisco voicemail network topology, the team planned to migrate all the Octel Serenade systems in the United States before starting to migrate the Octel Aria systems. In retrospect, network topology was not a significant factor or risk in the team's ability to cut over several sites simultaneously.

After considering all these factors, the Program team developed a three-phase migration strategy. For the first phase, the primary goals were to test the new migration and support processes and to migrate 75 percent of users from the Avaya system to Cisco Unity software as soon as the team had gained enough experience with the initial set of site migrations. Feedback and lessons learned during the smaller site migrations would be used to improve the subsequent, larger ones. Phase 1 of the Cisco Unity migration included the following sites, in this order:

Figure 3. Octel Voice Messaging Environment: Protocols and Message Flow Examples

Click on Image to Enlarge popup

  • Small Americas site: Grand Rapids, Michigan (12 clients)
  • Small Asia-Pacific site: Eight sites in Australia and New Zealand (about 200 clients)
  • Campus: Sydney, Australia (about 700 clients)
  • Medium-sized Americas site: Atlanta, Georgia (250 clients)
  • Campus: Research Triangle Park and San Jose campus locations (about 20,000 clients)
  • EMEA: Brussels and Amsterdam campus sites (about 1300 clients)
  • EMEA: Paris, London, and Munich campus sites (about 2800 clients)

As a lesson learned, the team found it beneficial to migrate smaller sites before larger ones. Spacing out the initial migrations by a minimum of four weeks helped the team in refining the migration and support processes and in improving user communications and training. "If we had rushed to deploy the larger sites after migrating the first set of smaller ones, the Day 2 Operations team would not have been as experienced and the end-user satisfaction scores from the San Jose and Research Triangle Park campus site cutovers would not have been as positive," says Beata Mielcarek, the implementation lead for the Americas region. "By following this strategy, we've become very efficient in completing site migrations-the time required to cut over each site has progressively shrunk and we've had fewer user issues." Larissa Kelly, the implementation lead for the Asia-Pacific region, agrees: "By scheduling all our sites in Australia and New Zealand over a two- to three-week period, we were able to cut over the smaller sites first to gain experience. Then, as we cut over the next sites, we used this knowledge to help us avoid issues experienced by the previous ones. This way, we could focus on discovering new issues at each site-by the time we cut over the Sydney campus, the team felt very comfortable in implementing the migration."

User Policies

Voice messaging user policies (Table 3) were clarified and refined during the planning process as a way to avoid user communications issues and to reduce day 2 support and other administrative costs.

Table 3. Voice Messaging User Policies
User Information Policy
Voice mailboxes Each employee has a single standard Cisco voice mailbox. No multiple voice mailboxes are supported, but in certain instances, authorized employees can arrange to have calls from different phone numbers transferred to the same mailbox.
Mailbox size Default voice mailbox capacity is 60 minutes, but depending on business need, authorized users can request an increased capacity of 180 minutes.
Voice message retention This policy was changed to more strictly enforce a 30-day message retention policy to save the cost of storing, retrieving, and reviewing messages, as well as preserve the company's ability to manage its electronic information. In addition, users may not save any voice messages to their computer or forward voice messages in any medium outside the company.
Distribution lists Previously, system distribution lists were created and maintained by Avaya. To save this cost, the program team developed the Distribution List Manager tool, which enables users to maintain their own shared or personal distribution lists. Users are responsible for identifying any distribution lists that they need and for creating them in the new system.
Faxes Because fax usage within Cisco for business use has declined, the team decided to implement a third-party fax-to-e-mail solution at a cost to those users who choose to request it.
Transcription service Voice message transcription service, where voicemail messages are transcribed and sent as e-mail, is available to employees with manager approval.
Access to large distribution lists Users who want to send voice messages to large voice distribution lists (more than 1500 recipients) are required to obtain access to special Cisco Unity Voice Messaging accounts and obtain training on the best practices for sending voice messages to such a large audience.

Message Addressing Issues Associated with the Octel Serenade International Gateway

In the legacy voice messaging environment, Octel Aria servers were situated in the San Jose, California and the Americas region and Octel Serenade servers primarily were in the EMEA, Americas International, and Asia-Pacific regions. These two server types use incompatible digital protocols to communicate. All communication between these two server types must use Octel analog networking. Because of poor line quality between North America and the rest of the world, a Serenade server using a specialized gateway role was deployed in San Jose to route voice messages between critical Aria servers in the United States and all Serenade servers (Figure 3). The presence of the Serenade gateway had no effect on the Cisco Unity architecture and design, but it did present some voice messaging numbering plan challenges for the migration process.

Figure 4. Voice Message Addressing with the Serenade Gateway

Click on Image to Enlarge popup

Although the Serenade gateway was only used for a few nodes, it required additional consideration from the design and architecture teams. Because the Serenade gateway acts as a relay, nodes routing messages through it had to address messages using the gateway serial number and longer mailbox IDs. However, not all nodes address messages to the same destination this way-in effect, the team was working with two separate legacy numbering plans for voice messaging (Figure 4).

The Avaya team was responsible for modifying the existing Octel systems with these configuration changes during each site migration, and the presence of the Serenade gateway somewhat complicated this process. The program team facilitated the Avaya team's efforts by:

  • Analyzing the Octel voicemail network and then conceptualizing the eventual Cisco Unity network as completely as possible to help plan how to migrate from one system to the other
  • Documenting the migration steps and changes for Avaya in explicit detail
  • Communicating frequently with the Avaya team during the migration process to avoid any possible confusion

Anthony Garcia, Cisco Unity Bridge design lead, explains further: "For each of the phase 1 sites, we analyzed the network paths for how the Octel systems sent messages to the site targeted for migration. Also, we used our requirements for Cisco Unity Bridge addressing format to determine both the specific system changes that we wanted Avaya to perform, as well as the configuration steps on the Cisco Unity Bridge and Cisco Unity Voice Messaging systems that we wanted our internal Cisco IT staff to complete."

SOLUTION

The program team developed a phased migration strategy, intended to move as many Octel system mailboxes as quickly and as prudently as possible. Because the flash-cut approach was deemed not feasible, the team used the Cisco Unity Bridge to preserve voice messaging networking to the Octel systems during the migration period. The team adopted a new voice messaging numbering plan of seven-digit mailbox addresses to be consistent with the centralized addressing model. In addition, the team opted to not carry over users' legacy voice messages and distribution lists, and instead created tools for users to build and maintain their own distribution lists. By ensuring data integrity from the onset, the program team hoped to reduce the number of day 2 support calls and improve user productivity.

The Cisco Unity migration had three phases:

  • Migrate 75 percent of Cisco users rapidly. In this phase, the team cut over a series of small and medium-sized sites to test the migration processes and the Cisco Unity Bridge as well as new communications, user materials, and support processes. During this phase, the team focused on the largest sites in the EMEA and Asia-Pacific regions, and it simultaneously cut over the San Jose and Research Triangle Park campus locations (or about 18,000 users) over one weekend.
  • Migrate all sites outside the United States. Next, the team planned to replace all Serenade systems located in countries that traditionally had poorer line quality to preserve the reliability of their voice message networking.
  • Migrate remaining sites in the United States. In the last phase, the team planned to cut over the remaining U.S. sites. These sites were planned for the last phase since they had the best line quality and the most servers to replace, and thus would take the longest time to migrate.

The Cisco Unity Bridge, specific migration processes, and the voice messaging numbering plan are discussed in the following sections.

Figure 5. Inside View of San Jose, California Data Center

Click on Image to Enlarge popup

Cisco Unity Bridge

The Cisco Unity Bridge was used to preserve voice messaging networking to the Octel systems during the migration period. Acting as a gateway between the Cisco Unity and Octel systems, the Cisco Unity Bridge uses analog connectivity to transport messages to and from the Octel systems and Simple Mail Transfer Protocol (SMTP) to deliver messages or directory updates to Microsoft Exchange. Each Cisco Unity Bridge server has 24 ports where all voice cards are housed in the associated expansion chassis, but each server can be expanded to 48 ports if necessary. All Cisco Unity Bridge servers run on Microsoft Windows 2000 Standard Edition. The Serenade gateway server has 32 analog ports that interact with a single Cisco Unity Bridge server (Figure 5).

The Cisco maintenance contract with Avaya did not include the kind of call detail reports that are typically available to customers, so the message traffic between the Octel systems and the peak and average utilization of the voice ports was not specifically known for migration planning. However, based on PBX calling user patterns and discussions with support, the team decided to have five Cisco Unity Bridge servers (reserving one server as a spare). After the initial migrations, the team used traffic reports from the Cisco Unity systems to determine that this allocation was more than sufficient to meet company needs. Eventually, the team plans to consolidate these servers. Table 4 shows how many ports of voice-messaging traffic between Cisco Unity and Octel systems occurred before and after the San Jose and Research Triangle Park campus cutovers, which took place on September 10-12, 2004.



Table 4. Peak Simultaneous Ports Busy Summary
  6 Sep 7 Sep 8 Sep 9 Sep 10 Sep 13 Sep 14 Sep 15 Sep 16 Sep 17 Sep 6 Sep
Bridge 510 2 9 5 5 8 12 8 5 10 10 2
Bridge 511 2 6 5 4 6 8 7 6 10 22 2
Bridge 512 2 5 2 3 6 12 14 7 14 23 2
Bridge 513 2 2 2 2 3 9 8 10 11 8 2
Total 8 22 14 14 23 41 37 28 45 63 8

The Cisco Unity Bridge servers are located in San Jose and are configured to communicate with the Octel servers. Messages to or from Cisco Unity subscribers, regardless of where they reside on the Cisco Unity network, are always routed using the specific Cisco Unity Bridge server configured to communicate with the Octel subscriber's Octel home server. The assignments of Octel nodes to Cisco Unity Bridge servers were initially based on region and geographic location but also are influenced by traffic loads. As traffic patterns change during the course of the migration, the assignment of Octel nodes can be modified as necessary to even out the message load distribution. However, no changes have yet been necessary.

A single Cisco Unity Bridgehead server in San Jose is configured to manage all Cisco Unity Bridge subscribers and delivery locations during the Octel migration phase, create all public distribution lists, and manage the Voice Profile for Internet Mail (VPIM) users during the future migration to Cisco Unity Unified Messaging. The server also administers the subscriber and delivery location information associated with the four Cisco Unity Bridge servers. These four servers send and receive messages from the following sites:

Figure 6. Cisco Voice Messaging Environment: Protocols and Message Flow During Migration

Click on Image to Enlarge popup

  • Bridge 1: San Jose, Research Triangle Park, one-third of Americas region locations
  • Bridge 2: San Jose, one-third of Americas region locations, partner locations
  • Bridge 3: One-third of Americas region locations
  • Bridge 4: San Jose-based Serenade International Gateway server traffic (EMEA and Asia-Pacific region locations)

Figure 7 shows an example of messages routed from Cisco Unity Voice Messaging to the Octel systems via the Cisco Unity Bridge.

Maintaining User Identities
The Octel NameNet feature allows callers to address messages by spelling the recipient's name and to hear a recorded name confirmation when addressing a message. To enable the dial-by-name feature from Cisco Unity Voice Messaging during the migration, Octel subscribers were also created as Cisco Unity Bridge subscribers. The Cisco Unity Bridge maintains a permanent directory of Cisco Unity subscribers and a NameNet directory of Octel subscribers. For enterprise customers planning to migrate to Cisco Unity software, it is important to decide whether to use this feature and decide how to format text names in the directory before starting the migration, because making changes during the migration is extremely disruptive.

The program team decided to use the NameNet capabilities of the Cisco Unity Bridge only to retrieve and send voice names and changes to voice names. To prepare for the migration, the implementation team spent several months preparing error-free lists of everyone in Cisco with their correct text-name aliases and site locations. The cleaned information was imported directly into the Cisco Unity directory, rather than retrieving directory information from each legacy server. As a best practice, this approach helped the team avoid a common source of non-delivery reports (NDRs) between the new and old messaging systems.

Figure 7. Bridge Message Routing

Click on Image to Enlarge popup

Standardized Global Voice Messaging Numbering Plan

Cisco Unity Voice Messaging allows subscribers to send voice messages in two ways: by leaving a message when the caller does not answer the phone and by using either the Cisco Unity Inbox or the phone to address a message to one or more recipients. Subscribers can address messages by spelling the recipient's name or by entering a number that has been assigned to the recipient by the organization's network voice messaging numbering plan.

Ideally, network voice messaging numbering plans are set up to be consistent with phone system dial plans so that network voice messages can be sent using the same sequence of numbers used to place a phone call. Within the Octel voice messaging environment, the Octel servers used a four- or five-digit mailbox address for users on the same Octel server (that is, at the same site). A seven-digit address was used to send a voice message to recipients on another Octel server (a location identifier prefix plus the recipient's extension). The exception was the San Jose corporate site, a domain of five Octel Aria servers within which all users could send messages using five digits.

The Cisco Unity deployment was a good time to implement a standardized global voice messaging numbering plan and assign unique site codes to each site. With the move to voice over IP (VoIP), employees could make IP telephony phone calls within their site by dialing the recipient's four- or five-digit extension. Calls to different sites required first pressing 8 (the access code) and then dialing the full seven-digit phone number. One of the goals of the new voice messaging numbering plan was for Cisco users to use the same number for phone calls and for voice messages to other users on the VoIP network.

In the new voice messaging numbering plan, telephony and voicemail are considered part of the same dial plan. As with VoIP, Cisco Unity subscribers can press "8" plus seven digits to send a voice message. All mailbox addresses are seven digits long and correspond to the telephone number. The main change is that when sending voice messages to someone at the same site, subscribers now dial seven digits instead of four or five. The San Jose site was large enough to continue allowing subscribers to call and send messages within the Cisco Unity dialing domain using five digits. San Jose employees continue to use the seven-digit address for making phone calls or sending messages outside this domain. In addition, subscribers outside San Jose can send messages to San Jose recipients using the five-digit address.

Migration Process

The Cisco Unity migration process consists of three primary components: data cleanup, data load, and cutover (see Figure 8). The data cleanup process was carried out before any sites were migrated to the Cisco Unity system. This process consisted of reviewing the user account data on the Avaya voice messaging system and associating this data with employee phone information in the Cisco Telephony Number Management (TNM) system. During migration, the prepared data was imported into the Cisco Unity Migration Application, which is part of TNM, and the sites were migrated by site code to Cisco Unity Voice Messaging.

The implementation team frequently split existing sites and assigned each office its own site code within Cisco Unity Voice Messaging. This step supported the new global voice messaging numbering plan and gave users a voice messaging number that matched their corporate phone number. This practice also saved the company money: in places where users were required to dial long-distance to retrieve messages from the shared Octel server, this cost was eliminated by assigning sites to the closest Cisco Unity server.

The program team expected some interoperability issues over the course of the migration. For example, it was possible that users might temporarily experience delayed messages or message delivery failure. It was important for users to be aware of these potential problems, understand the reasons behind them, and know that they were short-lived. The communications team drafted internal messages to address these potential operational issues, while the training team developed Web-based training and other instructional materials for users to have in advance of a site cutover. (See the Communications Strategy case study and the Training Strategy case study under Resources for more details.) Because of the detailed planning and migration testing, there were no interoperability issues. However, it is recommended that organizations prepare for this possible risk because every environment is different.

Before starting the migration, it is important to assess the network and upgrade it where necessary to ensure standard reliable analog or VoIP connectivity. In addition, the system infrastructure should be established and have the following elements installed and configured before setting up the Cisco Unity systems:

  • The Active Directory forest, including Domain Controller/Global Catalog servers (for directory services)
  • Microsoft Exchange organization (for message storage and transport)
  • Phone system (in this case, Cisco CallManager was the only PBX integrated with Cisco Unity Voice Messaging)
  • Cisco Unity Bridgehead server (for centralized location management of voice network delivery)
  • Cisco Unity Bridge servers (for communications with the Octel systems)

Figure 8. Cisco Unity Migration Process

Click on Image to Enlarge popup

Data Cleanup
During the data cleanup phase, the Avaya data files were split into separate comma-separated value (CSV) files, one for each site. This data was verified to accurately populate voicemail users in the Cisco TNM system, which tracks all of the Cisco telephone numbers and voicemail addresses. This step occurred before any site migrations, in an effort to ensure that the Avaya data was accurate and in a consistent format. The implementation team set up data cleanup as a monthly process before the one-time global import of this data into the Application Foundation Software (AFS) system; after this data load, Cisco Unity subscriber moves, adds, or changes (MACs) were done from the TNM system.

Data Load
Data load is a one-time step of importing the voicemail data files into the AFS database. Every site migrating to Cisco Unity Voice Messaging was loaded into the AFS system from the Cisco Unity Migration Console tool, developed by the automation team to manage this migration. The data load happened once, approximately two weeks before the first site migration, and lasted for one week. After this initial data load, the site migrations were managed by the implementation team from the Cisco Unity Migration Console.

During the week that AFS was loaded with the client database, the team halted all MACs as a way to ensure data integrity. The support team developed policies for documenting and tracking the MAC requests that came in during this freeze, then entering them into the system afterward. "Just before starting the deployments, we loosely synchronized the internal Cisco Unity and Cisco Unity Bridge showcase deployments with the information in the Octel directories," says Marc Holloman, manager of the support track. "That experience showed us the importance of having accurate and up-to-date information for starting the production Cisco Unity program."

Site Cutover
Site implementation of the Cisco Unity system starts at this point. Sites are migrated by site code. As shown in Figure 9, after the data load, users at a site progress through three different states: Bridge, Pre-Enroll, and full Cisco Unity status. When a site migrates to a given state, all users with that site code are migrated accordingly. Each of these different states is described in the next sections.

Figure 9. Subscriber Migration States During Cutover

Click on Image to Enlarge popup

Bridge

Using the Cisco Unity Migration Console, the implementation team triggers the creation of Bridge subscribers for a given site. This indicates to the AFS system to create the Octel subscribers as Bridge subscribers on the Cisco Unity Bridgehead server. When a site is in the Bridge migration state, its users are considered Bridge subscribers in Cisco Unity Voice Messaging, but their production mailboxes are still in the Octel voicemail system. Previously migrated Unity subscribers can exchange voice messages with these Bridge subscribers. This migration step is important for enabling the dial-by-name addressing feature from Cisco Unity to Bridge subscribers. This process is equivalent to using the Cisco Unity Bulk Import tool to import users into Cisco Unity Voice Messaging from a CSV file, from Microsoft Exchange, or from Lotus Domino.

Fran McBrearty, implementation manager for San Jose, tells about a lesson learned: "One of the key findings from the San Jose/Research Triangle Park cutover is that Cisco Unity Bridge 3.0.4 does not have a silence detection feature, to trim silence from recorded voice names received from the Octel systems. As a result, there was extra silence on the voice names that were automatically uploaded into Cisco Unity Voice Messaging during the site migration to Pre-Enrollment status, because of the delay in handshaking between the Cisco Unity Bridge and the Octel system. This extra silence created a slower experience for those users who use the spell-by-name feature or who address messages. To resolve this problem, the team developed a tool to trim this extra silence from the imported voice names, with the goal to build both silence detection and silence trimming features into Cisco Unity Bridge 3.0.5 to eliminate this problem for both recorded names and voice messages that go through the Cisco Unity Bridge."

A few days before a site moves from Bridge to Pre-Enrollment status, those sites homed on a single Octel server (and sharing a single site code) are assigned their own unique site code. The implementation lead uses the Change Internal Prefix tool developed by AFS to associate users at each site with a unique site code in TNM. When the site moves to Pre-Enrollment status, users can log in to Cisco Unity Voice Messaging using an identifier based on the new site code. However, the new site-based voicemail identifier is not available in the Cisco directory for these users until the site cuts over to Cisco Unity Voice Messaging and they are considered full Cisco Unity subscribers.

Pre-Enrollment

When a site moves to Pre-Enrollment status, its users exist as Bridge-Enabled Unity subscribers. They can access their Cisco Unity voice mailbox to perform setup tasks and record their voice name and greeting, but voice messages sent to them still go to their Octel production voice mailbox. Bridge-Enabled Unity subscribers can't send or receive any voice messages from their Cisco Unity account. The Bridge-Enabled Unity subscriber capability is not a feature generally available in Cisco Unity Voice Messaging.

About three weeks before the site cutover, the implementation team uses the Cisco Unity Migration Console to move the site to Pre-Enrollment migration status. During Pre-Enrollment, users cannot send or receive any voice messages from Cisco Unity software, but they can enroll in the Cisco Unity system and record their voice name. Each site has an open enrollment period of 2 to 3 weeks before the voice-messaging system cutover. The enrollment process typically takes about 10 minutes: first, users access Cisco Unity Voice Messaging from their computer to change their voice messaging password, then use this new password to access the system by telephone to record their name. This step gives users the opportunity to try the Cisco Unity system before the cutover, set up their personal greeting and notification devices, and gain familiarity with the user training and documentation.

The implementation team set a goal to have 70 percent of users enrolled by the cutover weekend. The team periodically measured enrollment progress with data collected by the Subscriber Information Dump tool (available on www.ciscounitytools.com) and encouraged enrollment participation with a series of well-timed communications. Andy Wishart, project manager for the EMEA region, says, "Our use of e-mail reminders from executive senior staff encouraging site users to pre-enroll helped to substantially reduce the number of support inquiries regarding the first-time enrollment process."

The team has consistently met the 70 percent goal for all site migrations to date. In hindsight, the U.S. team believes that 50 percent participation is probably enough to avoid large support call volumes in the week after site cutovers. Once a site has been cut over, the team has observed a consistent pattern where the percentage of enrolled users levels off when support begins. This percentage may remain stable for several months before increasing to 90 percent enrollment. That may reflect users who rarely use voicemail (such as subcontractors or employees on extended leave) and who do not complete enrollment at the same time as the rest of the site.

Cisco Unity Voice Messaging

Two to three weeks into the pre-enrollment period, after business hours, the implementation team triggers the site for Cisco Unity migration from the Cisco Unity Migration Console. The status of the site (and all its users) is changed to "Unity-Pending." After the migration is complete that evening, the users' status changes to full Cisco Unity subscribers. The Cisco CallManager PBX for the site is configured to forward calls to Cisco Unity Voice Messaging for all users at the site. The Octel system is kept running for 15 to 30 days to allow employees to listen to any saved voice messages and is then decommissioned.

Post Migration
Soon after the site cutover, the implementation team assesses the migration and communicates any lessons learned to the program team that could benefit future site cutovers. An online user survey is distributed to gather feedback on the experience with the site migration, communications, and training materials. A few weeks after the site cutover, a voice message is sent out to remind users at the site that legacy voicemail messages are available only for a limited time.

Migration Tools and Applications
The tools and features mentioned in this section were developed specifically for the Cisco Unity deployment by the Cisco AFS team using limited-release versions of the Cisco Unity system API. These include:

  • Distribution List Manager, a Web-based tool for managing global voicemail distribution lists
  • Cisco Unity Management Application, a Web-based suite of applications for migrating voicemail accounts from Avaya's OctelNet to Cisco Unity Voice Messaging
  • Bridge-Enabled Subscriber, a Cisco Unity subscriber migration status
  • Development of a GUI-based, first-time enrollment process that initializes Active Directory account passwords
  • Cisco Personal Communications Assistant Redirect, a client redirector for Cisco Personal Communication Assistant.

To learn more, see this application note:
http://www.cisco.com/en/US/customer/products/sw/voicesw/ps2237/products_configuration_example09186a00800c52e5.shtml.

Large enterprise customers that need similar migration capabilities should contact their Cisco account team regarding the availability of such Cisco Unity API features or solutions.

Testing
The Unity Program team frequently tested the Cisco Unity system to ensure that systems and processes were working as designed throughout the migration. Four separate test phases occur during the Cisco Unity migration:

  • Phase 1: Tests are run after the Cisco Unity system is installed and configured, but before subscriber accounts are added to the system by running the TNM migration process to create Bridge-Enabled Cisco Unity subscribers.
  • Phase 2: Tests are run after the Bridge-Enabled Unity subscribers exist on the Cisco Unity system, but before the implementation team has announced open enrollment for subscribers.
  • Phase 3: Readiness tests are run just before the start of the site cutover, but before site migration is activated in TNM.
  • Phase 4: Tests are run immediately after TNM confirms the success of the Cisco Unity migration. At this point, subscribers can access the Cisco Unity system and the system can accept both networked voice messages and messages from outside callers.

The program team found that this phased testing process is more efficient and more effective for validating the system configuration than manual system audits. "Developing implementation test procedures was important for avoiding simple configuration errors," says David Neustedter, a project architect with Cisco IT. "We had developed global standards for configuring the Cisco Unity systems, but there were a substantial number of Cisco Unity systems to verify. To facilitate this process, Cisco IT developed a Web-based configuration audit tool that allows the implementation project managers to quickly check configuration settings and verify that patches have been applied to the Cisco Unity systems at the appropriate time during the implementation."

LESSONS LEARNED

Lessons learned in developing the migration strategy include the following:

  • Determine policies for handling legacy voice messages. The program team set a policy to not allow users to move old voice messages to the Cisco Unity system. The team wanted to ensure that all users were using Cisco Unity Voice Messaging as their production voicemail system when the site was cut over. In addition, having two voicemail systems at once for a given site could have increased the support load and costs.
  • Consider partner sites for additional user services. Cisco IT worked with a partner company to provide a service that transcribes voicemail messages into documents. The program team overlooked this partner site during the initial migration planning—a lesson learned is to consider any partner sites that could be networked into your environment.
  • Purge NameNet entries. Given that each Octel system keeps its own copy of the directory information for users at another site, the team decided to purge the NameNet entries on the remaining Octel systems as a site migrated to Cisco Unity Voice Messaging. By purging these entries, the team avoided the possibility of text name mismatches from occurring when Octel users addressed messages to Cisco Unity subscribers. However, Octel users were subsequently required to first send a message to a Cisco Unity subscriber without having name confirmation before the sender's Octel system would update its directory to include the Cisco Unity subscriber's text name.
  • Eliminate blind addressing. Blind addressing was used for testing interoperability with the Octel test mailboxes. However, it also caused users to accidentally misaddress messages to other subscribers. The team overlooked creating Cisco Unity Bridge subscribers for these Octel test mailboxes; had they existed, the team could have turned off blind addressing and eliminated a source of addressing errors.
BEST PRACTICES

In developing the migration strategy, best practices include the following:

  • Single Number Access to Multiple Unity Systems (SNAMUS) application. Multiple Cisco Unity systems required the development of the SNAMUS application. Both the Research Triangle Park and San Jose campus locations used a Cisco IP Interactive Voice Response (IVR) solution to provide single-number access to multiple Cisco Unity servers. Subscribers at these sites dialed a single number to access messages without having to look up the number for their specific Cisco Unity server. The subsequent Cisco Unity release (4.0.4) eliminates the need for this application.
  • Cisco CallManager solutions to better handle Cisco Unity transfer to voicemail. Initially, administrative assistants were required to use a solution based on Cisco Unity Voice Messaging to send callers directly to voicemail without ringing the phone. However, the response from these users indicated that this solution was not as user-friendly as other options. Based on this feedback, the team enabled the Cisco CallManager Idivert feature to handle this transfer.
  • Automated processes to install operating system (OS) software on Cisco Unity servers. Given the number of servers in the Cisco IT deployment, it was necessary for the program team to develop automated processes to build the Cisco Unity servers consistently. Cisco IT wanted to use its own standard OS configuration with Cisco Unity Voice Messaging so its hosting group could better support the system. In addition, this OS configuration better corresponded to Cisco IT's data center support model of monitoring computer servers. By creating a standard process for installing and configuring the operating systems on the Cisco Unity servers, the Cisco IT team saved time during the deployment of the servers.
  • Automated processes to test the accuracy of data inputs during the build process. Cisco IT developed an application to audit the configuration of Cisco Unity systems to ensure accuracy and consistency of information entered into the system. Before starting the migrations, large numbers of user records were prepared for import into Cisco Unity Voice Messaging to initialize the directory information in the system. Custom scripts were generated to compare data from the Cisco human resources system, the Octel voice-mail systems, and the PBX systems. By verifying the data for consistency and ensuring that all users at a site and their voicemail extensions were correctly identified at the onset, the team avoided numerous support cases when the sites eventually migrated.
  • Distribution List Manager tool to manage distribution lists in a hybrid environment. Distribution lists from the Octel system were not carried over to the Cisco Unity system. Instead, the automation team developed the Distribution List Manager tool for users to create, maintain, and share distribution lists. Both end users and system administrators preferred this solution-users can directly create and manage distribution lists as needed, and administrators are now free of this responsibility. Organizations that want to develop a similar tool should contact their Cisco sales account representative, who can help assess whether they qualify for support of the product API for this capability. Alternatively, the Public Distribution List builder tool in Cisco Unity Voice Messaging enables administrators to create public distribution lists.