The U.S. Department of Commerce: Rapid Response

Background
The need to effectively communicate urgent messages to the people who work in large organizations, corporations and public sector groups — even schools and hospitals — was brought into sharp focus by the September 11, 2001 attacks on the World Trade Center and the Pentagon. The United States government, faced with the task of ensuring the safety of its employees in Washington, DC — and across the country — began seeking new ways to improve its ability to rapidly evacuate people from its buildings in the event of another attack, or threat of attack.

The Office of Personnel Management, the Federal Emergency Management Agency, and the General Services Administration established 24-hour operations centers that are in continuous contact with the FBI, antiterrorism task forces, and law enforcement. The Bush Administration then developed the Federal Emergency Decision and Notification Protocol, which authorizes the three agencies to release as many as 350,000 federal workers in Washington and 1.8 million throughout the United States once a threat has been confirmed. The mandate: evacuate everyone from government buildings within 15 minutes of the alarm signal.

The plan affects every government agency, and every facility. For the U.S. Department of Commerce (DoC), it meant finding a means to evacuate its downtown Washington headquarters, the Herbert C. Hoover Building, which houses 4,000 people (ten percent of the DoC's total workforce), nine of the department's 13 bureaus, and the office of the Secretary of Commerce. The agency's first challenge was to ensure that it could comprehensively and clearly notify every DoC worker of an impending emergency, and quickly convey subsequent evacuation plans.

Challenge
The evacuation plan mandated by the government is designed to give the emergency agencies as much time as possible to evacuate threatened facilities, "... even if it's two, three, four, [or] five minutes," an Office of Personnel Management spokesperson told The Washington Post in August 2002.

The traditional means for emergency notification are public-address (PA) or overhead paging systems. But these systems have notable shortcomings. PA and paging systems frequently are:

• Inequitable — They cannot effectively reach all hearing-impaired employees through a spoken message.
• Inflexible — They cannot easily send different messages to different groups or individuals. For example, an emergency evacuation plan may call for security personnel to receive special instructions, for people in certain building areas to use a specific exit route, or for people who require handicap egress to follow a specific evacuation procedure.
• Ineffective — PA and paging systems can be "tuned out" physically and psychologically. Many offices and conference rooms have volume and on/off switches that can demobilize the PA or paging systems. Frequent use of announcements or overhead pages for non-emergency reasons can cause listener fatigue.
• Insecure — They permit unauthorized messages to be sent. "Pranks" are extremely costly in terms of lost productivity, and reduce the effectiveness of emergency notification systems.
• Intrusive — Non-emergency messages can interrupt telephone and face-to-face conversations, reducing the long-term effectiveness of the notification system.
• Expensive — A single-function, dedicated PA system in a facility as large as the Hoover Building could cost as much as $2 million.

Concurrently, the DoC needed to replace its aging, circuit-switched telephone system. The agency was using 134 Merlin Legend key systems to run its 4,200 analog telephones. Before the events of September 11, the Department had begun wiring the Herbert C. Hoover Building with a digital network infrastructure capable of carrying data, voice and video to each of its occupants.

The Commerce Department had begun to explore IP communications — a comprehensive system of enterprise-class solutions including IP telephony, unified communications, IP video/audio conferencing, and contact center — that facilitate more efficient interactions among its users, and provide the foundation for a collaborative workforce.

IP communications would also allow the Department to replace its aging telephone systems with one phone system, and to provide employees with up-to-date improved communications productivity. The DoC also expected that the maintenance costs for the IP-based system would be less than the maintenance costs for the old voice network, based on a cost/benefit analysis that had been prepared by the U.S. Census Bureau, one of the DoC's divisions, which had implemented a similar system at its facility in Maryland.

The DoC was allotted $1 million by the General Services Administration to devise an effective emergency notification capability. Faced with the mandate to install such a system — and realizing the potential afforded by its new network — the DoC examined its options. It became readily apparent to the department's information technology staff that it would make better business sense to utilize the newly installed digital network infrastructure in the building for a multiservice, full-fledged communications system, rather than to install a separate solution that provided only public address capabilities.

The DoC decided to allocate the $1 million towards an IP-based communications solution with emergency broadcast capabilities. It began to evaluate possible solutions from multiple vendors, including Cisco and Avaya, to satisfy this requirement.

Solution
Cisco Systems worked with Berbee, one of its software partners, to explore the development of an emergency notification application that would take full advantage of the Cisco IP Communications infrastructure. In 2001, Berbee had won a best-in-show award at the annual Cisco Innovation Through Convergence (ITC) Exposition for its AutoAnswer application, which provides one-touch intercom functionality between IP phones.

"We have been developing on the Cisco platform since 2000," says Ken Bywaters, eSolutions sales manager at Berbee. "Cisco provides a superior XML [eXtensible Markup Language] development environment, because it adheres to standards, publishes open interfaces, and is an extremely responsive company with sound engineering competencies."

At the April 2002 Cisco ITC Exposition, Berbee and Cisco demonstrated the emergency notification application — now called InformaCast — to DoC officials. InformaCast simultaneously sends both an audio stream and a text message to multiple Cisco IP Phones. The use of voice and text enables people to receive urgent messages in the most effective way; audio and text can be sent to the entire user population, or to specific groups. If someone is on the phone when a voice announcement is transmitted, the two voice streams are blended so the user can hear both.

InformaCast was selected "Best IP Telephony Application for Government and Transportation" at the 2002 Cisco ITC Exposition. More importantly, the demonstration convinced the Commerce Department in May to select the Cisco/Berbee solution. One month later, the implementation phase began under the supervision of PlanetGov, Inc., of Chantilly, VA, a professional services company and "E-Government" specialist that has been certified in Cisco IP Communications solutions. PlanetGov was a subcontractor to Computer & Hi-tech Management, Inc. (CHM), a systems integrator that was awarded the primary contract by the DoC.

The solution at the Commerce Department includes eight Cisco CallManager call processing systems with Berbee InformaCast; 4,200 Cisco 7960 IP Phones; Cisco 7835-1266 Media Convergence Servers; Cisco Catalyst 4006 series switches in closet locations for access; Cisco Catalyst 6509 switches for distribution and core; and Cisco 7204 and Cisco 7507 routers in the Network Operations Center.

Results
"We did a lot of cut-overs during the night," recalls Mary Souther, PlanetGov's vice president, business development. "Our morning team would then come in and handle all the fixes and updates and patches, and configure the telephones."

The project was completed on schedule in December 2002. "It was an aggressive schedule," Souther says, "but we were able to achieve the deployment successfully within the Department's time frame."

The InformaCast notification system has been tested multiple times, including notification of different broadcast groups — which can be defined by geographic or departmental function criteria. The system has passed the DoC's requirement: to reach the 4,000 workers in the Herbert C. Hoover Building within 90 seconds. In addition, users of the IP Communications system were happy to have access to communications tools offering advanced functionality. According to its cost/benefit analysis, the DoC expects to achieve a return on its capital investment during its fourth year of deployment. The Department is considering additional capabilities, including Cisco Unity unified messaging, Cisco IP/TV and videoconferencing, and additional XML-based applications development.

While the Department of Commerce provided the impetus for the development of InformaCast, the solution is also being used in other environments. Schools are inquiring about implementing InformaCast as part of their new IP Communications infrastructure, where it can serve as both a teacher/student notification system and a classroom bell.

The Town of Palm Beach, FL, has successfully installed the Cisco-Berbee solution in a distributed environment. The system links nine remote sites, and its flexibility provides both inter- and intradepartmental communications; for example, it offers voice and text paging services for the town's three fire stations. It is also used by the municipal IT and human resources departments to send messages to all employees. What used to take an hour of phone calls can now be accomplished in minutes. "We're extremely satisfied with the solution," says Jim Palmer, the systems analyst for the town. "It makes communications so much easier."

For the people who work in the Department of Commerce's Herbert C. Hoover Building, it provides an extra degree of preparedness in case of an emergency. "You just don't know what might happen next," a federal official, discussing the Federal Emergency Decision and Notification Protocol, told The Washington Post. "Who knows? But next time, we feel we'll be better prepared."

April 28, 2003