Cisco on Cisco

Mobility Case Study: How Wireless IP Communications Supports Fast Emergency Response


Land Mobile Radio over IP improves command and control, avoids costly radios, and speeds up emergency medical response.
BACKGROUND

Radio devices typically cannot communicate with each other unless they operate on the same frequency and use the same mode—whether conventional, logic-trunked radio (LTR), or digital. Equally important, they also cannot communicate directly with wire-line, cell, or IP phones and PCs. Users of different types of communications devices must ask a dispatcher to act as an intermediary. In the case of medical emergencies, evacuations, or other incidents, this work-around is cumbersome and wastes critical time. One approach—providing radios to deskbound incident responders—increases capital expense, because these Push-to-Talk (PTT) devices cost US$400 to $500 per device.

The Cisco San Jose campus of 50 buildings experiences approximately 150 safety and security incidents per year. Examples include equipment thefts, forced doors, or unauthorized personnel entering buildings directly behind Cisco employees who are using access cards. Some 40 of these incidents are considered life threatening. Many of them require assistance from fire departments, ambulances, or the police, who need to work with Cisco onsite emergency teams. Clearly, effective communications and voice interoperability are important to large enterprises and public safety agencies alike.

CHALLENGE

Cisco’s 24-hour Security and Facility Operations Centers (SFOCs) in the Americas dispatch security officers to investigate fire, broken glass, thefts, medical emergencies, and other incidents at Cisco offices. The SFOC in Research Triangle Park (RTP), North Carolina, is responsible for all offices east of the Mississippi River, including those in Canada and South America. The SFOC in San Jose, California, manages offices west of the Mississippi. Security officers who patrol the two campuses communicate with each other and the SFOC via conventional and trunked ultra-high frequency (UHF) radios in the 450-MHz band (Figure 1).

Figure 1. Cisco Security Officers Could Only Communicate with UHF Radios

Click on Image to Enlarge popup

Because of radio device interoperability issues, Cisco security personnel initially could not talk to other Cisco campuses. For example, if the SFOC in RTP received a door-force alarm at the Herndon, Virginia, office—where security officers did not have radios—SFOC personnel would attempt to make contact using a wire-line or cell phone or pager. Unfortunately, patrol officers are not always near wireline phones. Cell phones are billed per minute, and connectivity can be unreliable. In addition, pagers often do not receive messages for several minutes after transmission and support only one-way communication; sometimes paging systems do not provide coverage in certain areas.

Lack of interoperability also slowed emergency medical response. Cisco employees reported health emergencies such as heart attacks by calling the SFOC. In response, the SFOC paged dozens of emergency response team (ERT) volunteers to find someone close enough to provide help. If the paging system was busy, medical care was delayed for five minutes or longer.

Yet another shortcoming of radio communications was that Cisco managers and executives away from campus could not use their phones to join radio conversations with personnel at the scene of an incident. “The Cisco safety and security team needs to be able to participate rapidly in decision-making if an emergency or incident occurs—regardless of where in the world we happen to be,” says Debbie Quintana, program manager, Cisco Global Security and Facilities Operations (SFOC). “It’s inefficient and a waste of time to require people on the scene to handle two communications devices: a radio to talk to onsite personnel and a cell phone to talk to offsite personnel.”

SOLUTION
“The Cisco Safety and Security team needs to be able to participate rapidly in decision-making if an emergency or incident occurs—regardless of where in the world where we happen to be.”

Debbie Quintana
Cisco Global Security and Facility Operations (SFOC) Manager

Cisco met this challenge in 2003 by deploying a Land Mobile Radio (LMR)-over-IP solution that uses Cisco’s multicast-enabled IP network.

The LMR-over-IP solution is straightforward, rapid, and efficient. A rooftop antenna receives transmissions from a handheld or mobile radio receiver and converts them into analog output streams. These signals are sent to an LMR-enabled Cisco 2600 series router with a voice interface card, which converts the signals to voice over IP (VoIP) packets and transmits them over the Cisco network. Cisco’s network infrastructure for the solution also includes routers that Cisco had deployed some years ago for a UHF radio system.

LMR Gateway Software

Cisco LMR Gateway software, which runs on any Cisco router that accommodates voice interface cards (VICs), offers additional capabilities for PTT radio communications. These capabilities include managing tone-controlled radios, smoothing out audio level fluctuation, and handling jammed PTT buttons on the radios.

The Cisco LMR Gateway software consists of three components. Cisco SFOC dispatch uses administration server software on a Windows PC to manage talk groups (Figure 2). Media server software makes radios look like H.323 gateways, allowing employees to conference in from public switched telephone networks (PSTNs). The software also enables a dispatcher to set up conferences of dissimilar radio systems. A small client application emulates a PTT radio, allowing Cisco to avoid purchasing radios for employees who have PCs or laptops. Alternatively, employees can use Cisco IP phones to select and talk on radio channels.

Figure 2. Dispatchers Can Manage Talk Groups, Including Participants with Radios, Wire-Line Phones, Cell Phones, and PC Clients (Mobile Vehicle Interior).

Click on Image to Enlarge popup

If either the RTP or San Jose SFOC experiences a disaster, it can transfer command and control of radios to the other site. An authorized employee can assume command of an incident or emergency using any networked PC with the appropriate software. The individual logs in and is authenticated. The software then downloads channels and listen-only or talk privileges. If the server is not available, the dispatcher can use the cached configuration from the previous session.

More recently, in October 2006, the SFOC significantly expanded the reach of its emergency communications system with the Cisco IP Interoperability and Collaboration System (IPICS), a service solution that offers a new approach to communications interoperability. Based on proven IP standards and technology, Cisco IPICS integrates the LMR network with existing IP telephony, cell phone, and paging systems to link emergency teams with each other through the most easily accessible communications systems. Combining Cisco IPICS with LMR over IP enables security staff and management to participate in talk groups from any location using radios, landline, wired, or wireless Cisco IP phones. It even allows these responders to reach each other through a PC or laptop loaded with Cisco IPICS Push to Talk Management Center (PMC) radio-emulation client software.

LMR-over-IP enables significant emergency response flexibility at Cisco. IPICS has taken its capabilities a critical step further by making it possible to integrate enterprise communications with radio networks. Through its ability to connect LMR/RMS gateways across the United States with Cisco dispatch centers, PMC users, and a variety of communications equipment, IPICS has enabled LMR technology to remain a critical part of the Cisco emergency response system.

RESULTS
More Efficient Command and Control

Response to emergencies and other events is no longer delayed as the SFOCs attempt to contact security officers in remote offices by phone or pager. Instead, SFOCs communicate directly with officers’ radios through the remote site’s UHF/VHF radio system, which connects to the SFOC via the Cisco IP network and the Cisco LMR Gateway solution. Lawrence Ingraham, Cisco Safety and Security program manager, estimates that Cisco recovered the cost of the UHF system in its Herndon office in about a year by reducing cell phone charges.

The same benefit applies to facilities management. Facilities personnel can talk directly to security at the scene of a leak or power outage. “LMR over IP simplifies SFOC communications by giving staff the flexibility to use radios, Cisco IP phones, or the PC client,” says Wayne Homell, Cisco Safety and Security Operations manager.

Reduced Costs

“LMR over IP eliminates the capital expense of purchasing Push-to-Talk (PTT) radios for personnel who have access to a PC,” says Homell. “We paid for the investment in less than six months.” Companies using dedicated leased lines to control radio systems can expect to achieve additional cost savings. Instead of paying US$100 to $1500 per month for a leased line, depending on its speed, they can use multicast-enabled IP networks and Cisco LMR Gateway software on their Cisco routers.

Faster Emergency Medical Response

Cisco now provides all ERT volunteers with PC client software and ERT team leaders with a PTT radio. When the SFOC receives a medical emergency call, the dispatcher presses a single button that sends a medical alert tone to all radios and PC clients and then broadcasts the nature of the event and location. “The LMR-over-IP solution empowers ERT to respond in less than a minute—two to five minutes faster than when we paged team members,” says Ingraham. “That time savings can potentially make the difference between life and death for a heart-attack victim.”

Improved Crisis Management and Executive Communications

All Cisco executives involved with incident management have PC clients, enabling the SFOC to notify them of significant events. “With LMR over IP, we can hold a conference with site supervisors on the scene, as well as executives and managers in any location, which allows them to make informed decisions more quickly,” says Debbie Quintana. Additional business benefits include increased employee safety and productivity during and after emergencies. “If an event requires building or even campus evacuation, the faster we can make that decision, the faster employees can resume working from home or a wireless hotspot,” says Quintana.

“During emergencies, the Cisco security team uses the LMR-over-IP solution as an intercom, sometimes called a hoot-n-holler circuit,“ such as those seen in financial organizations. In the event of a crisis or emergency, security managers meet in a specified location,” says Homell. “The dispatcher can activate radio channels to PCs in those locations.” Cisco has created intercom groups, which include dialup access, on campuses and for large offices.

Enhanced Standard Operations

Cisco uses its LMR over IP for normal operations as well as emergency response. For example, when Cisco celebrated its 20-year anniversary in 2005 at the North Carolina Museum of Art, security officers used the LMR-over-IP solution to manage the crowd. Similarly, for fundraisers such as 10K runs, security officers provide radios to event coordinators.

LMR over IP has also made it possible for the Cisco RTP office to establish a mutual aid agreement with other companies in the vicinity. “We’ve had meetings about interconnecting during emergency incidents,” says Homell.

Figure 3. Cisco IP Interoperability and Communications System Integrates Radio Communications systems with Cell, IP, and Wire-Line Phones, PCs, and Other Communications Devices.

Click on Image to Enlarge popup

Flexibility and Resilience

If staff in one Cisco SFOC cannot access the network for any reason, staff in another can now take over command and control. “Any authorized employee with network connectivity can act as a dispatcher, even from a parking lot with a wireless connection or other remote location using a Cisco VPN solution,” says Homell.

When the RTP campus lost power in July 2005, the SFOC program manager and his team were able to manage the incident from Mexico City, using laptops with PTT radio-emulation software. “Our laptops acted like radios, enabling us to communicate with onsite personnel as if we were there,” says Jeff Breeding, manager, Cisco Workplace Resources. “Without LMR over IP, I would have needed to call the SFOC by cell phone and rely on second-hand reports of radio conversations with officers on the scene.”

Communications Systems Integration and Expansion

The Cisco LMR-over-IP solution integrates with Cisco IPICS—a solution designed to connect PTT radio systems and other communication resources (Figure 3). While LMR over IP and radio emulation software have made it possible to connect disparate radio devices with each other and nonradio devices, Cisco IPICS moves a step further, meshing PCs, Wi-Fi laptops, IP, cell, and wire-line phones, pagers, and PDAs into a coherent communications fabric.

Cisco LMR over IP and IPICS have proven their worth in emergency response situations. In April 2005, Cisco Security Operations used the two solutions to manage response to a broken 15-inch gas main that had forced evacuation of half of the RTP campus during business hours. At the time of the incident, the SFOC program manager was in San Jose. Previously, standard procedure would have been for her to call the incident commander’s cell phone or the SFOC. In either case, communication would have been indirect. “Talking through an intermediary takes away valuable time from the incident commander, who must juggle multiple conversations,” says Quintana. “With IPICS and its Web-based administration console, I dialed in directly using a Cisco IP phone and could communicate as if I were on the scene using a radio. In fact, I could have done this even if I had been in our office in Santiago, Chile.”

The Cisco IPICS architecture leverages the company’s internal voice-quality network. The IPICS servers in use in the United States coordinate eight LMR gateway points distributed throughout Cisco’s large campus locations. This coverage is being expanded to Cisco office locations in Canada, Central and South America, Europe, and Asia Pacific.

At present, 17 different radio channels are available to the Cisco IPICS installation. In addition to the security and safety teams, Cisco Workplace Resources (WPR), the Cisco facilities management team, leverages IPICS—allowing WPR employees in Raleigh, NC to communicate with their East Coast dispatch center in Boxborough, Massachusetts.

Emergencies have already proven the unique capabilities of Cisco IPICS. Recently, a natural gas leak at a Raleigh construction site required the evacuation of the entire Cisco Raleigh campus, including its SFOC. During this major emergency, San Jose-based SFOC dispatchers took over dispatch for East Coast operations. Meanwhile, facilities and emergency managers in the United Kingdom and in several locations in South America communicated directly with the Incident Commander in Raleigh. Similarly, when a burst water pipe flooded one of the San Jose headquarters buildings, managers in Raleigh and Boxborough, using different radio systems, could be patched in to assist with the response. The bottom line? As long as the Cisco network is up, global communication is both possible and highly efficient.

LESSONS LEARNED

Configuring the multicast feature for LMR over IP required some experimentation. With the Protocol-Independent Multicast (PIM) protocol, the highest IP address becomes the designated router in the PIM election. In most networks, the upstream gateway router has the first available IP address in the subnet, such as 10.1.1.1. Downstream devices have higher IP addresses, such as 10.1.1.2. “For LMR over IP to work, the upstream gateway router must be the PIM designated router,” says Ingraham. “The solution is to either set the PIM designated router priority higher on the gateway router or help ensure that the IP address is higher than the LMR gateway’s interface IP, which allows it to win the designated router election.” Cisco IT wanted the downstream router to be the designated router, so it changed the routers’ IP addresses.

“When you use multicast in a VoIP network, any setting that is not perfect can degrade performance,” says Homell. Cisco previously had enabled multicast only on its desktop data subnets, but not its voice and data center subnets. To deploy the LMR-over-IP solution, Cisco also enabled multicast on its voice subnets—a task the company had already planned to do to introduce advanced Cisco Unified Communications Manager capabilities—as well as in data centers with LMR Gateway servers. “Cisco initially did not enable multicast on its WLAN networks, to avoid excessive bandwidth consumption,” says Ingraham. “We’re currently adding bandwidth to our WLAN infrastructure to support multicast.”

Homell advises companies deploying LMR-over-IP solutions to be aware that every radio is manufactured differently and requires somewhat different one-time configuration. For example, the repeaters used in the Cisco RTP and New York City sites require different plugs. “Keep a copy of the schematics and service manual so you can properly configure the interface between the radio and the Cisco LMR gateway,” he says.

NEXT STEPS

The Cisco Security Operations team plans to extend the LMR-over-IP solution to additional sites in Canada, Mexico, Central America, and South America, and possibly in Europe, the Middle East, and Africa. Each country has its own radio equipment and licensing requirements, so Cisco intends to work in each area with a local radio vendor that would manage the licensing and government interactions.

Another plan is to use the Cisco Unified Communications Manager group-page feature to send messages directly to the built-in speaker and display of Cisco IP phones as well as PC clients and radios. “We’re also considering configuring a ‘dial-out’ feature that, when triggered, will contact specified people at their office phone, home phone, cell phone, pager, or other number,” says Homell. After answering the call, the person will need to answer a challenge for identification purposes, and then will be able to join the intercom session.

Summarizing the benefits of the LMR-over-IP solution, Homell says, “The LMR-over-IP solution enhances the safety and security of Cisco campuses and employees. It also cuts costs because we need fewer circuits and handheld radios.” Quintana concurs, also crediting the LMR-over-IP solution with making the SFOC more efficient. “SFOC staff can communicate in real time via radio with officers in New York City, Canada, Mexico, and Brazil—any Cisco office,” she says. “We’ll continue to deploy the solution throughout the Americas, especially in offices with volunteer response teams. Our ability to respond to a crisis will improve exponentially.”