Cisco on Cisco

Radio devices ordinarily cannot interoperate unless they operate on the same frequency and use the same mode: conventional, logic-trunked radio (LTR), digital, or other. They also cannot interoperate with other communications devices such as wireline phones, cell phones, IP phones, or PCs. Therefore, people using different types of communications devices need to call a dispatcher with access to multiple types of devices, and the dispatchers relays the message. This wastes critical response time in situations ranging from evacuations to employee medical emergencies to alarm response. The popular alternative, providing radio devices for non-mobile employees, increases capital expense—push-to-talk (PTT) radios typically cost US$400 to $500 apiece. The need for voice interoperability applies to enterprises and public safety agencies alike.

Cisco Systems® has two 24x7 security operations centers (SOCs) in the Americas that dispatch security officers to investigate fire, broken glass, and other alarms in Cisco® offices. The SOC in Research Triangle Park, North Carolina (RTP) is responsible for all Cisco offices east of the Mississippi River, including those in Canada and South America, while the SOC in San Jose, California manages offices west of the Mississippi. Security officers who patrol the campuses communicate with each other and with the SOC using conventional and trunked UHF radios in the 450-MHz band (Figure 1).

But because radio devices can communicate only with other radio devices, which furthermore must operate on the same frequency band, Cisco security personnel could not communicate with people outside their facility. For example, if the SOC in RTP received a glass-break or door-force alarm for the Herndon, Virginia office, where security officers did not have radio systems, SOC personnel would attempt to contact the security officers on a wireline phone, cellular phone, or pager. But wireline phones are not available to officers on patrol, and cellular phones are billed per-minute and can be unreliable depending on weather conditions. Paging systems often do not send a page for several minutes, sometimes do not provide coverage in certain areas, and only allow one-way communication.

Lack of voice interoperability also impeded emergency medical response. Cisco employees report health emergencies such as heart attacks by calling the SOC. The SOC would page the dozens of emergency response team (ERT) volunteers at the emergency location to find someone close to the employee needing help. If the paging system was busy, the page might not be received for five minutes or longer, delaying medical care.

Yet another shortcoming of the conventional radio system was that Cisco managers and executives who were away from the facility could not use wireline or cell phones to participate in radio conversations with personnel at the scene of an incident. "The Cisco safety and security team needs to be able to participate in emergency decisions if an incident occurs-no matter where in the world we happen to be," says Jeff Breeding, Cisco program manager for safety and security. "It's inefficient to require people on the scene to handle two communications devices: a radio to talk to onsite personnel, and a cell phone to talk to offsite personnel."

Cisco solved its radio interoperability challenge by deploying a land mobile radio (LMR)-over-IP solution that takes advantage of Cisco's multicast-enabled IP network. A combination of technologies enables employees to participate in talk groups from any location, using a radio, landline phone, wired or wireless Cisco IP phone, or a PC or laptop with radio-emulation client software. "Employees can even participate from their PCs at home using a Cisco VPN solution," says Murry Gavin, Cisco quality systems staff engineer.

A rooftop antenna receives the signal from a handheld or mobile radio attached to a radio receiver, which converts it to analog output. This analog signal is sent to an LMR-enabled router with a voice interface card, which converts the analog signal to voice over IP (VoIP). (The Cisco SOC uses a Cisco 2600 Series router, but any other LMR-enabled router can be used.) The Cisco LMR Gateway software, which runs on any Cisco router that accommodates voice interface cards, provides additional capabilities for PTT radio communications, such as managing tone-controlled radios, handling fluctuation in audio levels, and overcoming problems with stuck PTT buttons on the radios. Cisco's network infrastructure for the solution includes repeaters that Cisco had deployed years ago for a UHF radio system.

The LMR-over-IP software consists of three components. The dispatcher uses administration server software on a Windows PC to manage talk groups (Figure 2). Media server software makes radios look like H.323 gateways, so that employees can dial into conferences from the public switched telephone network (PSTN); the software also enables the dispatcher to set up conferences combining dissimilar radio systems. A small client application enables PCs to emulate a PTT radio, avoiding the need to purchase radios for employees with access to PCs or laptops. Alternatively, employees can use Cisco IP phones to select and talk on radio channels.

Cisco deployed the LMR-over-IP solution in 2003, in eight offices, including RTP and San Jose. "Now security personnel can monitor real-time, on-the-ground resources via LMR, no matter where they are," says Breeding, who has joined talk groups from his home phone and cell phone. "I can monitor and manage a crisis from anywhere in the world that I have network connectivity-even, say, from a cell phone on the street of Santiago, Chile."

If either the RTP or San Jose SOC experiences a disaster, it can transfer command and control of radios to the other site. An authorized employee can assume command using any networked PC with the appropriate software. The individual logs in and is authenticated, and then the solution downloads the channels and privileges: listen-only or talk. If the server is not available, the dispatcher can use the cached configuration from the previous session.

Response to events like broken glass or fire alarms is no longer delayed while the RTP SOC attempts to contact security officers in remote offices via wireline, cell phone, or pager. Instead, the SOC can communicate directly to security officers' radios via the remote site's UFH radio system, which connects to the SOC via Cisco's multicast-enabled IP network and the Cisco LMR Gateway solution. Gavin estimates that Cisco recovered the cost of the UHF system in Herndon in about a year by reducing cell phone charges.

The same benefit applies to facilities management. In the event of a water leak or electronics failure, facilities personnel can talk directly to security at the scene of the incident. "LMR over IP simplifies SOC communications by giving staff the flexibility to use radios, Cisco IP phones, or the PC client," says Zack Schaefer, global transport wireless lead and network engineer for Cisco IT.

"The LMR-over-IP solution avoids the capital expense of purchasing PTT radios for personnel who have access to a PC," says Schaefer. "We paid for the investment in less than six months." Companies that use dedicated leased lines to control radio systems can achieve additional cost savings. Instead of paying US$100 to $1500 per month for a leased line, depending on line speed, they can use their multicast-enabled IP networks instead by adding Cisco LMR Gateway software to their Cisco routers.

To expedite emergency medical response, Cisco now provides all ERT volunteers with the PC software, and the two or three team leaders in each building with a PTT radio. When the SOC receives a call about a medical emergency such as a possible heart attack, the dispatcher presses a single button that sends a medical alert tone to all radios and PC clients, and then broadcasts the nature of the event and location. "The LMR-over-IP solution empowers ERT to respond in less than a minute-two to five minutes faster than when we paged team members," says Gavin. "That's a time savings that has the potential to make the difference between life and death for a heart-attack victim."

All executives at the Cisco RTP office have PC clients so that they can receive notification from the SOC when significant events occur. "With LMR over IP, we can hold a conference with site supervisors on the scene as well as executives and managers in any location to make informed decisions more quickly," says Breeding. Additional business benefits include increased employee safety as well as greater productivity during and following emergency events. "If an event might require building evacuation, the faster we can make that decision, the faster employees can resume working from home or a wireless hotspot," says Breeding.

The LMR-over-IP solution has twice proved its worth for emergency response. On April 6, 2005, Cisco Security Operations used the LMR-over-IP solution to effectively manage response to a broken 15-inch gas main adjacent to the RTP campus, which forced the company to evacuate one half of the campus during business hours. Breeding was in San Jose at the time. Previously, the standard procedure would have been for Breeding to call the incident commander's cell phone or the SOC. In either case, his comments would have been relayed to officers on the scene and he would not have had direct communication. "Talking through an intermediary takes away valuable time from the incident commander, who has to juggle multiple conversations," says Breeding. "With the LMR-over-IP solution, I dialed in directly using a Cisco IP phone and could communicate as if I were on the scene using a radio."

And when the RTP site lost power on July 6, 2005, Breeding and his team were able to manage the incident from Mexico City, using their laptops with PTT radio-emulation software. "Our laptops acted like radios, enabling us to communicate with onsite personnel as if we, too, were onsite," says Breeding. Without the LMR-over-IP solution, Breeding would have had to call into the SOC from his cell phone and rely on second-hand reports of radio conversations from officers on the scene.

During emergencies, the Cisco security team uses the LMR-over-IP solution as an intercom, sometimes called a "hoot-n-holler" circuit. "In the event of a crisis or emergency, security managers meet in a specified location," says Schaefer. "The dispatcher can activate radio channels to PCs in those locations." Cisco created intercom groups, which include dial-up access, for RTP, New York City, and other areas. Cisco emergency response teams plan to set up their own intercom solutions.

If the RTP or San Jose SOC staff cannot access the network for any reason, the staff in the other facility can now take over command and control. "Any authorized employee with network connectivity can act as the dispatcher, even from the parking lot with a wireless connection or at home using a Cisco VPN solution," says Gavin.

Cisco uses its radio interoperability solution for standard operations as well as emergency response. For example, when Cisco celebrated its 20-year anniversary in 2005 at the North Carolina Museum of Art, security officers used the LMR-over-IP solution to manage crowd control. Similarly, for fundraisers such as 5K or 10K runs, security officers pass out radios to event coordinators.

The solution is also making it possible for the Cisco RTP office to establish a mutual aid agreement with other companies in the vicinity. "We've had meetings about interconnecting during emergency incidents," says Gavin.

Configuring the multicast feature for LMR-over-IP required some experimentation. With the Protocol-Independent Multicast (PIM) protocol, the highest IP address becomes the designated router in the PIM election. In most networks, the upstream gateway router has the first available IP address in the subnet, such as 10.1.1.1, and downstream devices have higher IP addresses, such as 10.1.1.2. "For LMR-over-IP to work, the upstream gateway router must be the PIM designated router," says Schaefer. "The solution is to either set the PIM designated router priority higher on the gateway router or ensure that the IP address is higher than the LMR gateway's interface IP so that it wins the designated router election." Cisco IT wanted the downstream router to be the designated router, so it changed the routers' IP addresses.

"Using multicast in a VoIP network, any setting that is not perfect can degrade performance," says Gavin. Cisco previously had enabled multicast only on its desktop data subnets, but not its voice and data center subnets. To deploy the LMR-over-IP solution, Cisco also enabled multicast on its voice subnets-a task the company had already planned to do to introduce advanced Cisco CallManager capabilities-as well as in data centers with LMR Gateway servers. "Cisco initially did not enable multicast on its WLAN networks, to avoid excessive bandwidth consumption," says Schaefer. "We're currently adding bandwidth to our WLAN infrastructure to support multicast."

Gavin advises companies deploying LMR-over-IP solutions to be aware that every radio is manufactured differently and requires somewhat different one-time configuration. For example, the repeaters used in the Cisco RTP and New York City sites require different plugs. "Keep a copy of the schematics and service manual so you can properly configure the interface between the radio and the Cisco LMR gateway," he says.

The Cisco Security Operations team plans to extend the LMR-over-IP solution to additional sites in Canada, Mexico, Central America, and South America, and possibly in Europe, the Middle East, and Africa. Each country has its own radio equipment and licensing requirements, so Cisco intends to work in each area with a local radio vendor that would manage the licensing and government interactions.

Another plan is to use the Cisco CallManager group-page feature to send messages directly to the built-in speaker and display of Cisco IP phones as well as PC clients and radios. "We're also considering configuring a 'dial-out' feature that, when triggered, will contact specified people at their office phone, home phone, cell phone, pager, or other number," says Gavin. After answering the call, the person will need to answer a challenge for identification purposes, and then will be able to join the intercom session.

Summarizing the benefits of the LMR-over-IP solution, Gavin says, "The LMR-over-IP solution enhances the safety and security of Cisco campuses and employees. It also cuts costs because we need fewer circuits and handheld radios." Breeding concurs, also crediting the LMR-over-IP solution with making the SOC more efficient. "SOC staff can communicate in real time via radio with officers in New York City, Canada, Mexico, and Brazil-any Cisco office," he says. "We'll continue to deploy the solution throughout the Americas, especially in offices with volunteer response teams. Our ability to respond to a crisis will improve exponentially."