Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Eight IT disaster recovery lessons learned from Hurricane Maria

Teams who restored network connectivity after Hurricane Maria share disaster recovery lessons learned from the rebuilding efforts in Puerto Rico.

Rami Shakra, field connectivity director, NetHope

While the devastation of Hurricane Maria has left many lasting effects on the region, the storm provided valuable lessons in how to deal with a natural disaster of this scope. For aid workers like Rami Shakra and Matt Altman, capturing these lessons is critical.

“You’re only months away from the next hurricane season,” Shakra said. “It just overwhelms you with how much preparedness work you need to get into. We brought in all the key players and captured what we could do better and how we could prepare.”

Shakra is a field connectivity director at NetHope, a nonprofit that helps reestablish communications services after disaster strikes. He was the first NetHope aid worker on the scene in Puerto Rico and worked closely with other organizations like Federal Emergency Management Agency (FEMA), the Red Cross, and the Cisco Tactical Operations, or TacOps, team, to rebuild in Puerto Rico. Matt Altman is a network engineer on the Cisco TacOps team. (For more on Shakra’s work with NetHope and TacOps to reestablish connectivity, see “Rebuilding Puerto Rico after Hurricane Maria.”)

In January 2018, NetHope aid workers deployed in Puerto Rico met with volunteers from Google, Facebook and Cisco to brainstorm on key takeaways. The disaster recovery lessons learned are outlined here.

1. Triage your disaster recovery strategy. One disaster recovery strategy that worked in Puerto Rico was to use a three-tiered system to re-establish connectivity in the region. That allowed teams to triage and prioritize tasks, which was important given the scope of the disaster and the number of people in need. According to Shakra and Altman, there are three main goals:

  • Support first responders who distribute aid.
  • Connect local providers to help them recover services, enabling providers to help others.
  • Provide connectivity to the public. Teams identified town squares, for example, that could connect multiple providers and provide limited Wi-Fi access to the public.

2. Build indigenous capacity. Another key lesson is that regions like Puerto Rico benefit from building indigenous, local capacity that can act as first responders. “Ninety percent of the workforce is in Puerto Rico,” Shakra explained. “You now have people on the ground who are trained and are the real first responders. We have built that capacity in people."

3. Have lightweight, portable solutions. Another key disaster recovery lesson learned is that teams on the ground need easily portable satellite connections—sometimes inflatable or possibly just lightweight—that can be hand-carried on a plane, then substituted for more reliable, stable connections later on. Teams also need pre-tested technologies that are ready to roll out.

4. Have equipment reserves. Building capacity also means having satellites, generators and other equipment on-site ready to deploy in the event of a disaster. Shakra said that NetHope has appropriated space in a warehouse in Puerto Rico with this equipment.

5. Build redundancy into your DR plan. Another key tactic—often elusive in vulnerable and under-resourced areas—is building redundancy into your disaster recovery plan. Milton Riutort, who manages communications at the University of Puerto Rico's Utuado campus, noted that having relationships with more than one ISP can be critical in disaster situations. “This is the correct way to do it, to manage the risk,” Riutort emphasized. “We are clear that we need at least two providers.”

Matt Altman: network engineer, Cisco TacOps

6. Use cloud-based network technologies. Cloud-based technologies also play a critical role in managing tenuous connectivity. Technologies such as Cisco Meraki, a cloud-based wireless and switching technology, allow IT operators to manage and monitor network traffic remotely.

Even today, Altman monitors the networks set up in Puerto Rico from TacOps headquarters in the United States—often from his mobile phone. “With Meraki, it changed how we deploy completely,” he said. “It allows us to leave equipment behind and troubleshoot whether we’re back in the U.S. or off to another location.” Meraki also segments and manages network connections at a granular level, preventing the network from being flooded with traffic or enabling access to insecure sites.

7. Conditions change. TacOps’ Altman noted that as teams rebuilt infrastructure on the island, they might sometimes forget the ephemerality of the environment. So, he noted, a team might point a satellite in a certain direction, only to have vegetation that had been destroyed grow back and cover the signal.

8. Build your future with solar. Finally, Shakra, Altman and Riutort all stressed the importance that solar energy will play in the future. While the island continues to run on generators, these power sources need maintenance, hog petrol, can break down and are hardly a clean technology. For all these reasons, aid workers and residents alike understand the importance of cultivating renewable energy sources.

Milton Riutort: communications manager, University of Puerto Rico, Utuado

“We’re trying to move away from dependency on electricity, on generators and move more towards solar energy,” Shakra said. NetHope and others are working with organizations like Rocky Mountain Institute to build solar capacity and make sure that rebuilt infrastructure is placed in less vulnerable locations.
Riutort of the University of Puerto Rico already had a project under way to install solar panels on his home. “It is a personal project that I’ve been working on for a long time, and Maria hurried me up,” he joked. His house is up and running with power from generators, batteries and these panels.

For more on Cisco NetHope:

For more on Cisco resources:

Lauren Horwitz

Lauren Horwitz is the managing editor of Cisco.com, where she covers the IT infrastructure market and develops content strategy. Previously, Horwitz was a senior executive editor in the Business Applications and Architecture group at TechTarget;, a senior editor at Cutter Consortium, an IT research firm; and an editor at the American Prospect, a political journal. She has received awards from American Society of Business Publication Editors (ASBPE), a min Best of the Web award and the Kimmerling Prize for best graduate paper for her editing work on the journal article "The Fluid Jurisprudence of Israel's Emergency Powers.”