The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
As its city faces exponential growth, Houston Water faces seismic challenges in meeting the needs of the nation’s fourth-largest city: Pacing service expansion while keeping rates affordable, protecting public health and the environment, and providing superior customer service, says Yvonne Forrest, the utility’s deputy director.
“You have to move at a pace your ratepayers can afford.”
Public utilities like Houston Water serve 90 percent of Americans with clean, safe water through more than one million miles of pipes and other infrastructure, most of which was put into place during the first half of the 20th century. This outdated infrastructure can cause catastrophic issues, as cities in Texas saw in February 2021 after rare below freezing temperatures blanketed the state, causing thousands of burst pipes and other water-related challenges.
Just as importantly, the technology that promises to keep the water flowing also faces modernization challenges.
“MWRP (Municipal Waste Recycling Program) has had pipe in the ground for 80-plus years,” says John Sudduth, CIO of the Water Reclamation District of Greater Chicago. “And some back-end data systems have been in place for 20 years.”
Like other public agencies, the pandemic challenged utilities to shift to remote work and operations while maintaining critical services. And though they met this challenge, they must prepare for future disruptions.
“Water utilities and other critical infrastructure organizations will need to be resilient moving forward to maintain business continuity 24x7 and high levels of service while assuring public health in the face of various crises,” says Sielen Namdar, Cisco global industry executive, who leads the organization’s smart water business. Namdar also chairs Smart Water Networks Forum’s (SWAN) Americas Partnership Group.
Technology modernization — and people versed in putting it to work — offer utilities the opportunity to meet growing public health, environmental and consumer needs. This paper outlines the challenges and opportunities for water utility leaders and offers strategies they can follow to develop technology roadmaps that guide them into a smarter, more secure and more sustainable future for their communities.
“We are at the cusp of digital transformation,” says Sree Pulapaka, vice president and CIO of the San Antonio Water System (SAWS). “It’s incredibly important for us to be attracting the right talent, the right skillsets and the right resources to help us on this journey.”
Water utilities faced significant challenges before the pandemic. All told, more than $3.27 trillion in investment is needed to maintain and expand water and wastewater service through 2040, according to the American Society of Civil Engineers (ASCE).1 ASCE gave the nation’s drinking water systems a “C-” on its 2021 infrastructure report card, while wastewater and stormwater systems earned a “D+” and “D,” respectively.2
Aging infrastructure is compounding challenges to reliably deliver service. Many utilities are under environmental consent decrees to manage their combined sewer overflows (CSOs) and sanitary sewers overflows (SSO). An estimated 250,000 to 300,000 water main breaks occur each year nationwide, according to ASCE, and tight budgets have limited efforts to proactively replace infrastructure and improve operational efficiencies.
“I cannot replace a pump because the next pump is more efficient,” says Forrest. “I can replace a pump when the current one dies and no more maintenance can be performed on it.”
While water utilities have made significant investments in the sensors and systems that can provide actionable data to address these challenges, much of it remains “pretty manual,” according to Namdar.
As a result, even dynamic information can be 24 hours old, and where systems have been digitized, the information is often siloed.
Industrial solutions often are “generating a lot of data in their silos, and these various sources of data are not necessarily being integrated into a single source of truth,” Namdar says. “They weren’t designed for it.”
Utilities and public works agencies face renewed calls for more timely information — both to make their own operations more efficient and to better serve their stakeholders.
“Our constituents — the public and the Board — are calling for real-time data,” Sudduth says. “We have to get integrations in place to accommodate what our constituents are asking for.”
The pandemic, which forced utilities to move many operations off-site, also has prompted what Namdar calls a significant mindset shift that may ultimately accelerate modernization efforts, particularly around the security of critical infrastructure.
“Who would have thought that 70 to 80 percent of the workforce at a government utility would be working from home? With the pandemic thrusting everything forward and the added need for secure connectivity, we are seeing water utilities — even the ones who were not thinking about digital transformation — changing their minds,” she says. “The momentum is building up quite a bit.”
Sudduth agrees, pointing to efforts to connect disparate operational and enterprise systems. “We’re in the midst of an IT and OT convergence. We’re looking at the operating technology and trying to upgrade it, but the upgrade path is moving toward an IT solution that requires an entirely different skillset.”
Namdar likens the challenge water utilities face to tackling a 1,000-piece jigsaw puzzle. “Twenty years ago, when it came to asset operations, utilities had visibility into 400 pieces of the puzzle. With digital infrastructure modernization, you can have visibility into 900 or more, offering a much clearer picture.”
To that end, “smart water” systems integrate the full technology stack — from sensors, meters and other data-collection devices attached to industrial control systems to technology that moves this disparate data onto a common operational platform where analytics tools can draw from it to generate insights and inform decisions that can often save time and resources.
“With a denser sensor network, we can get more data, and with the secure connectivity tools we have, we can get actionable real-time data,” says Rocky Smith, Cisco industry solutions architect.
Among the focus areas for modernization efforts:
Operational systems. At the Downers Grove Sanitary District in Illinois, making data accessible from the supervisory control and data acquisition (SCADA) network, which oversees the operational systems, “enabled staff to have eyes and ears on the equipment in our wastewater system,” says Clay Campbell, the district’s administrative supervisor. “Remote access opened a whole new world of operational responsiveness and flexibility. We’ve got buildings an operator might walk through one to three times a day — no one else needs to be there.”
Water quality monitoring. Lack of real-time data has hampered efforts to address sudden changes in wastewater quality, for example after a storm surge. Houston Water partnered with the United States Geological Survey (USGS) to add water quality monitors that track water on its way to the utility’s treatment plants.
“If there’s a heavy rain and we’re seeing water quality changes, we aren’t caught by surprise when it comes to the plant,” Forrest says.
Real-time monitoring can save utilities significant capital expenditures, according to Namdar. “If the quality meets certain thresholds, utilities may not need to invest hundreds of millions of dollars in building additional gray infrastructure to meet the regulatory requirements.”
Environmental data is also important to outside constituents. “Various entities consume our rain gauge data to get weather projections and monitor waterways,” Sudduth says, “We’re in an on-demand world. The general public doesn’t want to wait to get a packet of paper.”
Leak detection. Nearly six billion gallons of drinking water is lost every day nationwide, representing as much as 14 to18 percent of the nation’s overall water use, according to one estimate.3 Some communities lose as much as half their water to leaks, according to Namdar.
With 7,000 miles of distribution and transmission lines, Houston Water historically responded to leaks “when somebody calls them in,” Forrest says. The utility has since installed pressure monitors near critical infrastructure, including hospitals and the airport, to speed response time when leaks occur.
Asset tracking and predictive maintenance. Utilities have billions of dollars of physical assets, much of it aging, but they typically have to wait for a schedule-based maintenance check or until equipment fails to take action. Predictive analytics powered by artificial intelligence and machine learning pulls real-time operating data from critical systems to identify failures before they happen, “preventing catastrophic outcomes and costly downtimes, and maximizing the use of existing infrastructure,” Namdar says.
“These technologies are going to become game-changers for our industry,” says Sudduth. “The sooner we can get immersed in that, the better off we’re going to be.”
Customer service. Many utilities have already modernized their customer-facing operations, creating online portals that allow customers to pay bills and track their water usage. But in a world where citizen expectations are shaped by ecommerce giants, utilities will need to innovate to meet their needs.
In San Antonio, SAWS has had a live chat feature with a customer service agent during business hours for some time, but it is now redoubling efforts to create a virtual assistant to help address the more frequently asked questions posed by its business customers (see sidebar). Ultimately, the project “will enable us to leverage a common and unified framework and expose the same set of questions and answers across a variety of digital formats — text, chat, web and screenless devices,” says Pulapaka.
Technology can also improve customer service at the same time it supports critical field operations. In Houston, tablets connected wirelessly to the utility’s work order system provide field crews with key information — but they also allow workers to update the status of projects for customers.
“Lots of people get frustrated — you see a leak, you call it in and unless it’s an emergency, you don’t see anything happening,” Forrest says.
At the same time as work crews update the website, they also leave paper doorhangers in affected neighborhoods.
“While we’re trying to use technology, we have to recognize we don’t want to leave any of our customers behind. It’s a balancing act.”
Sree Pulapaka came to the San Antonio Water System (SAWS) from an innovation team focused on modernizing airport systems. So, it is not surprising that he is implementing ways to ensure new technology can take flight for the utility’s 1.8 million customers.
“We established a swim lane where we do things faster and in a more agile fashion to prove to the organization the value of initiatives,” he says. “We keep working on relatively small initiatives that we think will bring value.”
Modeled after the legendary Lockheed team which revolutionized aerospace, the SAWS Skunkworks focuses on projects that can scale. It first experimented with using responsive web design on the utility’s portal, which ultimately provided better functionality for mobile devices.
It has now shifted efforts to create a virtual assistant for business customers. Equally important, Pulapaka says, it helps leaders look ahead.
“It helps me figure out my roadmap as opposed to living in the day-to-day role of support and operations and doing things retroactively.”
Clay Campbell calls utilities’ workforce needs “the elephant in the room.”
“An agile, well-equipped and retooled workforce is our biggest challenge,” says Campbell, administrative supervisor of the Downers Grove Sanitary District in Illinois. We have to pivot to identify ways to better leverage our human resources and help them understand what the technology and tools will do for them.”
The key challenge, says John Sudduth, CIO of the Water Reclamation District of Greater Chicago, is “overcoming cultural boundaries and ties.”
“Our operational technology networks have been around for a while, and change is difficult,” he says. “Our approach was to have an open conversation and outline what our objectives were, which was to train employees on the new technology to support upcoming systems.”
Training is critical, adds Yvonne Forrest, deputy director of Houston Water. “Adults learn differently,” she says, pointing to efforts like peer-to-peer training that can more readily acclimate staff.
Another key is getting employees to see the benefits of modernization, Campbell says. Team collaboration tools — common in IT workplaces, less so in utilities — allow “everybody to understand what the plan is — the roadmap for the day, week and month. That’s the beginning of improving employee morale.”
Finally, emphasizing the vital nature of utilities’ work can help attract savvy technology workers. “It’s a challenge, but if we’re able to communicate our mission and show how they can make a contribution, we can overcome this challenge,” says Sree Pulapaka, vice president and CIO of the San Antonio Water System.
As water utilities tackle modernization efforts, many start relatively small — but must ensure that the pieces connect. For example, Houston Water’s most recent efforts with modernization center around manhole monitors, which track when sewer lines begin to back up. What’s new, Forrest says, is that the systems monitoring data automatically add tickets to the work order system to speed the response.
“It’s not cutting edge, but we’re integrating a lot of different tools. Where we have found value, we are really pushing forward.”
That’s important, because the evolving smart water world “is becoming a busy space where there are a lot of disparate solutions,” says Namdar. “We’re trying to work through the clutter and develop trusted solutions that are foundational and won’t stop being relevant in three to four years.”
Modernization is an ongoing journey and water utilities should develop strategies to guide themselves along the way. Among them:
Identify needs and gaps. The most critical part of developing a technology roadmap is truly understanding what information utilities need the most. “Look for your gaps — where you need actionable information,” Forrest says. “Without a plan, you’ll just be chasing the flavor of the month.” And don’t overlook key constituencies who want better actionable data of their own, including elected officials and customers.
Start small. “The important thing is to get started and build from there,” Namdar says. Pilots are an effective strategy — one southwestern utility is testing modernization technology in one of its water treatment plants, with plans to ultimately scale the solution across all of its facilities.
Break down silos. Even utilities that have digitized operational systems often have trouble aggregating it. One key is moving to a common platform. In San Antonio, SAWS had three different platforms handling separate parts of operational technology, according to Pulapaka. “A lot of these systems had their own proprietary hardware, and it was just unmanageable,” he says. The utility engaged a consultant to identify ways to move to a common platform as part of a multi-year plan. “Once they showed us the ROI, we standardized on a common stack,” Pulapaka says.
Today, “SAWS works with enormous amounts of information collected from the SCADA systems and synthesizes the data with enterprise data to make decisions,” he says. “The more information we collect helps guide us to more objective decisions.”
In addition, better information can help automate many decisions or empower less-senior staff to make them, he adds.
Emphasize connectivity. Using traditional wired connections, it could take between three and seven minutes for data from the Downers Grove Sanitary District’s nine wastewater stations to reach operators.
“That’s like light years in this space,” Campbell says. “You have to anticipate where things are right now.”
After rolling out cellular routers, transmission times fell to 12 seconds or less, allowing staff to respond to fast-changing situations immediately.
“During a high flow event, people could be driving from station to station to check,” says Campbell. “Now you can check all nine stations in two to three minutes to see where things stand.”
The district also increased its overall connectivity during the pandemic. “When you have a distributed workforce, it matters significantly,” according to Campbell.
As more employees are becoming accustomed to working remotely, utilities will also face growing pressure to provide cloud-based services and applications, Campbell states. “The expectation is that you’re going to be able to equip them with the access to make an operational decision wherever they are.”
Develop partnerships. It’s important for utilities to learn from each other as they implement tools in the growing smart water space. “Collaboration among different leaders of utilities is important,” Pulapaka says. Engaging with industry organizations such as SWAN, which is dedicated to accelerating digital water for utilities, consultants and technology partners can help utilities navigate new technologies.
Reassess cybersecurity. For Houston, hosting the Super Bowl in 2017 led to a region-wide assessment of cybersecurity — including for utilities. “It showed us where we had vulnerabilities,” Forrest says. More recently, in February 2021, leaders in Oldsmar, Florida, revealed an individual had accessed the computer controlling the chemicals used to treat drinking water for the city and changed the level of sodium hydroxide to 11,100 parts per million (ppm), a significant increase from the normal amount of 100 ppm. While security controls such as automated pH testing would have prevented the poisoned water from being distributed, the incident highlights how much critical infrastructures like water utilities have become vulnerable to cyberattacks.
Smart city devices, such as the IoT sensors that utilities use, present a growing number of potential access points for threats — and could ultimately account for 20 percent of cybersecurity events, according to IDC.4
The massive shift to remote work over the past year has also created the urgent need for reassessment. According to Namdar, utilities must invest in a “holistic and in-depth security approach that addresses internal and external security threats.”
Such a strategy often focuses on minimizing the exposure of industrial control systems through a well-defined separation of the operations environment, the enterprise network and cloud-based applications.
“We’re looking at various technologies to segregate the networks and have developed separate emergency response plans for our corporate and control networks,” Sudduth says.
Given the proliferation of ransomware and other phishing attacks, it is also important to educate the workforce through safety and security audits that assess their ability to resist remote threats. “Educate the soft targets of your staff,” Cambell states.
Rethink risk assessment. Passed into law in 2018, the America’s Water Infrastructure Act (AWIA) required all utilities to develop formal risk assessment and resilience plans, including addressing cybersecurity in those plans for the first time. Many utilities had already developed emergency response plans, and AWIA helped “formalize them and put things in the appropriate buckets,” Forrest says.
At the same time, water utilities must increasingly include climate change in their resilience planning, with the deep freeze in Texas in February 2021 being the latest example of what can happen when unexpected — and severe — weather hits.
The pandemic-related events of the past year also represent an opportunity to think about resilience and disaster recovery through an additional lens.
“Pre-pandemic, the biggest risks organizations were worried about included cyberattacks, climate-type events and terrorism,” Namdar says. “Those are still present, but the pandemic has shone a light on making sure folks can take care of essential services and operations remotely in a sustained way no matter what crisis comes along. This kind of remote connectivity and secure access is key to building resilience for the future.”
In San Antonio, SAWS hopes to use funding earmarked for AWIA compliance to conduct disaster recovery exercises. “We’ve integrated our disaster recovery exercise into business continuity planning,” Pulapaka says.
Address budget challenges. The Water Reclamation District of Greater Chicago is leveraging about $98 million in Water Infrastructure Finance and Innovation Act (WIFIA) loans from the federal Environmental Protection Agency (EPA) to address long-delayed capital improvements, including significant investments in green infrastructure, according to Sudduth. However, most utilities lack funding sufficient to meet their capital needs, which challenges their leaders to rethink the life cycle of all assets — including commodified IT products.
“I’m done buying desktop workstations,” says Campbell. “Last year pretty much dictated that.”
To win support for IT investments, it’s important to connect pilots to specific metrics to measure their impact.
“As with all things in the public sector, it is critical to focus on the most important operational and constituent needs, Forrest states. “At the end of the day, I tell my team the pie isn’t getting any bigger, so we really have to find the best use cases.”
One silver lining of the pandemic is the growing recognition that critical infrastructure like utilities and the individuals who work for them are essential during times of crisis. Forrest remembers that when Houston Water employees were embedded with rescue operations during Hurricane Harvey in 2017, a fast-food chain gave the firefighters free meals but required utility workers to pay. “They didn’t see that we’re the first responders to the first responders. That’s changing.”
This is particularly true as climate change-related events threaten water and other infrastructure. Thousands of Texas residents were without water or under boil orders for many days after the state experienced snow and sub-freezing temperatures in February. Water utility employees frantically worked to restore access and repair broken pipes.
Digital transformation has the potential to bring even greater changes to the industry. Having come to his utility from the healthcare industry, Sudduth says that the sector “is where the healthcare industry was seven to eight years back.” Accelerating efforts, he adds, will require leaders “to bring your organizations together and be open and collaborative about what needs to happen.”
These efforts are critical, though, as utilities face looming threats, like what the city of Oldsmar experienced in the attack on its water treatment plant.
It’s also important to focus on what Sudduth calls the “change temperament” of the organization and identify change agents who can help build support for modernization efforts. “They are the ones who will help you as a CIO pushing change to actually get it through,” he says.
Campbell agrees, urging leaders to focus on how the overall vision is supported by ongoing investments. “You have to have the vision to anticipate what you want to do down the road with these foundational step-by-step technologies, but they do dovetail and layer on top of each other.”
Cisco Smart Water Resources
Cybersecurity for Water Utilities White Paper
Smart Water Networks Forum (SWAN)
Cisco Cyber Vision Enables Active Defense of Industrial Operations
This piece was written and produced by the Center for Digital Government Content Studio, with information and input from Cisco.