Good Fight

There’s a war being waged on all our networks, and security researchers around the world are on the front lines. Here’s the inside story of how our elite security-research team neutralized one of the biggest threats in years.

By Eric Adams
Photography by Kenny Braun

What began as an ordinary Saturday afternoon in 2015 for cybersecurity expert Matt Olney (pictured above) turned out to be anything but. He was at home in Maryland, settled in on the sofa with a computer on his lap, doing what he loves best – engaging cybercriminals in a never ending war between good and evil.

Olney is one of more than 250 members of Cisco Talos, our elite security-research team that takes the fight to the bad guys every day, helping to defend the world’s networks from the ever-evolving threat posed by malicious cyberattacks.

On this particular Saturday, Olney set up what’s called a honeypot, an innocuous-looking server designed to attract would-be attackers thanks to alarmingly weak passwords and out-of-date security patches. To further camouflage his intent, Olney made the server appear as if it was located in Singapore.

It might surprise you that Olney, like many of his Talos compatriots, actually enjoys working weekends. “It’s not out of a sense of obligation,” says Olney. “It’s the knowledge that cybercriminals aren’t taking any time off, so why should we?”

Talos analyzes some 1.5 million instances of malware every day, and helps stop 7.2 trillion attacks annually. To do so, Talos maintains the largest threat-detection network in the world, using leading-edge detection and prevention techniques designed to discover, assess, and respond to the latest trends in hacking activities, intrusion attempts, malware, and vulnerabilities.

Within Talos, a handful of closely knit teams focus on different fronts in the fight. The outreach team constantly scans for emerging threats. Other teams reverse-engineer new malware and vulnerabilities and create protections for our customers. Still others focus on getting the word out, issuing public security reports and communicating directly with customers, IT vendors, and service providers – even competitors.

Sometimes, like Olney, they engage in a good, old-fashioned game of gotcha for the fun of it. He activated the honeypot and reached over to take a sip of his drink. Almost before he could set it down, he got a hit.

“Even before I had the honeypot software fully installed, I saw the first failed login attempt,” Olney says.

In other words, cyber attackers were already tinkering with the honeypot’s defenses, attempting a quick infiltration and infection. The attackers’ weapon of choice was a brute-force attack, which amounts to rapidly attempting a barrage of known or common passwords in succession, in hopes of gaining access. “It was surprisingly fast,” Olney says.

With a fish on the hook, he quickly set up more honeypots on a dozen separate networks on different continents just in case he could lure more attacks and zero in on the bad actors involved. These would be impossible to associate with each other. “We routinely set up honeypots pretending to be industrial control systems, web servers, elastic search servers, IP telephony servers, even gas pumps,” explains Olney.

“There’s no legitimate traffic going to these honeypots, so the only traffic is coming from the attackers. Honestly, this was the most blatant daylight robbery we had ever seen.”

Craig Williams
Threat Intelligence Outreach Manager

Digital Detectives

Now the true detective work began. Because Olney’s honeypots surreptitiously recorded every failed login attempt, Talos analysts could match attempts to known “dictionaries” of passwords, databases of more than 450,000 passwords often traded online among hackers.

The threat intelligence team assumed the task of analyzing the traffic patterns. The detection research team reverse-engineered the code for the threat’s DDOS trojan. A team of Talos data analysts sniffed out more clues that might help to zero in on the attackers.

“Our data scientists are experts at plucking the nuances out of data. They have the mathematical rigor to understand the set theory and the clustering algorithms to confidently determine the who, what, and where of an attack,” says Senior Director Matt Watchinski, the security-research veteran who heads up Talos from Columbia, Maryland.

Working together, the Talos teams pinpointed the source of the attacks – just two networks in Hong Kong. After quickly trying out a few colorful names, the members of Talos dubbed the attackers SSHPsychos.

The team quickly learned that SSHPsyhcos was no simple operation. Talos and other security experts estimated the SSHPsychos infrastructure alone had to cost more than $100,000 to set up, not to mention ongoing operational expenses. “You wouldn’t believe the complexity of some of the outfits we see today,” Williams says.

“Our data scientists are experts at plucking the nuances out of data. They have the mathematical rigor to understand the set theory and the clustering algorithms to confidently determine the who, what, and where of an attack.”

Matt Watchinski
Senior Director of Cisco Talos

Cat and Mouse

With the SSHPsychos attackers located, it was time to go after them.

Talos was ready. And it wasn’t alone. Talos has developed trusted relationships across the cybersecurity community, one of which proved vital in the effort to neutralize SSHPsychos.

Olney and team quietly shared what they discovered with threat-research colleagues at Level 3 Communications in Broomfield, Colorado, a multinational telecommunications provider that helps to form the Internet’s global “backbone” infrastructure.

“We collaborate with Cisco Talos to mitigate threats on our network and our customers’ networks. By working together, we create a stronger understanding of bad actors that makes them less successful in their operations,” says Mike Benjamin at Level 3 Threat Research Labs.

The professionals at Level 3 quickly understood the scope of the threat and immediately moved into action. According to the customer protocol it follows, Level 3 couldn’t simply blocklist a site, regardless of the traffic it was generating. Instead, it alerted China Telecom to the illicit activity on its network.

As the investigation quickly expanded, somehow the attackers found out. Almost immediately, SSHPsychos fell silent, for the first time ever making it impossible for Talos to pick up the trail.

One day, these attackers are generating a third of the world’s SSH activity. The next day, they’re gone. And so, perhaps, was the chance to stop them.

“It’s our job in Talos to bring people together and ensure they are well informed as threats evolve. We’re all in this together.”

Joel Esler
Threat Intelligence Manager
at Cisco Talos

Esler wears many hats at Talos. Along with working with law enforcement, he also leads a team that manages several open-source security communities including Snort, the universally acclaimed intrusion-detection system created by Martin Roesch, who then founded Sourcefire. When Sourcefire joined Cisco in 2013, Snort came with it. Esler’s team also manages, where Talos shares information with the public.

“It’s our job in Talos to bring people together and ensure they are well informed as threats evolve,” Esler says. “We’re all in this together.”

Talos and Level 3 were ready to go after the attackers again. This time, they weren’t so polite.

On April 7, 2015, Level 3 took the unprecedented action of black holing, or blocking, all SSHPsychos traffic on its global networks. They also contacted other Internet providers and urged them to follow suit.