The
Good Fight

There’s a war being waged on all our networks, and security researchers around the world are on the front lines. Here’s the inside story of how our elite security-research team neutralized one of the biggest threats in years.

By Eric Adams
Photography by Kenny Braun

What began as an ordinary Saturday afternoon in 2015 for cybersecurity expert Matt Olney (pictured above) turned out to be anything but. He was at home in Maryland, settled in on the sofa with a computer on his lap, doing what he loves best – engaging cybercriminals in a never ending war between good and evil.

Olney is one of more than 250 members of Cisco Talos, our elite security-research team that takes the fight to the bad guys every day, helping to defend the world’s networks from the ever-evolving threat posed by malicious cyberattacks.

On this particular Saturday, Olney set up what’s called a honeypot, an innocuous-looking server designed to attract would-be attackers thanks to alarmingly weak passwords and out-of-date security patches. To further camouflage his intent, Olney made the server appear as if it was located in Singapore.

It might surprise you that Olney, like many of his Talos compatriots, actually enjoys working weekends. “It’s not out of a sense of obligation,” says Olney. “It’s the knowledge that cybercriminals aren’t taking any time off, so why should we?”

Cisco Talos At-A-Glance

Talos analyzes some 1.5 million instances of malware every day, and helps stop 7.2 trillion attacks annually. To do so, Talos maintains the largest threat-detection network in the world, using leading-edge detection and prevention techniques designed to discover, assess, and respond to the latest trends in hacking activities, intrusion attempts, malware, and vulnerabilities.

Within Talos, a handful of closely knit teams focus on different fronts in the fight. The outreach team constantly scans for emerging threats. Other teams reverse-engineer new malware and vulnerabilities and create protections for our customers. Still others focus on getting the word out, issuing public security reports and communicating directly with customers, IT vendors, and service providers – even competitors.

Sometimes, like Olney, they engage in a good, old-fashioned game of gotcha for the fun of it. He activated the honeypot and reached over to take a sip of his drink. Almost before he could set it down, he got a hit.

“Even before I had the honeypot software fully installed, I saw the first failed login attempt,” Olney says.

In other words, cyber attackers were already tinkering with the honeypot’s defenses, attempting a quick infiltration and infection. The attackers’ weapon of choice was a brute-force attack, which amounts to rapidly attempting a barrage of known or common passwords in succession, in hopes of gaining access. “It was surprisingly fast,” Olney says.

With a fish on the hook, he quickly set up more honeypots on a dozen separate networks on different continents just in case he could lure more attacks and zero in on the bad actors involved. These would be impossible to associate with each other. “We routinely set up honeypots pretending to be industrial control systems, web servers, elastic search servers, IP telephony servers, even gas pumps,” explains Olney.

Massive Attack

What happened next was nothing short of stunning. Within 20 minutes, an attack had logged into all the honeypots with fake security credentials. Every single one.

Olney immediately suspected that someone was building a distributed denial of service (DDOS) attack capability, in which thousands of systems are compromised and infected with a Trojan that sits dormant and undetected for as long as the attacker chooses. Later, these infected systems can be activated to overwhelm or corrupt another system or site, effectively shutting it down.

It wasn’t the type of attack that surprised Talos, but rather its scale. The hackers used Secure Shell, or SSH, an encryption protocol that allows legitimate users to gain secure remote access to networks. Talos discovered that this single group of bad actors sending out brute-force attacks was at one point responsible for fully a third of all SSH activity on the entire Internet.

“That’s unheard of,” says Craig Williams, who leads the Talos outreach team. “This was a massive brute-force attack. There’s no legitimate traffic going to these honeypots, so the only traffic is coming from the attackers. Honestly, this was the most blatant daylight robbery we had ever seen.”

“There’s no legitimate traffic going to these honeypots, so the only traffic is coming from the attackers. Honestly, this was the most blatant daylight robbery we had ever seen.”

Craig Williams
Threat Intelligence Outreach Manager

Digital Detectives

Now the true detective work began. Because Olney’s honeypots surreptitiously recorded every failed login attempt, Talos analysts could match attempts to known “dictionaries” of passwords, databases of more than 450,000 passwords often traded online among hackers.

The threat intelligence team assumed the task of analyzing the traffic patterns. The detection research team reverse-engineered the code for the threat’s DDOS trojan. A team of Talos data analysts sniffed out more clues that might help to zero in on the attackers.

“Our data scientists are experts at plucking the nuances out of data. They have the mathematical rigor to understand the set theory and the clustering algorithms to confidently determine the who, what, and where of an attack,” says Senior Director Matt Watchinski, the security-research veteran who heads up Talos from Columbia, Maryland.

Working together, the Talos teams pinpointed the source of the attacks – just two networks in Hong Kong. After quickly trying out a few colorful names, the members of Talos dubbed the attackers SSHPsychos.

The team quickly learned that SSHPsyhcos was no simple operation. Talos and other security experts estimated the SSHPsychos infrastructure alone had to cost more than $100,000 to set up, not to mention ongoing operational expenses. “You wouldn’t believe the complexity of some of the outfits we see today,” Williams says.

“Our data scientists are experts at plucking the nuances out of data. They have the mathematical rigor to understand the set theory and the clustering algorithms to confidently determine the who, what, and where of an attack.”

Matt Watchinski
Senior Director of Cisco Talos

Cat and Mouse

With the SSHPsychos attackers located, it was time to go after them.

Talos was ready. And it wasn’t alone. Talos has developed trusted relationships across the cybersecurity community, one of which proved vital in the effort to neutralize SSHPsychos.

Olney and team quietly shared what they discovered with threat-research colleagues at Level 3 Communications in Broomfield, Colorado, a multinational telecommunications provider that helps to form the Internet’s global “backbone” infrastructure.

Good Guys vs. Bad Guys

“We collaborate with Cisco Talos to mitigate threats on our network and our customers’ networks. By working together, we create a stronger understanding of bad actors that makes them less successful in their operations,” says Mike Benjamin at Level 3 Threat Research Labs.

The professionals at Level 3 quickly understood the scope of the threat and immediately moved into action. According to the customer protocol it follows, Level 3 couldn’t simply block a site, regardless of the traffic it was generating. Instead, it alerted China Telecom to the illicit activity on its network.

As the investigation quickly expanded, somehow the attackers found out. Almost immediately, SSHPsychos fell silent, for the first time ever making it impossible for Talos to pick up the trail.

One day, these attackers are generating a third of the world’s SSH activity. The next day, they’re gone. And so, perhaps, was the chance to stop them.

Takedown

Frustrated, the Talos team suspected the SSHPsychos masterminds had been spooked into hiding for good. Incredibly, just as suddenly as they disappeared, the attackers reappeared.

“They shut down for five full days,” explains Williams. “Then, instead of at least resuming on a different network, they picked up their boxes and simply went down the street and started up again.”

It didn’t take long for Talos and Level 3 to zero in the operation once again. They also alerted law enforcement. “We know their timelines are a bit longer, but we gave them the all the information we had about servers, IP addresses and actors involved,” Threat Intelligence Manager Joel Esler says. “We can’t operate in isolation.”

“It’s our job in Talos to bring people together and ensure they are well informed as threats evolve. We’re all in this together.”

Joel Esler
Threat Intelligence Manager
at Cisco Talos

Esler wears many hats at Talos. Along with working with law enforcement, he also leads a team that manages several open-source security communities including Snort, the universally acclaimed intrusion-detection system created by Martin Roesch, who then founded Sourcefire. When Sourcefire joined Cisco in 2013, Snort came with it. Esler’s team also manages TalosIntel.com, where Talos shares information with the public.

“It’s our job in Talos to bring people together and ensure they are well informed as threats evolve,” Esler says. “We’re all in this together.”

Talos and Level 3 were ready to go after the attackers again. This time, they weren’t so polite.

On April 7, 2015, Level 3 took the unprecedented action of black holing, or blocking, all SSHPsychos traffic on its global networks. They also contacted other Internet providers and urged them to follow suit.

Cisco Talos on Emerging Security Threats

Quiet Satisfaction

SSHPsychos turned out to be one of the biggest takedowns of the year, and Talos was hailed in security circles for its contribution in stopping the rogue operation.

While law enforcement made no arrests, and SSHPsychos still exists today, it operates at a fraction of the force it managed before. “We continue to see activity from them, but the traffic is a couple of orders less in magnitude. It gives me a warm and fuzzy,” says Williams.

Williams’s reaction captures security researchers’ central struggle. They usually can’t actually catch the perpetrators, so they must take satisfaction in knowing they’ve done some good by making it harder for the bad guys to operate.

“You have to accept that the best you can hope for much of the time is a tie,” says Watchinski.

Don’t think for a minute that stops Watchinski and the rest of the Talos team from taking pride in the important work they do. “I’m proud of Cisco’s security vision, not just in building devices and detecting malware, but in protecting our customers and the Internet as a whole. We are proud to be the white hats in a world of good and evil,” he concludes.

All told, Olney suspects SSHPsychos compromised tens of thousands of servers worldwide – many unknowingly still infected. The ultimate intent of the SSHPsychos attacks is still unclear. Are the bad actors behind them after profits? Intellectual property? Other sensitive corporate or government information? “There are a lot of clues, but intent is often difficult to nail down,” admits Olney.

Talos will continue to monitor this threat and work to keep us protected from it. But because it’s still unclear exactly who is behind SSHPsychos and where they are located, the odds that the attackers will ever face prosecution are relatively low.

“Welcome to the world of cybersecurity,” says Esler with a dry smile.

Next Steps

Learn more about Talos at TalosIntel.com.

Join the conversation on Twitter: @talossecurity

Learn more about Cisco security solutions here.

Inside the World’s Leading Security-Research Team

If a car is stolen, there is a thief to find. If a bank is robbed, there’s a robber to chase. But when it comes to cybercrime, law enforcement is far less often involved. Modern cybercriminals rarely leave a physical trace. Even when crimes are clearly committed, jurisdiction can be less clear. As a result, fighting cybercrime is largely done by dedicated security researchers like the more than 250 members of Cisco Talos.

“That’s a heavy burden to bear, but it’s a responsibility the team relishes,” says Nick Biasini, a Talos outreach engineer based in Austin, Texas.

Nick Biasini recently moved to Austin and joined Talos after being both a Cisco customer and a competitor.

The industry’s leading threat-intelligence organization, Talos analyzes 1.5 million malware events every day, ultimately handling 7.2 trillion threats a year. “A lot of our job is trying to find the needle in the haystack. When you have a massive dataset of information, the hard part is finding what’s interesting, unique, or new,” says Senior Director Matt Watchinski, who leads the entire Talos team from Columbia, Maryland.

Veteran security researcher Matt Watchinski leads the Talos team from Columbia, Maryland.

Talos researchers are concentrated in Columbia and Austin as well as San Jose, California, along with smaller sites around the world. The group is divided into five teams. The first four are threat intelligence, detection research, engine development, and outreach.

“Threat Intelligence powers everything by helping to consolidate and make sense of the data we receive on a continual basis,” explains Craig Williams, who leads the outreach team from Austin and acts as a spokesman of sorts for Talos.

The detection-research team then uses that data to fuel all supported security. “This team includes reverse engineers, malware analysts, and domain-reputation and spam experts. They take that distilled data and turn it into something actionable,” Williams explains.

The engine-development team does just what its name implies – works on engines that help deliver intelligence to all platforms. “These can be either APIs, backend engines that detect threats, or actual in-field detection engines that are deployed on platforms,” Williams continues.

The fifth, more elusive, team within Talos is known as Vulnerability R&D. It has a different charter altogether. These experts find what’re called zero day vulnerabilities, or previously unknown security issues that could be exploited at any time by attackers. Zero day implies that an issue has never been seen before, making it potentially more dangerous because nothing has yet been done to thwart it.

“These guys help us secure the platforms we all depend on by finding new threats before the bad guys do, and making sure our products cover the issues as quickly as possible,” says Williams. “They also develop new ways to mitigate classes of vulnerabilities to help protect customers.”

Based in Austin, Texas, Craig Williams leads a team of threat researchers and acts as the Talos spokesperson.

Talos traces its roots in large part to the storied vulnerability research team (VRT) at Sourcefire, one of the very first firms focused on network intrusion. Sourcefire, which we acquired in 2013, was founded in 2001 by Martin Roesch, who is now a Cisco vice president and the chief architect of the Security Business Group.

“Most of the original Sourcefire VRT employees are with us today,” says Talos Threat Intelligence Manager Joel Esler, an original VRT member and military veteran based in Delaware. “This is a smart, tight-knit team – dedicated beyond belief.”

When Sourcefire joined the company, Watchinski, Esler and the rest of the VRT connected with like-minded security researchers within Cisco, like Williams, to form Talos.

The team takes its name from Talos, the mythical man of bronze created by Zeus to circle the island of Crete three times daily to protect its citizens. Watchinski and other leaders wanted to create an entirely new organization that put the just-acquainted teams on equal footing. This started with the naming process, in which everyone had a say.

The various teams have embraced the new integrated culture, which grows continuously stronger, the sum seemingly greater than its parts. Today, Talos plays a leading role in global security research, and represents our growing commitment to helping customers – and the rest of the world – secure their networks.

“I have seen it first hand,” says Biasini, a recent addition to the Talos team who was previously a customer of both Sourcefire and Cisco. “There have been gigantic strides in security over the last couple of years and it’s been great to see how Cisco has evolved.”

What it Takes to be a White-Hat Hacker

When Talos Threat Intelligence Manager Joel Esler was 14 years old, he collected used radios and phones just to take them apart. “I liked to see how they ran and see if I could put them together again,” he says. “Sometimes I succeeded.”

His passion for hacking machines led him to focus on telecommunications and security in the Army for six years. The leadership skills he learned there prompted him to run for (and win) as mayor of the small town of Townsend, Delaware, where he lives with his wife and family.

Local politics and raising kids are the farthest things from Talos threat researcher Alex Chiu’s mind. A recent graduate from University of Texas and one of the youngest members of the team, Chiu loves the single life in Austin, with its burgeoning music scene. But like Esler, he is compelled to crack open boxes.

Recent University of Texas graduate Alex Chiu is one of the youngest members of Talos.

“I love puzzles. And security is not all that different,” Chiu says. “You don’t get all the pieces of the puzzle, but you still have to put them together to make a complete story, and that is a lot of fun.”

Not surprisingly, that hacker mindset that Esler and Chiu share is common among the members of Talos.

Threat Intelligence Analytics Manager Matt Olney agrees. “What it really takes is an inquisitive mind and nimble thought processes,” he says.

Tenacity is another common trait, according to Maryland-based data analyst Kate Nolan, one of a small but growing number of female security researchers within Talos. “We never forget that the bad guys are always trying to get one step ahead of us,” she says.

Nolan also believes that a diversity of backgrounds and expertise help to make the team more effective. With a background in data analytics rather than traditional security, she sees things differently.

“My colleagues are always looking for a back door, but I think more like a developer, so I think like malware builders,” Nolan says. “How would you put a threat together, rather than how do you take it apart?”

All the while, security researchers are racing the clock. It’s fast paced and relentless. “We have to work really hard just to stay even,” Nolan admits.

Despite the pressure, turnover is remarkably low at Talos, which boasts an annual personnel retention rate of more than 95 percent. For Olney, it comes down to the people he works with.

“They’re the reason I'm still here,” he says. “We’re from all parts of the world and it’s a fun place work. We work hard and we play hard and we wouldn’t have it any other way.”

Many talents and characteristics can be useful in security research, but if there’s one trait universal to everyone in Talos, it’s simply the desire to do good in a world where bad guys unfortunately exist.

“I love the fact that we are making a difference,” says Alain Zidouemba, the Talos malware research manager who also lives in Maryland. “Cisco’s security solutions monitor one-third of the world’s enterprise email traffic on a daily basis, and what we do really does affect the security of the Internet as a whole. We all share tremendous satisfaction in that.”

Alain Zidouemba leads a team that reverse engineers malware to find out exactly how it works and the best ways to protect against it.

Zidouemba came to security almost by accident. The son of diplomats from Burkina Faso, he and his family lived in France, Denmark, Senegal and Italy before coming to the United States in the late 90s, where he eventually attended university and earned an electrical engineering degree.

“After college, I got a recruiting call from a company called PestPatrol and I told them I didn’t want to exterminate bugs,” he laughs. “Then they told me it was a security firm.”

Zidouemba interviewed and the security bug bit. Today, he runs a team of 10 engineers responsible for reverse-engineering threats and developing new malware-detection software.

That knowledge that you’re making the world a better place is what makes it all worthwhile, according to Senior Director Matt Watchinski, who leads the entire Talos team from Columbia, Maryland.

“Every day we get up to do the best job we can,” Watchinski says. “No matter how many bad guys we get rid of, we know there’s always going to be another one around the corner. No matter how brilliant we think we are, there is always somebody else out there trying to outsmart us.”

Inquisitiveness. Tenacity. Diversity. And a desire to help the world. The security researchers within Talos need them all, as they know all too well that their job is never finished.

“A good day for us is when we know we’ve kept our customers protected and made it more difficult for our adversaries to conduct their business,” concludes Talos Outreach Manager Craig Williams. “But we wouldn’t have it any other way.”

The
Good Fight