Contents
Overview
Defending a Customer from a Botmaster
An Unsuspecting Customer
Stopping the Bot
Conversations with a Botmaster
You Can Find Anything on the Internet
One-Stop Botnet Shopping
Pretense to Avoid Pwning
Conclusion
References
Acknowledgments
Overview
Many teams at Cisco are dedicated to security research. One team recently investigated botnets with the goal of improving existing detection methods and discovering the techniques botmasters use to compromise machines. The team’s efforts were rewarded through their protection of an important customer’s network. Their discovery efforts also yielded extraordinary insights into the mind and motives of a botmaster. This paper discusses exploit protection and reports on the interviews the team held with the botmaster they encountered.
Defending a Customer from a Botmaster
Typically, administrators patch vulnerable machines or deploy some sort of intrusion prevention system (IPS) to protect against exploits. Both approaches are effective the majority of the time, but neither approach protects systems against the uneducated user. These approaches may not even protect people who take their machines home if the IPS is network-based. The user who will click and run anything is the greatest threat to any network.
Internet relay chat (IRC) traffic on non-standard ports is a good indicator of malicious activity. Simple botnets often use IRC as a command-and-control framework because the source code is readily available. Joining a chat network is not botnet activity, but it is usually not work-appropriate activity. Cisco offers a service that monitors and manages network-based IPS. By monitoring certain alerts from this data feed, suspicious IRC traffic was easily found.
An Unsuspecting Customer
A Cisco customer was unaware of dozens of compromised machines. A tremendous number of alerts including IRC activity, far larger than anything that could be benign, were occurring on the customer’s network. The traffic from several machines stood out from other systems on the network. There are occasionally oddities in a network, but when a small subset of machines is observed sharing the same odd behavior, researchers should take note.
Figure 1 shows a data feed from a Cisco IPS device.
Figure 1. Monitoring a Cisco IPS Data Feed
Looking at the signature alerts, it was clear that the affected machines had been compromised. The Cisco IPS detected the attack, but unfortunately the customer was not running it inline or connected to the router, so the hits were not blocked. There were several different botnets involved that looked strangely similar. When inspecting the history of the machines, in addition to the IRC traffic, exploitation and reconnaissance attempts were discovered.
None of the traffic was encrypted, indicating that the attackers were either unsophisticated or unconcerned about hiding their tracks. The botmaster occasionally took basic precautions, such as using server and channel passwords, but failed to encrypt the data. Challenge-response exchanges were hidden in normal IRC traffic.
For example, upon connecting to a server, the bot would immediately have to ping “MrB|g” with the key “s3cr3+sq|_|rr3l” or it would be denied access to the server. This challenge-response method was also used by the clients in response to certain RFC 2812 commands, such as a client-to-client protocol version request. Additionally, the bots would respond to non-RFC 2812 commands. It was noted that different botnets seemed to share many of the same commands. This led to the belief that most, if not all, of these clients were based on a common source code. Figure 2 shows the botnet.
Figure 2. Botnet
At this point in the investigation, the team’s largest concern was for the customer. There was an urgent need to determine what the botnet was doing and what information had been compromised. By using the Cisco IPS, a wide range of data about the command-and-control networks was captured. For example, the network was separated into several discrete command-and-control nodes with different IP addresses. Over the course of a few weeks, commands were captured and the network was monitored to see what information the botmaster was targeting. An open source IRC client was set up to emulate an infected machine and join the network. This allowed continuous monitoring without having to leave any compromised machines active.
The botmaster targeted employees instead of the company itself. This action likely helped the attacker to remain undetected. The botmaster’s mode of attack involved stealing employees’ passwords that were stored in Internet Explorer and then adding a redirect in the hosts file that enabled a man-in-the-middle attack against a bank in Latin America.
Stopping the Bot
Once the extent of the damage was determined, the bot needed to be stopped. The team was able to demonstrate the modification of the hosts file, making the compromise irrefutable. Some of the machines appeared to have been compromised dozens of times. A worm or trojan would compromise the machine and update the hosts file, only to have it corrected by the virus scanner (or other malware).
The correction would prevent the man-in-the-middle attack but the botnet would load normally. This action occurred each time the system booted. Some machines had dozens of entries that showed that the hosts file was corrected repeatedly. It became clear that it was not feasible to clean every infected machine, because in the time it took to alert the company to the problem, the personal information of hundreds of employees could be compromised.
The customer did not have the capability of running Cisco IPS inline, so the firewall was examined. When monitoring the botnet, it became very clear that the IRC servers would update fairly frequently. The IRC servers would move ports, servers, and change passwords, sometimes several times a day. This frequent updating made it extremely hard to block the command-and-control servers.
The team found a visible flaw in the attack: the botmaster had overlooked the update servers. The botmaster had several domain names for the update server that could be broadcast by means of IRC to update the bots before a server change-over. Close inspection revealed that the IP address of the update server always belonged to one of a small group of machines. Blocks were put in place immediately.
When the botmaster issued the next update, only a few of the systems (and none from the customer) followed. The botmaster repeatedly issued the update command to no avail. When the machines were locked down to one IRC server, a single block was sufficient to disable the network.
Had the botnet-client been more robust, it would have been necessary to block backup networks. In this case, the customer was lucky, and the single block was sufficient. The team continued monitoring the network to ensure the systems were unable to reconnect to the network, giving the customer the needed time to reimage all of the compromised machines. It was not a perfect solution, but it stopped the data leakage and prevented the systems from compromising other machines on the network.
Conversations with a Botmaster
With the customer protected, only curiosity remained. The team wondered what type of attacker would go to such complicated lengths but leave such a simple hole in their plan of attack. Did the botmaster have so many networks that it didn’t matter if one was blocked? Was the botmaster a script kiddie? For answers, one of the researchers decided to go back to a monitoring box and ask. At this stage the customer was protected and the botmaster was likely away from the keyboard. The researcher sent out an intrepid “hey” and received a response from the botmaster: “?” Thus began what turned into a months-long conversation.
The botmaster, upon realizing that one of his bots was suddenly sentient, appeared to assume that the researcher was a fellow botmaster and that their respective networks had “collided.” The researcher worked to strengthen the botmaster’s assumption. Pretending to be a fellow botmaster, the researcher asked about the server software. Figure 3 shows the initial conversation with the botmaster.
Figure 3. Starting a Conversation
After some inconsequential chat, the researcher asked if the botmaster was using his network for anything interesting. The botmaster readily revealed his master plan: to compromise a few thousand machines and then sell them off in big batches. With careful questions, the researcher learned from the botmaster that the market rate was about US$0.10-$0.25 per machine and that the botmaster had recently sold 10,000 machines for US$800. In attempts to bond with the botmaster, the researcher discussed popular exploits, sharing stories of “pwning,” or gaining control of, dozens of machines at a time.
With a solid background in IPS, the researcher was aware of current trends in vulnerability research but asked in what area the botmaster focused his efforts. The expected answer was a Microsoft vulnerability that worms such as Conficker exploit. The botmaster’s answer, however, was that he mostly focused on instant messaging software. No vulnerability was required to grow his network. Instead, he could spam 10,000 people with a simple “check out this cool software” message and rely on at least a one percent response from the recipients. As an approach, it made sense, because the same process continues to work for spammers as it has worked for years, despite efforts at user education.
After revealing his methodology, the botmaster appeared to suddenly realize that perhaps he had shared too much information with an unknown person. He quizzed the researcher on “old school” (that is, previously well-known) hackers. The researcher responded that he did not know any of the old-school attackers. (When the researcher later did online searches on the old-school attackers, most of them had been apprehended by the Federal Bureau of Investigations.) By saying he did not know any of the attackers that the botmaster named, the researcher established credibility with the botmaster that he was not a law enforcement agent. After this exchange, the botmaster gave the researcher his contact information through Microsoft Network (MSN) so the two could communicate more easily. Figure 4 shows a trust-building session between the researcher and the botmaster.
Figure 4. Building Trust
You Can Find Everything on the Internet
As any good hacker would, the researcher immediately keyed the botmaster’s screen name into Google and found a few posts. The posts led the researcher to additional handles, or usernames, which then led to hundreds of posts by the same author. Under a different handle, the botmaster was the author of an enormous amount of IRC-based botnet software. He was also very well known in the black hat community, where attackers and hackers share information.
Intrigued, the researcher decided to accept the botmaster’s invitation to stay in touch. The researcher created an MSN account and, over the course of several days, multiple MSN conversations were held between the researcher and the botmaster. Topics focused primarily on secrets of the botnet trade and discussions of various software packages. The botmaster confirmed the researcher’s theory that many of the IRC-based botnets stemmed from a single source; however, he declined to provide a copy of the modified version he had adapted. Instead, the botmaster directed the researcher to an online forum that was contained a profusion of information regarding botnet activity.
Figures 5 and 6 shows the conversations of the botmaster directing the researcher to the forum.
Figure 5. Directions to a Forum—Part 1
Figure 6. Directions to a Forum—Part 2
One-Stop Botnet Shopping
The forum hosted discussions on all the information that anyone would need to form a botnet, including several detailed how-to guides. The researcher was able to acquire source code for the bot and the server from the forum. The server code was based upon a modified Unreal IRC server. The client code, which would have no legitimate use, was a valuable source for IPS signatures. While it would be possible that all of the command functionality could be rewritten, if botmasters were capable of doing that, they likely wouldn’t use the publicly available source code.
Entire sections of the forum were dedicated to the buying and selling of botnet paraphernalia, such as RapidShare file hosting accounts, packers, password lists, bot software, and password stealers. The bot software is advertised much like any other software, listing various features such as “four methods of command and control,” “undetected by virus scanners,” “anti-x (sandbox, debugger, etc.),” “process monitoring,” and so forth. Several bot software authors have followed the Microsoft practice of offering multiple versions of software at tiered pricing levels.
The “For Sale” sections were governed by a very specific set of rules, including a rule that stated that botnet software could not be sold in the forum, likely due to the laws of the country in which the server hosting the forum resides.
The software for creating botnets, including directions and tutorials, was widely available for download or purchase. It was forbidden to sell the software on the forum if the software was publicly available, a rule that seemed to be an attempt to deter botmasters from taking advantage of one another. For concerned bot shoppers, all software was verified by a trusted moderator so that the buyer could trust that they would be receiving the software for which they were paying.
Pretense to Avoid Pwning
The customer who had originally been infected had been clean for months when the researcher decided to seek out the botmaster again to learn more about the botnet community. The researcher was concerned about not revealing any specifics about the customer or himself but needed to establish a level of trust with the botmaster. With these considerations in mind, the researcher decided it was more likely that the botmaster would speak with a journalist than an IPS specialist pretending to be a fellow botmaster.
After re-establishing contact with the botmaster, the researcher promptly “confessed” to being a reporter researching an article on botnets. The researcher explained that he had contacted the botmaster again because he was seeking the most accurate data possible for his article. The researcher expected his virtual disguise would pass the botmaster’s scrutiny because of the widespread disdain by certain groups in the security community of the quality of security reporting.
Within the security community, there is a perception that only a few reporters actually understand the security topics that they cover. Some part of nearly every story is wrong or greatly exaggerated. For example, a recent article in PC World [1] stated that the widely publicized attack on the creator of Metasploit was performed by an unknown attacker who had weaponized Dan Kaminsky’s discovery of a fundamental flaw in DNS [2]. The article contained so many errors that, in addition to a printed retraction, additional reporting was required to correct the published description of the issue [3]. H D Moore, Metasploit’s creator, claimed that the statements attributed to him were completely fabricated [4].
The botmaster had strong opinions on security reporting. He mentioned conflickr in particular as example of faulty reporting that resulted in exaggerated numbers of systems affected. One antivirus software company [5] had based its publicly reported numbers of affected machine on a variable in the URI sent from the compromised machines. The variable, known as Q, reported the number of machines that had been successfully attacked to the botnet’s command and control system. This method of calculating the number of compromised machines was easily manipulated by Dynamic Host Configuration Protocol (DHCP) and other means. A user on the company’s public blog even posted a comment pointing out the possibility of easy manipulation, but the company dismissed the comment and did not correct its reported numbers.
The actual count of affected systems would not be realized until weeks later in a subsequent report by a separate research company [6] that explicitly pointed out that “Q reports the number of machines that each victim claims to have infected. Q may be artificially inflated by reinfections and DHCP effects” (The Q variable is the value in the URI that was thought to report the number of compromised machines.) According to the latest report [7], the numbers reported using the initial method was was off by a multiple of 50.
Figure 7 shows a continuation of the conversation between the researcher and the botmaster.
Figure 7. A Little Anonymous Fame
The researcher assured the botmaster that even if he chose not to be interviewed or answer questions, he could have a little "anonymous fame." Surprisingly, the botmaster agreed to participate. Recognizing the unusual opportunity, the researcher suggested a TOR audio conference [8] using a method known as onion routing that would be untraceable. The botmaster reported he did not have a microphone available, but more likely, he feared the researcher would try to trace him through the tunneling connections. His reluctance to audio conference may also have been based on his lack of knowledge of the onion routing protocol. The researcher’s first question was why it would be preferable to sell bots instead of turning them into spam or phishing networks. The botmaster’s answer was that selling bots was a rarity; normally, bots would be used as a network for phishing attacks. When the researcher asked how much money could actually be made from phishing activities, the botmaster was evasive about his most lucrative bot activities, but said “a guy he knew” was able to earn US$5000 to US$10,000 a week solely through phishing activities.
Figures 8 and 9 show conversations about selling bots.
Figure 8. Phish or Sell
Figure 9. Lucrative or Unprofitable?
The researcher offered the botmaster a lighter question, asking what was the strangest thing the botmaster had found on a compromised machine. The botmaster replied he had found inappropriate pictures of a minor and had promptly reported the issue to the authorities.
The researcher asked the botnet owner about his proudest moment as a botmaster. The answer involved the Windows Distributed Component Object Model (DCOM) attack (MS03-026, IAM 11104), which exploited a bug in the DCOM Remote Procedure Call (RPC) interface. The vulnerability existed in all modern versions of Windows at the time and was remarkably easy to exploit. The botmaster said that when he ran the attack, his server was flooded with joins, with each join representing the compromised machine of an unsuspecting user.
The researcher and the botmaster continued to chat, discussing protective measures. New security features in Windows Vista, such as kernel patch protection, prevented his bot from running in Ring0 [9]. Ring0 is a term from system management mode. Briefly, this protection system involves three rings. Ring0 is the most privileged ring, where the kernel resides, and Ring3 is the least privileged ring, where a user’s programs execute. Given this limitation, the botmaster’s bot would not function on the Vista OS.
Figure 10 shows the conversation about Ring0
Figure 10. Protective Measures
Over the course of the conversations, the botmaster revealed that he could never trust anyone 100 percent of the time and that it was necessary for him to be on guard constantly and follow good computing practices. Other botmasters would act on any opportunity to take over his networks, and according to Google-cache hits, they had tried in the past. However, lack of trust is common among botnet owners, and with good reason. In one instance, the botmaster had used a hijacked account to impersonate a law enforcement official and force another botmaster to abandon a 6,000-node network. The botmaster had to remain alert at all times, his firewall blocking nearly all inbound connections, and surfing the Internet via proxy chains [10] to remain anonymous.
The botmaster recommended a list of best and worst sites that are based around forums. Forums may facilitate the code re-use that is often associated with botnet clients. The botnet community is similar to the open source community, in which more experienced users in the forum help or humiliate new botmasters. According to the botmaster, only 20 percent really understand the code offered through the forums; the rest simply run the code and do their best to follow the help files. He estimated another three to five percent of botmasters write unique code.
Figure 11 shows a conversation in which the botmaster estimates the percentage of unique coders.
Figure 11. Who Writes Code?
Conclusion
Many people, researchers included, wonder why attackers do not pursue legitimate IT or programming jobs. According to the botmaster, the barriers to legitimate work are a criminal record and a lack of professional education; frequently, both factors prevent attackers from gaining regular employment.
Figure 12 shows the conversation regarding barriers to legitimate occupations.
Figure 12. Why Not Get a Real Job?
The researcher asked how attackers experience security companies and services, and whether the botmaster felt pressured by the security companies. His response was that “a few companies catch on very quickly but, for the most part, it is business as usual.”
As the botmaster stated, running a botnet was his business. The criminality of running a botnet was simply a by-product of his primary means of employment. The botmaster's product is a management interface to a multinode network that can be sold to other customers for a profit. Perceiving himself as a small business owner, the botmaster is not concerned with impacting the functionality of a user's personal computer or with the possibilities of identity theft or data leakage, but instead with generating income.
Anyone with basic computer experience is able to run a botnet. It is not necessary to understand the code, nor is there a need to understand networking. Both traditional and new media organizations frequently report on the need to patch against the latest threat that exploits a recent vulnerability. Readers rarely hear, however, about the kid who lives in their neighborhood who runs a 10,000-node botnet based off of MSN instant message spam. All bots are not created with equal proficiency. Botmasters are implementing cutting-edge evasion techniques to avoid detection and prevent reverse engineering. It is imperative to keep attackers of both types in mind, professionals and script kiddies, when designing a network’s defenses. To effectively combat the bot economy, the cost of doing business must be raised by educating users and following security best practices. Attackers pursue easy money. Maximum gain with minimal effort is the prime motivator for a botmaster. If the time required to compromise machines increases, attackers will move on to easier targets.
Patching is important, but user education is key. A corporation can deploy the latest security measures but remain vulnerable to data theft, hosting spam servers, or worse. Business users must be educated to comply with safe behavior. If policies are not in place to limit the infiltration on non business communications or if users do not understand the importance of leaving random files unopened, there is little point in administrators patching machines.
Using Cisco IPS alerts, the research team was able to successfully identify and disable a botnet. The IRC traffic on non-standard ports was a clear sign of compromised systems. Without Cisco IPS, the customer would have been blind to the botnet activity and its employees could have had their stored password compromised leading to bank fraud or identify theft. If the customer had been able to run the IPS inline, the compromise would not have occurred. An intrusion detection system also has its place in a network; without the history of previous alerts from a management tool, the remediation for the customer’s system would not have been possible.
References
[1] McMillan, Robert. "DNS Attack Writer a Victim of His Own Creation." PC World, July 29, 2008. (http://www.pcworld.com/article/149125/dns_attack_writer_a_victim_of_his_own_creation.html)
[2] Invisible Denizen. "Kaminsky's DNS Issue Accidentally Leaked?" July 21 2008. (http://blog.invisibledenizen.org/2008/07/kaminskys-dns-issue-accidentally-leaked.html)
[3] IDG News Service staff. "DNS Attack Writer a Victim of His Own Creation." PC World, July 30, 2008. (http://www.pcworld.com/businesscenter/article/149136/dns_attack_writer_a_victim_of_his_own_creation.html)
[4] Moore, H D. "DNS Attacks in the Wild." Metasploit, July 29, 2008. (https://blog.rapid7.com/2008/07/30/dns-attacks-in-the-wild/)
[5] F-Secure. "Calculating the Size of the Downadup Outbreak." Weblog: News from the Lab. January 16, 2009. (http://www.f-secure.com/weblog/archives/00001584.html)
[6] Phillip Porras, Hassen Saidi, and Vinod Yegneswaran. "An Analysis of Conficker's Logic and Rendezvous Points." SRI International, February 4, 2009. (http://www.csl.sri.com/users/vinod/papers/Conficker/)
[7] Phillip Porras, Hassen Saidi, and Vinod Yegneswaran. "An Analysis of Conficker's Logic and Rendezvous Points.: SRI International, February 4, 2009. (http://mtc.sri.com/Conficker/#appendix-1)
[8] The Tor Project (http://www.torproject.org/)
[9] Federico Biancuzzi. "The Quest for ring 0." Security Focus, May 10, 2006. (http://www.securityfocus.com/columnists/402)
[10] "Proxy Chains" (http://proxychains.sourceforge.net/)
Acknowledgments
Cisco Security Intelligence Operations
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.