The Rise and Fall of Reactor Mailer
| Author: | Henry Stern, Cisco IronPort Systems LLC <henry.stern@cisco.com> |
|---|---|
| Revision: | #9 |
| Date: | 2009/03/25 |
Contents
Introduction
June 2007 saw the beginning of a long-term increase in spam volume, as illustrated in figure 1. This coincided with the discovery of the "Srizbi" trojan [3], the worker component of the third version of the Reactor Mailer Mass Mailing Cluster System. In this instance, a cluster mailing system is one where the user interacts with a single user interface and mailing tasks are distributed among several systems. Reactor Mailer is a template-driven spam system sold under the "Software as a Service" model by Elphisoft. [1] While previous versions sent messages using SOCKS [5] proxies hosted on machines infected with malware such as Sobig.a, [19] Reactor Mailer 3 sent messages using a distributed computing model with special-purpose malware. [18]
Figure 1: SenderBase spam volume (billions/day) for 2007-2008.
At its peak, IronPort estimates that Reactor Mailer was responsible for 60% of global spam, as many as 80 billion messages per day sent by no less than 260k hosts infected with the Srizbi trojan.
In this paper, we will argue that the increase in spam since June 2007 is not merely correlated with Reactor Mailer 3's inception, but was caused by it. We begin with a brief discussion of Reactor Mailer's architecture and why it enabled Reactor Mailer to send spam at such a high rate as compared to its peers. This is followed by lower-bound measurements of the size of the Srizbi botnet during much of 2008. We continue with a discussion of the hosting provider, McColo, that enabled Reactor Mailer to be so prolific. We conclude by relating the shutdown of McColo and the February 2009 Microsoft Malicious Software Removal Tool to the decline of spam and Reactor Mailer.
The Architecture of Reactor Mailer
According to Elphisoft's marketing material [1] and IronPort's own research, Reactor Mailer has been a cluster mailing system for at least two major revisions. Version 3 of Reactor Mailer moved the worker process from Elphisoft's data center to hosts infected with the Srizbi trojan.
Figure 2: Life cycle of a Reactor Mailer spam campaign.
The Reactor Mailer system architecture is illustrated in figure 2. When a Reactor Mailer user wishes to send a spam campaign, they create a "task" in the user interface and then instruct the system to run the task. The Reactor Server then separates the task into indiviudal work units called atoms. Each atom contains message templates, data files and email lists. [18] Workers request atoms from the Reactor Server, process the work unit by sending the emails and then report the completion of the work unit to the Reactor Server. The Reactor Server aggregates the results from the workers and reports the status of the task through the user interface.
Moving the worker processes from the data center to infected hosts eliminated Reactor Mailer's main bottleneck: communication with and maintenance of large numbers of SOCKS proxies. Previously, all of the spam was generated in the data center and transmitted byte-for-byte through the data center's network connection. By running the worker process on infected hosts exponentially increases the amount of available processor power and network bandwidth. Combined with a sufficiently large pool of infected hosts, this allowed Reactor Mailer to transmit spam at an unprecedented rate.
The Reactor Mailer Worker, Srizbi
The Srizbi trojan is a very stealthy piece of malware. It is implemented as a kernel-level driver, attaching itself to the NTFS system to hide its files from the Windows kernel and the user. It also attaches NDIS and TCP/IP drivers to evade firewall and sniffer tools. [2] Researchers at IronPort have also observed that the malware stops sending spam whenever the user touches the mouse or keyboard, presumably to prevent observable network latency.
At its inception in mid-2007, Srizbi was being distributed using drive-by downloads by exploited sites linking to the MPack toolkit. [4] In February 2008, as the US Presidential primaries were heating up, Srizbi was distributed by email using web links to executables posing as videos of candidates such as Hillary Clinton. [17] Email lures continued through mid-2008, with a highly successful campaign in June purporting to contain a video where the user looks "stupid." [15] Spikes in spam volume and botnet membership and corresponding to the February and June 2008 campaigns can be observed in figure 1 and figure 3. Figure 4 focuses on the social engineering campaign in June.
Tracking Srizbi
Researchers at IronPort have engaged in efforts to track the number of hosts infected with Srizbi and its contribution to spam from late 2007 until the present day. Srizbi's TCP/IP driver is unique in its implementation of RFCs 791 [12] and 793 [13]. TCP SYN packets emitted by hosts infected with Srizbi can be trivially identified using passive fingerprinting. [16]
The p0f passive OS fingerprinting tool [23] uses six features to generate a fingerprint of a TCP packet.
- Initial TCP window size.
- Initial time to live (TTL).
- Whether or not the "Don't fragment" (DF) IP option is set.
- Total length of the IP and TCP headers.
- Number and order of TCP options.
- Quirks in the TCP/IP implementation.
Srizbi's TCP/IP signatures stand out from those of every other implementation in p0f's fingerprint database and IronPort's investigations suggest that they are unique. While not a discriminator on its own, the DF bit being set to 0 is very rare among modern TCP/IP stack implementations. The earlier version of Srizbi set the maximum segment size to the lowest allowed by RFC 879 [14]. Both versions of Srizbi use a constant initial window size.
| Field | Srizbi Version 1 | Srizbi Version 2 |
|---|---|---|
| Window Size | 24000 | 6144 |
| TTL | 128 | 255 |
| DF | 0 | 0 |
| Length | 44 | 44 |
| Options | <mss 536> OR <mss 528> | <mss 1024> |
The maximum segment size (MSS) varies when the user connects to the internet with PPP over Ethernet (PPPoE) and uses a broadband router that reduces the maximum segment size by 8. This is done because the maximum transmission unit (MTU) for Ethernet is 1500 and the MTU for PPPoE is 1492, although with such a small MSS, it is unnecesary to do so. The MSS of 536 corresponds to an MTU of 576, the default IP maximum datagram size in RFC 879.
Srizbi's SYN packets can be identified using the following p0f signatures:
24000:128:0:44:M536:.:Srizbi:V1 Ethernet 24000:128:0:44:M528:.:Srizbi:V1 ADSL 6144:255:0:44:M1024:.:Srizbi:V2
Throughout much of 2008, IronPort captured a sample of incoming TCP SYN packets to the SpamCop spam trap system from which those packets originating from Srizbi-infected hosts were extracted. The number of unique infected IP addresses observed per 24-hour period is plotted in figure 3. This figure is a lower bound because it only includes the IP addresses that were seen by IronPort's sensor. This figure should be corrected by the probability that a given Srizbi host would send email to SpamCop within a 24-hour period and by the sampling probability. The former is unknown at this time but could be derived using statistics about the contents of distribution lists in use and actual rate of spam transmission by Reactor Mailer, should they ever become available.
Figure 3: Lower-bound estimate of the number of hosts infected with Srizbi throughout 2008.
IronPort observed an average of 140k infected hosts per day in 2008 with a maximum of nearly 260k infected hosts. SecureWorks estimated that there were 315k infected hosts in April 2008. [20]
On several occasions throughout 2008, IronPort measured that 60% of all TCP SYN packets sent to SpamCop originated from Srizbi-infected hosts. This agrees with published estimates from Marshal's TRACE. [10]
Figure 4: The number of hosts infected with Srizbi grows dramatically during a successful social engineering campaign. [15]
Reactor Mailer at McColo
The McColo data centre facility was located in Market Post Tower, 55 S Market Street in San Jose, CA. McColo was notorious as a haven for botnet command and control servers and online pharmacy payment processors, among other unsavoury activities. [7] McColo purchased network connectivity from Global Crossing and Hurricane Electric.
McColo announced only four netblocks from AS26780, every one of which was implicated in some sort of illicit activities:
208.66.192.0/22 208.66.195.0/22 208.72.168.0/21 208.72.173.0/24
Reactor Mailer appeared to be McColo's largest single customer, occupying at least 27 addresses on the 208.72.168.0/23 netblock. Reactor Mailer leased at least 25 unique servers running Linux 2.4.20 on Itanium. Each of these servers had its own pool of workers and was, presumably, leased to different sets of customers.
Reactor Mailer's Downfall
The Shutdown of McColo and Move to Estonia
Despite an abundance of infrastructure and, presumably, capital, Elphisoft made a critical error when planning their network for Reactor Mailer. All of their command and control infrastructure was in the same colocation facility, on the same netblock. On November 11, 2008, McColo was disconnected from the public internet by all of its upstream providers. [6] IronPort observed that spam volumes immediately dropped by two thirds, shown in figure 5. [8]
Figure 5: SpamCop volumes dropping off by two thirds after the McColo shutdown.
It took Reactor Mailer two weeks to relocate to Estonia [11] and resurrect their servers using a domain name generation algorithm. [22] Unsubstantiated rumours suggested that Elphisoft was able to recover their servers from the McColo data centre. Reactor Mailer's recovery can be seen in figure 6.
Figure 6: The percentage of spam attributable to Srizbi around the time of the McColo shutdown.
By early February, 2009, Spam volumes were back to their pre-McColo shutdown levels and Reactor Mailer was once again accounting for 60% of spam.
The Killing Blow
February 2009's "Patch Tuesday" signalled what appears to be the end of Reactor Mailer. [21] Microsoft added a signature for Srizbi to the Malicious Software Removal Tool, cleaning up nearly all infections within a matter of weeks, as evidenced by figure 7. No attempts seem to have been made by Elphisoft to recover the remaining Srizbi infections. [9]
Figure 7: The percentage of spam attributable to Srizbi around the time of the February 2009 MSRT release.
With Srizbi nearly eliminated and Reactor Mailer all but shut down, spam volumes quickly returned to the levels of June 2007, as shown in figure 8.
Figure 8: SenderBase spam volumes dropping to early 2007 levels following the February 2009 MSRT release.
Conclusions
We have demonstrated how Reactor Mailer's modern architecture enabled it to transmit spam messages at an unprecedented rate. We have connected several significant events in spam from June 2007 through the present to events involving Reactor Mailer. Spam volumes increased dramaticaly when Reactor Mailer 3 and Srizbi were first deployed and dropped to the original volumes both times that Reactor Mailer was disabled. Whenever the Srizbi botnet was expanded using a successful social engineering campaign, global spam volumes increased. We can therefore conclude that Reactor Mailer was responsible for a 19-month global increase in spam volume between June 2007 and February 2009.
The McColo shutdown and MSRT update were both effective methods of crippling the Reactor Mailer system's spam transmission capacity and had noticeable effects on global spam volumes. While it is still too early to draw any conclusions about the long-term effectiveness of the MSRT update, the effects of shutting down the command and control infrastructure of the botnet were only short-lived.
While it seems easier to affect change by targeting a botnet's command and control server while leaving the rest of the botnet intact, it is demonstrably not an effective long-term method of stopping spam and mitigating botnets. Further study of the long-term success of cleaning up infected hosts at mitigating botnets is required. Until a practical, permanent solution to this problem can be devised, both tactics should continue to be employed in concert with other botnet and spam mitigation techniques presently in use.
References
| [1] | (1, 2) Anri Erinin. Mass Mailing Cluster System Reactor Mailer. Web Server Talk, Email abuse and Spam Forum. August 7, 2005. http://www.webservertalk.com/archive154-2005-8-1158732.html |
| [2] | Kaoru Hayashi. Spam from the Kernel: Full-Kernel Malware Installed by MPack. Symantec Security Response Weblog, June 29, 2007. https://forums2.symantec.com/t5/blogs/blogarticlepage/blog-id/malicious_code/article-id/139 |
| [3] | Kaoru Hayashi. Trojan.Srizbi. Symantec Security Response, June 20, 2007. http://www.symantec.com/business/security_response/writeup.jsp?docid=2007-062007-0946-99&tabid=1 |
| [4] | Gregg Keizer. Mpack installs ultra-invisible Trojan. ComputerWorld Security. July 5, 2007. http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9026323 |
| [5] | David Koblas and Michelle R. Koblas. SOCKS. 3rd USENIX Security Symposium, pages 77-83. September 1992. Baltimore, USA. http://www.usenix.org/events/sec92/full_papers/koblas.pdf |
| [6] | Brian Krebs. Major Source of Online Scams and Spams Knocked Offline. Security Fix. November 11, 2008. http://voices.washingtonpost.com/securityfix/2008/11/major_source_of_online_scams_a.html |
| [7] | Brian Krebs. A Closer Look at McColo. Security Fix. November 13, 2008. http://voices.washingtonpost.com/securityfix/2008/11/the_badness_that_was_mccolo.html |
| [8] | Brian Krebs. Spam Volumes Drop by Two-Thirds After Firm Goes Offline. Security Fix. November 12, 2008. http://voices.washingtonpost.com/securityfix/2008/11/spam_volumes_drop_by_23_after.html |
| [9] | Alex Lanstein. Bad Actors Part 1 - Starline Web Services. FireEye Malware Intelligence Lab. February 11, 2009. http://blog.fireeye.com/research/2009/02/bad-actors-part-1-compic.html |
| [10] | Srizbi now leads the spam pack. Marshal TRACE Blog. February 29, 2008. http://www.marshal.com/trace/traceitem.asp?article=567 |
| [11] | Atif Mushtaq and Alex Lanstein. Srizbi control regained by original owner. FireEye Malware Intelligence Lab. November 25, 2008. http://blog.fireeye.com/research/2008/11/its-srizbi-trun-now.html |
| [12] | Jon Postel. Internet Protocol. RFC 791. September 1981. http://www.ietf.org/rfc/rfc0791.txt |
| [13] | Jon Postel. Transmission Control Protocol. RFC 793. September 1981. http://www.ietf.org/rfc/rfc0793.txt |
| [14] | Jon Postel. The TCP Maximum Segment Size and Related Topics. RFC 879. November 1983. http://tools.ietf.org/rfc/rfc879.txt |
| [15] | (1, 2) Negar Salek. One of the biggest threats to Internet users today: Srizbi. SC Magazine Australia/NZ. June 25, 2008. http://www.securecomputing.net.au/News/115170,one-of-the-biggest-threats-to-internet-users-today-srizbi.aspx |
| [16] | Lance Spitzner. Passive Fingerprinting. SecurityFocus. May 3, 2000. http://www.securityfocus.com/infocus/1224 |
| [17] | SpywareRemove. Trojan.Srizbi Masquerades as Hillary Clinton Video February 15, 2008. http://spywareremove.com/security/trojansrizbi-masquerades-as-hillary-clinton-video/ |
| [18] | (1, 2) Henry Stern. A Survey of Modern Spam Tools. Fifth Conference on Email and Anti-Spam (CEAS 2008). August 21-22, 2008. Mountain View, USA. http://www.ceas.cc/2008/papers/ceas2008-paper-35.pdf |
| [19] | Joe Stewart. Sobig.a and the Spam You Recevied Today. SecureWorks. April 21, 2003. http://www.secureworks.com/research/threats/sobig/ |
| [20] | Joe Stewart. Top Spam Botnets Exposed. SecureWorks. April 8, 2008. http://www.secureworks.com/research/threats/topbotnets/ |
| [21] | Vincent Tiu. MSRT February 2009 - Win32/Srizbi. Microsoft Malware Protection Center. February 10, 2009. http://blogs.technet.com/mmpc/archive/2009/02/10/msrt-february-2009-win32-srizbi.aspx |
| [22] | Julia Wolf. Technical details of Srizbi's domain generation algorithm. FireEye Malware Intelligence Lab. November 25, 2008. http://blog.fireeye.com/research/2008/11/technical-details-of-srizbis-domain-generation-algorithm.html |
| [23] | Michal Zalewski. p0f http://lcamtuf.coredump.cx/p0f.shtml |