It was the end of another 12-hour day filled with hostility, deception and confusion, and an exhausted Suresh Ramasubramanian, a systems administrator at a Hong Kong ISP, was finally getting ready to head home.
On his way out the door he happened to take one last look at the network status and noticed that a mail bomb -- a flood of incoming spam messages -- had just begun.
Ramasubramanian realized he probably wouldn't be getting any sleep that night.
He spent the next eight hours struggling to block the spam attack and contain the damage. The huge volumes of mail the spammer was sending -- several hundred thousand messages at a time from different Internet protocol (IP) addresses at the rate of 20,000 every 10 minutes -- was clogging his servers and seriously slowing down mail service to his legitimate users.
Stopping a spam surge usually isn't rocket science; skilled workers can trace and trap a spam flood within a few minutes by determining what IP address the spam is coming from and then blocking access to the spammed servers from that IP address.
Unfortunately, expert spammers can also switch IP addresses as quickly as the blocks are applied. Ramasubramanian wasn't surprised to see that each time he located the IP address the spammer was spewing from and blocked it, the spammer quickly jumped to another IP address.
"It's intense, because you're trying to stay right ahead of them and at the same time you're having to clean up the unholy mess they are making on your servers," says Ramasubramanian. "This kind of job tends to age a guy very quickly."
Eventually the spammer gave up -- at least for that night. But there would be plenty more to replace him the next day. And the next. Working the spam-abuse desk is an endless job, filled with hostility coming from all sides. Spammers complain that their rights are being infringed upon. Spam recipients howl with anger at the daily flood. Even other spam fighters fuel the frenzy, mistakenly considering people like Ramasubramanian enemies instead of friends. And no matter what you do, still the spam comes.
Despite many people's best efforts, spam is on the rise. According to statistics from Brightmail, an anti-spam software vendor, about 40 percent of all e-mail traffic in the United States is spam, up from 8 percent in late 2001. Most spam experts assume that within the next year at least half of all e-mail will be spam.
Imagine what your in box might look like without the efforts of Ramasubramanian and his cohorts.
Ramasubramanian has spent the last five years smacking spam, first as the founder and president of the Indian chapter of the Coalition Against Unsolicited Commercial E-Mail (CAUCE) and then as an abuse-desk worker at various ISPs. He is currently the abuse-desk administrator at Outblaze, a Hong Kong provider of outsourced e-mail services. Outblaze is one of the largest such services in the world, with over 30 million users.
Most of his workdays are spent battling a predictable blend of both spammers and, sometimes, angry anti-spam advocates. But occasionally the spam really hits the servers and he and his team are faced with "a full-blown crisis situation straight out of M*A*S*H."
In recent weeks he's been battling one very persistent spammer who sends millions of spams every day with forged headers and return e-mail addresses that make it appear as if the spam is coming from Outblaze's servers.
"So the bounces come straight to our servers as there's no where else for them to go, given the way he's forged these headers. Millions of spam bounces a day."
And that's just from one targeted attack. Every day, 80 percent of all incoming mail to Outblaze is rejected as spam and filtered out before Ramasubramanian and his team have to deal with it. Out of the remaining 15 million messages per day that do pass through Outblaze servers, about 15 percent is spam that managed to sneak through the filters.
"My job is like trying to keep cockroaches and rats out of a warehouse. Only, in my case, the warehouse is huge and surrounded by swamps full of the damned pests," Ramasubramanian says. "The spam doesn't ever stop coming, and we just have to grit our teeth and hang on, blocking as much as we can."
The spam that comes into Ramasubramanian's servers originates from no particular locale and is more or less evenly divided between "dedicated spam factories, some run by some pretty technically smart people" that spam for themselves and others on a contract basis, and "newbie" spammers.
The newbies tend to be smalltime spammers who buy a CD full of e-mail addresses which they are assured are "guaranteed 100 percent opt-in targeted biz leads!!!" These folks are easy to catch as they usually spam directly from their personal e-mail accounts.
But Ramasubramanian says he's been watching a troubling new development, people whose computers have been hijacked by computer viruses or other sneaky software programs and then transformed into spam-generating factories.
"Sometimes the spam is highly objectionable, ads for things like bestiality, child porn and cracked software," says Ramasubramanian. "And quite frequently the people with these infected computers are unaware that they are generating spam, and are horrified when someone from their ISP contacts them about the stuff being sent from their computers."
Figuring out new and better ways to stop spam sent from both the clever and the clueless is a big part of Ramasubramanian's job.
Ramasubramanian currently uses eight blacklists from different anti-spam groups, and also independently blocks chronic spam sources that have troubled his network before. Outblaze and many ISPs also scan all the mail servers that connect to their service, checking to see if they are running "open relays" that spammers can use to pass e-mail through, thereby hiding their own identity and the real source of the spam. In the United States, it is increasingly standard practice for ISPs to block all mail coming from mail servers that are configured for open relaying.
Ramasubramanian knows these broad blocks often seem unfair to the legitimate users of the mail server, who see their perfectly valid mail blocked because of a single spammer. But he said it is a sad necessity when you consider that a single spammer can pump as many messages through a server in an hour as all the other users will send in a month.
No matter what he does, he can't please everyone. According to Tiffiany Mork, senior abuse engineer at Allegiance Internet, a very thick skin is a requirement for an abuse-desk worker. Her typical day includes verbal harassment, screaming, threats, and "all manner of nasty things."
Ramasubramanian's business card includes the odd titles: "Email Sturmbahnfuehrer" (sic) and "Lower Middle Class Sysadmin." The names were bestowed on him by spammers.
Sturmbahnfuehrer came from a Usenet post by a spammer whom Ramasubramanian had blocked. The spammer protested that systems administrators were stopping him from sending out his "legitimate business offers" to the Internet at large, and specifically raged at Ramasubramanian, calling him the evil "E-mail Sturmbahnfuehrer."
"That spammer also claimed that he'd reported me to the INS for stealing office supplies. Nice, trying to deport me from India to India. I never figured out quite what that was all about."
The other title came from a spammer who asked Ramasubramanian what she'd done that made him report her to her ISP.
"I gave her a standard set of links and information on why spam is bad, and took the time to explain all this to her. She then asked me what I did for a living. When I replied that I was a Unix administrator at an ISP, she blew up and said, 'I thought you were a successful businessman and marketer, but you are only a lower-middle-class Unix sysadmin. Don't you dare talk to me like this!!!'"
The abuse that abuse-desk workers are subjected to doesn't just come from spammers. Mork says a lot of aggravation comes from other spam fighters.
"It always bothers me when I'm being yelled at by people that I consider to be on the same side," Mork says. "I do understand people get frustrated. They think we're not taking action if they don't hear back from us in response to their complaints. But sometimes understaffed abuse desks have to choose between dealing with the spammer or dealing with spam fighters. We always opt to go after the spammers."
Mork also noted that the triage atmosphere of abuse desks often requires workers to rank spam in the order of the disturbance it's causing and deal with it accordingly.
"The rare cases of kiddie porn spam always get priority; we work with the FBI on those. After that, we look at who is the most active, who is causing the most damage today," Mork explained.
"So while stopping a person who is spamming, say, offensively graphic animal sex material will usually strike us as an urgent task, we will go after the guy who is flooding us with thousands of messages before we go after a small-time mailing by Mr. Barnyard Sex. But meanwhile the people who are receiving e-mails with images of horses in compromising positions are screaming at us. It can get difficult."
Ramasubramanian says he's troubled by "radical fringe" anti-spammers who firmly believe that the only way to get an ISP from mainland China to do anything at all about spam coming from their network is "to e-mail the spammer, assorted other addresses at his/her ISP, and several Chinese government e-mail accounts, and including in the message words like "Falun Gong" and "Free Tibet."
"Since the government of China supposedly filters and monitors every single e-mail sent into mainland China, the general idea of this stupid trick is to scare the admins into taking action, or get them into serious trouble with their government, all because they are unwise enough to allow spam onto their network."
Ramasubramanian's situation as the head spam fighter at an ISP based in Asia is particularly complicated and sometimes puts him squarely in the middle of many anti-spam efforts.
"Some of the most aggressive spammers around now have servers in China, India and several other countries around the world, hosted at ISPs where the management is apparently happy to do nothing as long as the dollars keep flowing in," Ramasubramanian says. "So some systems administrators have responded by blocking much or all of the traffic from Asian ISPs."
Outblaze has servers around the world, so when Ramasubramanian sees blocks that take out a wide swath of Asia he can route his users' legitimate mail through servers elsewhere. And since he's well-known in the abuse-desk world, most large ISPs try not to block Outblaze e-mail. But Ramasubramanian still spends a significant amount of time every day contacting systems administrators of smaller networks who have blocked Outblaze's servers.
"They'll see a stray item of spam from our network, learn from a Whois lookup that we are based in Hong Kong and then figure it's OK to block over 30 million users," Ramasubramanian sighs.
Laura Atkins, president of the SpamCon Foundation, an anti-spam organization, has been fighting spam since 1996. She agrees that some spam fighters sometimes don't know how to work effectively with abuse-desk staffers.
"Abuse desks are understaffed and people are overworked. But spam is such an emotional issue; it's hard not to get frustrated when you feel like you are under siege. It's important to remember that we are all on the same side here. With almost no exceptions, Internet service providers hate spam, and will cut off the spammer's connectivity when they find out about it."
And then wait for the next attack.
"The challenge we face is the same challenge little Hans Brinker faced when he stuck his finger into that dam," Ramasubramanian said. "We know that as soon as we let our collective fingers slip out of the thousands of tiny holes we are plugging we will drown in a massive sea of spam."
Shares