Google’s war on spam and how encryption could finally win it – for the spammers

When you cut the knees out from under a complex society, as Edward Snowden and the NSA have done to the internet over the past year or so, the effects ripple outward unpredictably. Right away, there was a rush on cryptography software, which immediately threatened the online status quo; privacy software might just as accurately be called “anti-analytics” or “anti-big-data” software, and your details and behavioral data (tracking cookies) are the lifeblood of the online economy. That looming problem can only get so big while encryption solutions remain clunky and intimidating to newbies — but pressure is mounting for more aggressive, far-reaching protection of online traffic. In particular, large email providers are looking forward to a future in which they must try to protect a user’s inbox while encryption prevents them from knowing virtually anything about it. By using encryption to protect ourselves from Google, hackers, and the NSA, we could be making ourselves vulnerable to spam.

Mike Hearn, a former security engineer with Google’s Gmail team, has posted a fascinating explanation of his team’s long-term war on spam. In it, he details the company’s back-and-forth struggles with spammers for power over the inbox, as simple phrase bans gave way to more powerful methods of filtering. He recounts the time an unfortunate Italian woman named OliVIA GRAdina had 100% of her email flagged as spam. Each breakthrough in security was met with a new innovation from the spammers; filter hyperlinks, and the spammers start hiding them and abusing link-shortening services; start a curated blacklist shared across providers, and they start releasing botnets to distribute the act of sending; get rid of botnets with IP analysis, and they start hijacking whole top-level domains.

Snowden, Assange, Greenwald, Dotcom and other figures in the privacy debate recently gathered to push for better general online security.

Today, says Hearn, we’ve reached a point that could credibly be called a win for the defenders at Google. The “mark as spam” button has worked in conjunction with a complex reputation system to quickly quash any new spam accounts that crop up. We all get a bit of spam, but by removing it ourselves we do our part to keep the overall ecosystem clean — think of it as community service. User-generated reputations have been the lynchpin of Gmail’s spam success — and encryption would make those reputations next to impossible to compile. For Google to filter spammy links from your email it must first be able to seethose links, which means being able to see those emails. For Google to weigh the collective implications of 10 spammy emails from a single source, it must know that 10 people have received such an email, all from that single source.

Search can’t automatically suggest directions to the discussed restaurant, and Now can’t auto-log the reservation, if all our files and communications are obscured. Google does a great job marrying value to the corporation with value to the user, making sure that most of the info it collects for its own gain is also used to provide some value to the user. It’s Google’s best shot at being able to maintain the business model long-term; get people hooked on conveniences and they’ll put up with almost anything to maintain them. Whether the public will choose information security and privacy over Google’s lavish feature set remains to be seen, but in all likelihood most people will choose to keep low security standards rather than have to delete twenty Viagra ads every morning.

Lavabit, a former encrypted email service, had to close its doors because even partial encryption couldn’t provide strong enough security for the company’ own liking.

As regards email specifically, there might be some solutions that beat both prying eyes and annoying ads: voluntary decryption for email filtering. The “mark as spam” button would obviously have to decrypt the marked email for Google, but it would have to be supplemented by an (inevitably ignored) button called “mark as legit.” As Hearn explains in his post, for the purposes of building reputations, the act of not marking something as spam is just as important as the reverse. With full encryption, though, a user’slack of action could never be useful.

We might also imagine some sort of client-side filtering algorithm that reads your email locally and pulls out some mutually acceptable scrubbed version of the metadata for transmission to the security server. This introduces all sorts of new points of attack, though, and it has not been successful in the past. The former encrypted email service Lavabit ended up charging users for effective spam filtering, and Lavabit didn’t even offer true end-to-end encryption. Charging for basic account features is not an option for mass providers like Gmail, anyway.

In general, there’s no clear path to both protecting and maintaining email as we know it today; much of the online functionality we take for granted derives specifically from Google’s all seeing eye, and encryption is a big pointy stick. Until someone figures out how to let providers simultaneously both see and not see into email (and our lives in general) encryption will remain a conflicted issue for large online corporations. They genuinely want to protect their users, but not at the expense of their own ability to make money and provide useful services. It’s a tough problem, and not one we’ll be able to ignore for much