• by George Bilbrey , Op-Ed Contributor, April 13, 2011

The history of deliverability for the last decade has been driven by the fight that large mailbox providers have waged against the Internet "axis of evil" -- those that abuse messaging.  A brief history of the battle (highly simplified, highly impressionistic, with approximate dates) would look something like this:

Prior to the mid-1990s:  Spam and phishing were not major problems. 

Mid-1990s to 2001: Spam starts to become a serious problem. There aren't that many different spam attacks per week, so big mailbox providers build content-based filtering. It's somewhat effective, but has a lot of false positives.  Much of the mail comes from servers that are actually run by spammers (not stolen resources) or through open relays.  IP-based blacklists come to the fore.

2001-2005:  Spam continues to take off.  Phishing starts to take off. Open relays are still a problem but start to come under control. IP-based blacklists are the state of the art. The first IP-based reputation systems start to get significant investment.  ISPs start to build internal whitelists and use third-party whitelists to reduce false positives.

2005-2009:  Spam becomes a huge problem -- it quickly rises north of 80% of all inbound mail.  IP reputation systems, in combination with content filtering systems, get very sophisticated.  Spammers start to use botnets to avoid IP reputation system by changing IPs frequently.  Phishing continues to rise as a problem.  Authentication schemes (SPF, DKIM) are advanced and used, with varying levels of success.

2009-Now:  Real spam in the inbox falls to very low levels (at least for now).  Domain reputation systems, URL reputation systems and IP reputation systems start to put a real dent in spam making it into the inbox.  But that doesn't mean spam has gone away -- at the gateway it exceeds 90% on incoming messages.  Phishing continues to rise and spreads to non-financial service brands.   Graymail (messages that customers sign up for, but aren't universally loved by all subscribers) starts to become the next big problem.  ISPs invest in individual-level filtering to filter out mail based on whether or not an individual interacts with messages coming from a particular source.

So what are the current trends that will drive changes to how ISPs block and filter?  Two trends that we're already seeing some investment in are as follows:

1.     Helping consumer manage graymail:  The large mailbox providers are doing a pretty good job of keeping unsolicited mail from making it into the inbox.  This isn't a source of complaints from mailbox subscribers.  However, graymail continues to drive dissatisfaction at the major mailbox providers.  Mailbox providers will respond by continued investment in new user interface designs that make important mail -- as determined by individual preferences -- more prominent and accessible.  Priority Inbox at Gmail and the Hotmail Sweep function are only the beginning.  You will need to stay attentive to changes in interface design and filtering techniques that could have a major impact on your marketing programs.

2.     Reducing the threat of phishing and malware:  Because it's harder to get large masses of spam through to the inbox, it's harder to make money by sending spam.  This means phishing has become more attractive, because each success has a high value and you only need a few "successes" to make phishing and malware distribution worthwhile.  Stealing credentials and installing malware on an end-user's machine has a larger financial value than sending spam on a per message basis. Going forward, large mailbox providers will invest more in making authentication easy for spoofed senders as well as making it easier for senders to register unauthenticated mail so that it can be blocked from inboxes.  

What trends do you see?  What are your predictions for how spam and abuse trends might influence the entire email channel?