Commentary

Kill The Bots First, Then Tackle Viewability

by Augustine Fou , March 31, 2015

This recommendation is based on the simple concept that even if an ad is 100% viewable, if it was caused to load by a bot, it is still fraud, and useless for advertisers, who thought their ad was shown to a prospective human customer.

Bots have been around for as long as the Internet, first known as viruses and malware that infected people’s home computers. They stole personal information and visited websites, per the direction of their botmasters, to create fake traffic, ad impressions, and clicks.

But the days of relying on unsuspecting humans to accidentally click and download malware onto their PCs are limited. Modern-day botmasters can “spin up” millions of headless browsers in data centers to commit ad fraud on a far larger scale than could be done with individual PCs. These headless browsers were designed to help developers pressure test websites before launching them. The bad guys use millions of these fake browsers to simulate humans visiting websites, therefore generating ad impressions, video views, and clicks.

The more sophisticated bots have been observed to create fake mouse movements and page scrolling. They can add items to shopping carts, deliberately abandon them, collect cookies, and wait for retargeting to follow them to websites owned by the bad guys. When the ad impression is served on those sites, the bad guys make off with the money.

These bots are irresistible because they generate the lift in traffic that every publisher lusts for. They create ad impressions at very low cost, something that media buyers lust for -- to give their clients more impressions at lower average cost. And they generate the click-throughs that are thought to mean engagement and actions leading to conversions that advertisers love.

If you analyze more deeply and carefully, these ad impressions, visits and clicks all end with no conversions. I don’t mean just ecommerce sales; I mean any conversion event that brand advertisers could measure on the path towards sales. This is because all these actions are caused by bots.

So, the bot problem and NHT (non-human traffic) need to be solved first. Some commercial anti-fraud vendors are doing what they are required to do: screen against blacklists provided by industry associations. Unfortunately those lists contain a fraction of the approximately 10,000 known bots that have been observed in the wild.

But the bad-guy bots are NOT these. Bad-guy bots do not declare themselves honestly in the user agent, as Googlebot, Bingbot, Facebookbot, etc. do. Bad-guy bots disguise themselves as Internet Explorer, mobile Safari, and every other flavor of browser that humans would use.

Some published numbers suggest that the bot problem is only 1% - 3% of total traffic. That would be true if you only counted search engine crawlers and known bots. It would be a dramatic underestimate if you are looking for fraud bots.

What advertisers and agencies should do is immediately leverage the anti-fraud technology vendors to help them identify highly suspicious bot activity using big data analytics, machine learning, and heuristics around network traffic. Using this kind of “broad strokes” detection, the most obvious bots can be caught. The bottom-most decile (10%) should be lopped off, by blacklisting those sites that loaded the impressions or sent the traffic. This will have a positive business impact even while viewability is still being worked out.

ad fraud, online fraud, programmatic, rtb, viewability

7 comments about "Kill The Bots First, Then Tackle Viewability".

Check to receive email when comments are posted.

Rich Kahn from eZanga.com, Inc., March 31, 2015 at 1:54 p.m.
Augustine, I also believe if we as an industry make big strides in eliminating click fraud then visibility will automatically increase to more acceptable levels by default.
In addition, make sure you buy traffic from a provider that has been around for a while and either owns a system of eliminating click fraud or employees a 3rd party tool. You as the advertiser should not have to pay the bill to clean the traffic...leave that up to the provider you buy from.
Reply

Augustine Fou from FouAnalytics, March 31, 2015 at 3:39 p.m.

Agreed.. but even advertisers need to make an effort to detect and mitigate fraud. They can no longer just assume that someone else will take care of it -- like their media agency or the publishers.

One key point that I always emphasize to my clients is to get first-hand data so they can independently verify what their agencies and vendors are telling them.

Rick Monihan from None, March 31, 2015 at 4:17 p.m.

It's hard to accept that "viewability" is the end game and the solution. I recently had a client tell me they are more concerned with NHT than viewability because it matters little of an ad is seen by a bot.
The focus on viewability alone, without commensurate, or greater, emphasis on the NHT issue, has been a misplaced focus. Hopefully in the months ahead, we will pivot away from simply viewability and onto the more tangible question of "what does it mean to be human?" A question which Isaac Asimov may have been better positioned to answer, but we still need to emphasize and address.
Premium publishers have a need to push this agenda hard against long-tail purchasers.

Rich Kahn from eZanga.com, Inc., March 31, 2015 at 8:44 p.m.

Augustine, what I am saying is that if you buy traffic from a company that does not clean the traffic, then you may be paying for a percentage of traffic that you can't use based on your own or some 3rd party analytics. Make sure the company you buy from is cleaning the traffic so you are not paying for traffic you don't want. I agree, you still need to verify the results yourself, but if you work with a reputable company that is doing the right thing you won't be paying for traffic you can't use.

Augustine Fou from FouAnalytics, April 1, 2015 at 7:38 a.m.

Rich, yes of course you would expect the company to "cleanse" the traffic but most companies do not. They make more money the more traffic (and therefore ad impressions) they get. So they have little motivation to cleanse and reduce that volume.

Mainstream publishers have very strict rules about not purchasing traffic of any kind, plus other good policies.

Augustine Fou from FouAnalytics, April 1, 2015 at 7:42 a.m.

Agreed with Rick -- any viewability efforts must work hand-in-hand with efforts that detect and mitigate bots. Otherwise, even a 100% viewable ad is still useless if it were caused to load by a bot.

And the key point I mentioned above is that the bad guys bots are not in ANY of the known bot lists. They disguise themselves in order to trick the ad networks to allow the ad impression to load. So just using industry association bot lists to filter is NOT enough.

Rich Kahn from eZanga.com, Inc., April 1, 2015 at 4:28 p.m.

Augustine, I know that is the sad truth. I wish more companies out there were more proactive. Don't they realize that they can get higher prices for clean traffic?