Bots have been around for as long as the Internet, first known as viruses and malware that infected people’s home computers. They stole personal information and visited websites, per the direction of their botmasters, to create fake traffic, ad impressions, and clicks.
But the days of relying on unsuspecting humans to accidentally click and download malware onto their PCs are limited. Modern-day botmasters can “spin up” millions of headless browsers in data centers to commit ad fraud on a far larger scale than could be done with individual PCs. These headless browsers were designed to help developers pressure test websites before launching them. The bad guys use millions of these fake browsers to simulate humans visiting websites, therefore generating ad impressions, video views, and clicks.
The more sophisticated bots have been observed to create fake mouse movements and page scrolling. They can add items to shopping carts, deliberately abandon them, collect cookies, and wait for retargeting to follow them to websites owned by the bad guys. When the ad impression is served on those sites, the bad guys make off with the money.
These bots are irresistible because they generate the lift in traffic that every publisher lusts for. They create ad impressions at very low cost, something that media buyers lust for -- to give their clients more impressions at lower average cost. And they generate the click-throughs that are thought to mean engagement and actions leading to conversions that advertisers love.
If you analyze more deeply and carefully, these ad impressions, visits and clicks all end with no conversions. I don’t mean just ecommerce sales; I mean any conversion event that brand advertisers could measure on the path towards sales. This is because all these actions are caused by bots.
So, the bot problem and NHT (non-human traffic) need to be solved first. Some commercial anti-fraud vendors are doing what they are required to do: screen against blacklists provided by industry associations. Unfortunately those lists contain a fraction of the approximately 10,000 known bots that have been observed in the wild.
But the bad-guy bots are NOT these. Bad-guy bots do not declare themselves honestly in the user agent, as Googlebot, Bingbot, Facebookbot, etc. do. Bad-guy bots disguise themselves as Internet Explorer, mobile Safari, and every other flavor of browser that humans would use.
Some published numbers suggest that the bot problem is only 1% - 3% of total traffic. That would be true if you only counted search engine crawlers and known bots. It would be a dramatic underestimate if you are looking for fraud bots.
What advertisers and agencies should do is immediately leverage the anti-fraud technology vendors to help them identify highly suspicious bot activity using big data analytics, machine learning, and heuristics around network traffic. Using this kind of “broad strokes” detection, the most obvious bots can be caught. The bottom-most decile (10%) should be lopped off, by blacklisting those sites that loaded the impressions or sent the traffic. This will have a positive business impact even while viewability is still being worked out.