• by Shankar Gupta , February 8, 2006

Hoping to cut down on overcounting of ad impressions, The Interactive Advertising Bureau Tuesday released an updated list of bots and spiders--programs that automatically crawl the Internet for information.

The organization aims to catalogue all the common bots and spiders, in an effort to warn publishers, analytics firms, and third-party ad servers about which ads are being served to automated programs instead of eyeballs.

"If there's a spider who goes on to a particular Web site, we want to make sure we know that the Web traffic generated and the ad impressions were created by those spiders, so we're able to discount those," said Jack Smith, vice president of product strategy for 24/7 Real Media, which subscribes to the list and aided in its creation. "We want to make sure that every page view, every ad impression, is seen by humans."

The updated list will be maintained by two companies--U.K.-based ABC Electronics, and Elgin, Ill.-based ImServices--and will include bots and spiders from both U.K. and U.S. sites. The updated list contains 315 known bots and spiders on the exclude list. Previously, the list was maintained by ABC Interactive, a circulation auditing firm.

IAB CEO Greg Stuart said that discounting hits on Web sites and advertisements is enormously important to the industry, because each false-positive represents an unnecessary payment by an advertiser. "I would be shocked [if] there's not 100 percent use of the list," he said. "I would think that an agency or marketer should confirm that whatever tech provider they have connected are absolutely using this list--otherwise, they're paying for nonhuman activity, which according to the guidelines they shouldn't be."

24/7's Smith estimated that the proportion of traffic made up by bots and spiders can range from .5 percent on average to as high as 3 percent for sites that are especially popular or are updated very frequently.