Around the Net

Ah - Googlebot Mysterious Crawl Behavior

There are a variety of ways that Google can visit Web sites. Ted Ives shows us a few, and raises questions about the company's crawling behavior. Aside from pointing to what he considers a "behavior problem," Ives discloses that when it comes to Google Instant Preview, the Googlebot reports a different user-agent, and in addition to running automatically, it appears to update or schedule an update on demand when users request previews. Ives provides a long list of what he calls "relatively mysterious" acts during crawls.

Read the whole story at Coconut Headphones »

Next story loading loading..