
Google has launched GoogleOther, a new web crawler that
various internal teams within the company will use to crawl the public web.
It is unclear whether Google plans to use GoogleOther for Bard, its conversational artificial intelligence (AI)
service powered by language model for dialog application (LaMDA).
The new crawler will use the same infrastructure and protocols of the main Googlebot crawler such as hostload limitations
and robotstxt -- through a different user agent token, http protocol version, and fetch size. The addition aims to take some strain off the main Googlebot crawlers, as content across the web grows
exponentially.
“As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot's crawl jobs are only used internally for building the index that's used by
search,” Google Analyst Gary Illyes wrote in a post.
advertisement
advertisement
It's basically Googlebot under a different name, he wrote.
The user agent
token and full user agent string are “GoogleOther.”
It is the generic crawler that may be used by product teams for retrieving publicly accessible content from sites.
For
example, it may be used for one-off crawls for internal research and development, according to a post on the Google Search Central blog.