• by Laurie Sullivan  @lauriesullivan, April 20, 2023

Google has launched GoogleOther, a new web crawler that various internal teams within the company will use to crawl the public web.

It is unclear whether Google plans to use GoogleOther for Bard, its conversational artificial intelligence (AI) service powered by language model for dialog application (LaMDA).

The new crawler will use the same infrastructure and protocols of the main Googlebot crawler such as hostload limitations and robotstxt -- through a different user agent token, http protocol version, and fetch size. The addition aims to take some strain off the main Googlebot crawlers, as content across the web grows exponentially.

“As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot's crawl jobs are only used internally for building the index that's used by search,” Google Analyst Gary Illyes wrote in a post.

It's basically Googlebot under a different name, he wrote.

The user agent token and full user agent string are “GoogleOther.”

It is the generic crawler that may be used by product teams for retrieving publicly accessible content from sites.

For example, it may be used for one-off crawls for internal research and development, according to a post on the Google Search Central blog.