Around the Net

Google Explains The Meaning Of 'Crawl Budget'

  • Google, Tuesday, January 17, 2017 4:02 PM

Google wants to clarify' what "crawl budgets" mean for Googlebot and Web sites. Mostly larger Web sites, according to Gary Illyes. He writes that if new pages are crawled the same day they're published, crawl budget is not something webmasters need to focus on. Likewise, if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently. He runs through things like crawl rate limits, crawl health, and factors affecting crawl budgets. Prioritizing what and when to crawl sites, and the amount of resources the server hosting the site can allocate to crawl the content becomes more important for bigger sites or those that automatically generate pages based on URL parameters, he explains in the post.

Read the whole story at Google »

Next story loading loading..