• Live Search Webmaster Central , Tuesday, September 16, 2008 12:15 PM

Jeremiah Andrick cautions against the practice of having dynamic robots.txt files, or using them to help manage server load as the engines come crawling throughout the day.

"You may perceive a benefit from changing your file throughout the day as you may see less server load from some search engines on your site, but this behavior also works against you," he says. "When the crawler returns it does not retry fetching the content for URLs previously disallowed a few hours ago and the crawler may fetch content outside of your new directive as the crawler may be using the previously cached robots.txt file."

So you get content indexed that you may have wanted to hide, and a crawler that might ignore the things you actually want searchers to see. Andrick posts an example of a Web site that was seemingly not being indexed by Live Search, and shows how a dynamic robots.txt file was keeping all of the site's content (including items like the homepage, products pages and ads) from being crawled.

Read the whole story at Live Search Webmaster Central »