Rick DeJarnette tells us how to communicate with search engine robots without getting "lost in space," referring to the 1960s television show. In this lengthy post, he provides sample code, tips and tricks to block bots from crawling specified directories and files.
A matrix offers insight into the purpose for specific code, such as noindex, nosnippet, noarchive, nocache, and noodp. For example, "noodp" instructs bots not to use a title and snippet from the Open Directory Project (ODP) for that page in the SERP.
Read the whole story at Bing Community »