• Jennifer Slegg , Thursday, February 14, 2008 12 PM

Search engine spiders aren't the only ones that can interact with a Web site's robots.txt file. As Jennifer Slegg illustrates, sometimes the very file Webmasters use to keep the crawlers out are like beacons drawing competitors in.

Slegg notes that when you exclude a directory from being indexed with robots.txt,, crafty searchers can actually use the file to find it, and without password protection, they'll be able to see all of the content you tried to hide. In addition, if you're developing any new content, or pushing out a new product and plan to promote it on the site, then using robots.txt to keep it unseen is a no go.

"Play it smart with your robots.txt file so you don't inadvertently hand your competitors the keys to your site or give them the edge up by alerting them to what you are working on," Slegg says.

Read the whole story at Jennifer Slegg »