Microsoft's Live Search team has upgraded the engine's crawler, changes that should "significantly improve the efficiency with which we crawl and index your Web sites," according to Nathan
Buggia.
MSNbot (now version 1.1 as opposed to 1.0) now supports HTTP Compression and Conditional Get. HTTP Compression helps speed up communication between a site's server, the MSNbot and
the Live Search Index by compressing static files and application responses. Better communication means that the crawler can scan new content, update the index and present searchers with more relevant
info more efficiently.
Meanwhile, Conditional Get can help reduce server load, as it prevents the MSNbot from downloading a page unless it has changed since the last time it was crawled.