Understanding that the search engines' spiders crawl a site page-by-page, and rank each page's content individually, is one of basic tenets of SEO. Bill Slawski says that the next phase may be
about site-wide analysis and assessment, as evidenced by a patent Yahoo recently released a paper for (but actually filed for in mid-2006).
With site-wide analysis, Yahoo's
algorithm could consider factors like the keywords and meta data, in- and outbound links, as well as each page's topic and the way it's presented--all in aggregate, and all to develop a more accurate,
comprehensive hierarchy.
Slawski links to, and includes some info from, Yahoo's patent paper, saying: "What I found most interesting about the document was the change in focus of a search engine from crawling and understanding individual pages to understanding how pages within a site relate to each other."