With this latter point in mind, this week's column attempts to wrestle with some large-frame issues that I think deserve attention, because, as we learned from the financial crisis, it's often the most basic, most obvious issues (such as whether all subprime mortgages that were being issued could possibly ever be repaid) that can blow everyone out of the water. In other words, just because nobody's talking about something doesn't mean that it's not real enough to kill you.
There's been a lot of ink spilled in recent months about what it takes to differentiate a search engine enough so that users switch from one to another. Frankly, I think this conversation totally misses the mark. Journalists may care a lot about the bells and whistles adorning a particular engine's UI, but users seem totally oblivious. The only thing that really matters to end users is finding the information they want, as quickly as possible -- and, as we all know, Google's PageRank system represented a genuine breakthrough in an era when SERPs were chaotic, meaningless, repetitive nightmares. Today, however, we're in an era wherein all of the major engines use very similar algorithms, yielding largely identical (or at least comparatively similar) results.
I find it useful to think of search engines as digital cameras all aimed at the same landscape. One camera might have slightly better resolution than another, or better low-light performance, or faster shutter performance. But they all take pretty good pictures, and unless you're a professional photographer or digital camera reviewer, you really don't care. Most of us just want to capture the moment and move on to another.
What changes things is when one of these cameras is able to capture colors, or parts of the landscape, or parts of the visible light spectrum, that the others can't, and that's the one you'd want if you needed to capture these elements. Put another way, if you were going to visit Mount Rushmore, and you knew that Mount Rushmore could only be satisfactorily photographed with a Nikon camera, you'd pack that one, and leave the others behind.
Let's get back to search engines. Today, search engines index the same body of information: the crawlable World Wide Web. But suppose a large part of this body of data suddenly "went dark" to all but one engine? Suppose additionally that the portion of this data which became invisible happened to be strategic to your information task at hand? Well, you'd likely switch to that engine immediately, because it would be your only portal into this body of knowledge.
It's obvious that search engines have a powerful economic incentive to "lock up" such data because it is the only real way that they can meaningfully differentiate themselves enough to compel users to switch. Nor am I convinced that the reason this hasn't happened -- that most Web site operators have en masse opted to welcome any spider attempting to access their sites' content -- is an immutable situation. My opinion of most site operators (and yes, I am one) is quite cynical: most of them share characteristics with street musicians performing in the subway for spare change (which is not to say that some of them don't get a lot of spare change through AdSense and other ad networks). In other words, it wouldn't take much to bribe them into refusing another spider's visit, provided the price is right, and right now, this price is almost ridiculously low.
The problem, of course, is that The New York Times and other A-level content sources aren't street musicians: they're world-class concert performers who happen to be working at street-musician rates, which is why they'll fail until something radical changes in the fundamentals of the information economy. In fact, they've got more of an incentive to "lock down" their content than even the search engines, which is why, I think, within the next year or so, we'll start seeing the most authoritative sources of content on the World Wide Web "going dark" to information aggregators (which is all the search engines really are) that don't cut them a better deal. Many will mourn the fracturing and Balkanization of the Web into pieces which are no longer universally accessible in the way we're accustomed to, but unless somebody comes up with a better solution, very soon, such a Balkanization will soon come to pass.