If you're at all lucky, you'll spend at least part of this summer far away from spreadsheets, reports, ad consoles, and bid management software. While there's nothing wrong with single-mindedly
focusing on all the granular data associated with search, too much data, all the time, can rot your brain.
With this latter point in mind, this week's column attempts to wrestle with some
large-frame issues that I think deserve attention, because, as we learned from the financial crisis, it's often the most basic, most obvious issues (such as whether all subprime mortgages that were
being issued could possibly ever be repaid) that can blow everyone out of the water. In other words, just because nobody's talking about something doesn't mean that it's not real enough to kill you.
There's been a lot of ink spilled in recent months about what it takes to differentiate a search engine enough so that users switch from one to another. Frankly, I think this conversation totally
misses the mark. Journalists may care a lot about the bells and whistles adorning a particular engine's UI, but users seem totally oblivious. The only thing that really matters to end users is finding
the information they want, as quickly as possible -- and, as we all know, Google's PageRank system represented a genuine breakthrough in an era when SERPs were chaotic, meaningless, repetitive
nightmares. Today, however, we're in an era wherein all of the major engines use very similar algorithms, yielding largely identical (or at least comparatively similar) results.
advertisement
advertisement
I find it
useful to think of search engines as digital cameras all aimed at the same landscape. One camera might have slightly better resolution than another, or better low-light performance, or faster shutter
performance. But they all take pretty good pictures, and unless you're a professional photographer or digital camera reviewer, you really don't care. Most of us just want to capture the moment and
move on to another.
What changes things is when one of these cameras is able to capture colors, or parts of the landscape, or parts of the visible light spectrum, that the others can't, and
that's the one you'd want if you needed to capture these elements. Put another way, if you were going to visit Mount Rushmore, and you knew that Mount Rushmore could only be satisfactorily
photographed with a Nikon camera, you'd pack that one, and leave the others behind.
Let's get back to search engines. Today, search engines index the same body of information: the crawlable
World Wide Web. But suppose a large part of this body of data suddenly "went dark" to all but one engine? Suppose additionally that the portion of this data which became invisible happened to be
strategic to your information task at hand? Well, you'd likely switch to that engine immediately, because it would be your only portal into this body of knowledge.
It's obvious that search
engines have a powerful economic incentive to "lock up" such data because it is the only real way that they can meaningfully differentiate themselves enough to compel users to switch. Nor am I
convinced that the reason this hasn't happened -- that most Web site operators have en masse opted to welcome any spider attempting to access their sites' content -- is an immutable situation. My
opinion of most site operators (and yes, I am one) is quite cynical: most of them share characteristics with street musicians performing in the subway for spare change (which is not to say that some
of them don't get a lot of spare change through AdSense and other ad networks). In other words, it wouldn't take much to bribe them into refusing another spider's visit, provided the price is right,
and right now, this price is almost ridiculously low.
The problem, of course, is that The New York Times and other A-level content sources aren't street musicians: they're world-class
concert performers who happen to be working at street-musician rates, which is why they'll fail until something radical changes in the fundamentals of the information economy. In fact, they've got
more of an incentive to "lock down" their content than even the search engines, which is why, I think, within the next year or so, we'll start seeing the most authoritative sources of content on the
World Wide Web "going dark" to information aggregators (which is all the search engines really are) that don't cut them a better deal. Many will mourn the fracturing and Balkanization of the Web into
pieces which are no longer universally accessible in the way we're accustomed to, but unless somebody comes up with a better solution, very soon, such a Balkanization will soon come to pass.