• by Gord Hotchkiss , Featured Contributor, July 5, 2007

Aaron, Aaron, Aaron. Could I possibly leave you as a lone voice out in the wilderness, prophesizing about personalized search? Of course not.

Last week, fellow Search Insider Aaron Goldman pointed out some loopholes in personalized search nirvana. It's hard to find fault with his points. They're all very real flaws in making personalization a credible evolution in search relevancy. Also, somewhere along the line, it appears that I've become the cheerleader for personalized search. I do admit I'm somewhat bullish on it, but I think I should clarify why I think personalization is important.

It's Time to Break Search's Paradigm

Search has hit the ceiling, at least in its current embodiment. We've pushed the paradigm as far as it will go. Search's nose is smashed up against the window. (I should stop writing these columns late in the evening, after a 15-hour day!). Search needs to go somewhere, and after looking at the alternatives, I believe personalization is the most probable path.

All the improvements in search over the past decade have largely been in the background. The interface you and I use has hardly changed since I first discovered Infoseek and AlltheWeb back in 1995. Sure, the algorithms have been tweaked, but they've all been improvements down the same path, and that path is at a dead end. For search to evolve, it needs to move beyond a pure query-initiated, algorithmic-driven exercise. Even universal search, which is the biggest change we've seen to the results page in the past few years, is really still a tweak on the existing paradigm. It's just mixing the bag of results, powered by the same algorithm.

So, when we look at where search can go, there are precious few alternatives. They all aim at the holy grail, disambiguating intent. We can look at human-powered search. The idea behind this is that real, live human beings can deliver greater relevancy than an algorithm ever could. Here tread Jason Calacanis (Mahalo) and Jimbo Wales (Wikia).  Then we have the very close cousin (and in some cases, a stand-in) social search. If we somehow tag results, or implicitly give our vote, even through a click-through, will others who share our interests find the same results more relevant? Finally, we have personalization.

Don't Expect Perfection Anytime Soon

Each approach has potential flaws. Any time you break a paradigm, iterative failure is almost a given. Nobody is going to get it perfect out of the gate. Getting to the next evolution of search will involve trial and error. That's why I think it's particularly brave of Google, given its current market leading position, to be moving aggressively down the personalization path. They're eating their own lunch. It's an inevitable move, but one that it takes guts to make. And don't judge the potential of personalization based on what you're seeing today. It would be akin to trying to determine the eventual impact of the automobile based on your impression of the first horseless carriage that lurched through town. There's a reason it's in beta.

Aaron worries about the search "ruts" that may evolve with personalization. If we tend to go down the same paths again and again, what happens when we want to explore new territory? Will personalization have formed a groove so deep we can't crawl out of it?

Aaron is also concerned about multiple profiles on the same machine within a household. Or for that matter, multiple profiles with the same person. I search differently at work than I do at home. How will a search engine reconcile this search schizophrenia?

Of course, we haven't even touched on the biggest challenge facing personalization: the privacy issue. Personalization is powered by mountains of sensitive data. The potential pushback on this is the biggest red flag that personalization has to contend with.

Making the Leap

But no matter which path search chooses to follow, there will be monumental challenges to address. That's the whole crux of innovation. If it was easy, everyone would do it. But search has no option. For it to evolve into its next stage, which is to take its rightful place as the fundamental glue that connects us all to the highly functional, highly personal semantic Web, search needs to break the current paradigm. And that's why I'm bullish on personalization. As Google's Matt Cutts said to me once (about a totally different topic), if I had a dozen eggs, I'd be putting 11 of them in this particular basket. Sure, personalization has some big hurdles to jump. So do the alternatives. And I think the potential wins for personalization are far bigger. I have the suspicion that if personalization works as well as I think it can, we'll look back five years from now with bemusement at the concerns we had in 2007 around the issue. 

That's the problem when you come to the end of a development path -- and fundamental change, rather than incremental change, is required. It's very difficult to see what lies ahead.