| |||||||||||
PRINT
COMMENT
SUBSCRIBE
RSSLast week, fellow Search Insider Aaron Goldman pointed out some loopholes in personalized search nirvana. It's hard to find fault with his points. They're all very real flaws in making personalization a credible evolution in search relevancy. Also, somewhere along the line, it appears that I've become the cheerleader for personalized search. I do admit I'm somewhat bullish on it, but I think I should clarify why I think personalization is important.
It's Time to Break Search's Paradigm
Search has hit the ceiling, at least in its current embodiment. We've pushed the paradigm as far as it will go. Search's nose is smashed up against the window. (I should stop writing these columns late in the evening, after a 15-hour day!). Search needs to go somewhere, and after looking at the alternatives, I believe personalization is the most probable path.
All the improvements in search over the past decade have largely been in the background. The interface you and I use has hardly changed since I first discovered Infoseek and AlltheWeb back in 1995. Sure, the algorithms have been tweaked, but they've all been improvements down the same path, and that path is at a dead end. For search to evolve, it needs to move beyond a pure query-initiated, algorithmic-driven exercise. Even universal search, which is the biggest change we've seen to the results page in the past few years, is really still a tweak on the existing paradigm. It's just mixing the bag of results, powered by the same algorithm.
So, when we look at where search can go, there are precious few alternatives. They all aim at the holy grail, disambiguating intent. We can look at human-powered search. The idea behind this is that real, live human beings can deliver greater relevancy than an algorithm ever could. Here tread Jason Calacanis (Mahalo) and Jimbo Wales (Wikia). Then we have the very close cousin (and in some cases, a stand-in) social search. If we somehow tag results, or implicitly give our vote, even through a click-through, will others who share our interests find the same results more relevant? Finally, we have personalization.
Don't Expect Perfection Anytime Soon
Each approach has potential flaws. Any time you break a paradigm, iterative failure is almost a given. Nobody is going to get it perfect out of the gate. Getting to the next evolution of search will involve trial and error. That's why I think it's particularly brave of Google, given its current market leading position, to be moving aggressively down the personalization path. They're eating their own lunch. It's an inevitable move, but one that it takes guts to make. And don't judge the potential of personalization based on what you're seeing today. It would be akin to trying to determine the eventual impact of the automobile based on your impression of the first horseless carriage that lurched through town. There's a reason it's in beta.
Aaron worries about the search "ruts" that may evolve with personalization. If we tend to go down the same paths again and again, what happens when we want to explore new territory? Will personalization have formed a groove so deep we can't crawl out of it?
Aaron is also concerned about multiple profiles on the same machine within a household. Or for that matter, multiple profiles with the same person. I search differently at work than I do at home. How will a search engine reconcile this search schizophrenia?
Of course, we haven't even touched on the biggest challenge facing personalization: the privacy issue. Personalization is powered by mountains of sensitive data. The potential pushback on this is the biggest red flag that personalization has to contend with.
Making the Leap
But no matter which path search chooses to follow, there will be monumental challenges to address. That's the whole crux of innovation. If it was easy, everyone would do it. But search has no option. For it to evolve into its next stage, which is to take its rightful place as the fundamental glue that connects us all to the highly functional, highly personal semantic Web, search needs to break the current paradigm. And that's why I'm bullish on personalization. As Google's Matt Cutts said to me once (about a totally different topic), if I had a dozen eggs, I'd be putting 11 of them in this particular basket. Sure, personalization has some big hurdles to jump. So do the alternatives. And I think the potential wins for personalization are far bigger. I have the suspicion that if personalization works as well as I think it can, we'll look back five years from now with bemusement at the concerns we had in 2007 around the issue.
That's the problem when you come to the end of a development path -- and fundamental change, rather than incremental change, is required. It's very difficult to see what lies ahead.
Meet Gord Hotchkiss at Search Insider Summit Utah!
Gord Hotchkiss will be there speaking during "Conference Opens and Opening Remarks" on December 03 at 8:45 AM. Top executives will be there. Will you?
Register today and save.
Register today and save.
RSS FEEDS
You make a great point when you say that the reason for personalized search is to provide better, more relevant results. The fact that your history doesn't define your future (at least, not exclusively) just shows that personalization technology shouldn't rely only on historical activity.
We've proven that relevance can be increased by calculating and mathematically representing a user's purpose and values. As you change and grow, or as others who share your values find different experiences, you'll get different results (which also addresses Aaron's point from last week about eventually narrowing what you see down to the three possibilities you've clicked on in the past).
If your results are driven by what you care most deeply about, it doesn't matter if you don't know what you want for a given search! You're likely to find new things that you never knew you'd connect with.
Thanks for a good debate guys! I picked up on it in my blog -- can't wait to see Aaron's response next week!
All the best, Kaila Colbin
The problem of giving people what they want when they don't know what they are asking for is probably unsolvable.
In fact, the most helpful answer may well be an "am I getting warmer" function -- given a page of unhelpful search results, let people click a red button (hot) by the result that is closest to what they want, and a blue button (ice-cold) by the result that is farthest off.
There is the secondary problem of giving people what they want when THEY know what they mean but express their request poorly.
I'm no computer scientist, but this seems much more solvable. Why? Because individual personalities have communications quirks that can be learned. For example, when I can't understand what someone at my office wants, I often quietly go visit their administrative assistant.
A great admin is used to that person's patterns of speech and thinking and has usually created a sort of lookup table in their heads -- "when she says X, she almost always means Y". This intelligent, on-the-fly translation is what enables the admin to be great at the job of giving great results.
With enough search history, a good algorithim should be able to make similar predictions about me and what I want. Even when I'm asking for it in a truly clumsy way.
Now I am an anomaly. I have experience in relational database design and SQL. I love Boolean algebra. I rarely have any trouble finding exactly what I want, using any search engine. But I'm not that much of an anomaly. My three children also find what they want quickly, using Google or other search enginers.
Glad to see you didn’t try and combat the challenges involved with making Personalization accurate and valuable. Instead you focused on why it is so important to address them.
I took a similar approach in my column, spending less time on how we can make Personalization work and more on what will happen when Personalization works (as I do agree that it’s inevitable, it may just take a while).
Your horseless carriage comment reminded me of the famous Henry Ford quip, “If I’d have listened to what people wanted, I’d have built a faster horse.�
We need to be patient and stay the course of Personalization. With each baby step of innovation we get closer to what you astutely point out is the holy grail - “disambiguating content.� That is what search is all about. And that is what Personalization is all about.
It will be interesting to see how this all plays out.
You make it sound like search no longer works. Sure, there may be some aspects where personalized search helps with the process, but I also see where it's not going to be effective.
What I don't understand is why people spend so much time trying to "fix" search, when it would make more sense to me to educate the searchers? Most people don't use the advanced search options of most search engines and they don't even know how to use quotes when they search. No wonder they can't find anything!
Personalization sounds like it's going to take a hammer and pad it with foam and add a laser/radar guidance system to help you strike smart-chip embedded nail heads...