• by Rob Garner , September 16, 2009

I'm not sure why, but the phrase "real-time search" has somehow largely become associated with the phrase "Twitter search." While much of the conversation around real-time search centers conceptually on Twitter search and other engine algorithms that remix their data, the fact is that the Google crawl and many other viable contributors are critical to getting a fresh and relevant answer in real-time, though they are often lost in the discussion.  Make no mistake about it, crawler-based real-time search and status-update search are both important to the successful development of a state-of-the-art robust and relevant real-time search engine. 

Real-time search is an entirely different animal from its two main components, "traditional" crawler search, and keyword discovery via social streams. The end goal in "real-time" search is about getting the best results now or within a recent period of time, versus getting the best results over history.  It is entirely different from the keyword monitoring of a social stream for discovery, and is also different from fresh, crawler-based results, even if they are delivered within a shorter timeframe.  Perhaps the most unique aspects of real-time search (in its current incarnation) are that they are keyword-triggered, viewed in reverse chronological order (or other recency parameters), and that many of the best results are collated by a human audience.  These attributes ultimately differentiate it from either typical keyword-triggered stream view, or keyword search in your favorite search box.

Sources of "in-the-moment" content, aside from Twitter: Beyond Twitter and Google's up-to-the minute crawl, there are many other elements of discovery and search that are not often found in a discussion about real-time search, but can be very helpful in obtaining the right answer at the right moment (either individually, through social discovery, or crawler-based search).  Here are just a few examples that, with a little bit of social propagation and a robust crawl, can add to the real-time search experience:

Forum chatter:  Forums and bulletin boards are the core of the Internet's continuing evolution, and were "social" long before the term was coined in 2004.  While forum content is often timely and useful, it has long been buried in the search results. To see an example of how the recency of forum posts is improving via Google search, see this query for results crawled in the last 24 hours for "obama" sorted in chronological order.

Microformats:  Microformatted attributes (semantic interpretation of information like reviews and locations, among many other uses) could become very useful in real-time search, as adoption of these formats continues to increase.  The Google crawl could conceivably sort this segmented data to help you find what you are looking for in very short order.

Comments.  User participation in the comments sections of online publications and blogs has exploded in the last year.  For example, it is not uncommon for sites like the Huffington Post to get 1,000 or more timely comments on a single story or post.  But much of the potential usefulness of this data is currently wasted, as it is either unedited, or unsorted (much as Twitter streams are today).  Adoption of an algorithmic/social ranking approach to comments could change this (see Digg as an example of how mass volumes of comments are parsed to become useful to its readers). 

Google Hot Trends.  If people think that Twitter's measly top 10 trending topics are the apex of social search  discovery, then I would recommend they check out Google Hot Trends, or other buzz indexes for some more fascinating sources of what is being talked about right now, and what people are searching for, right now.

Are we there yet, are we there yet, are we there yet?

Real-time is in its earliest stages, and while it is fun to prognosticate about the possibilities, the charge with the major engines is clear.  Larry Page has stated that real-time is a priority of Google search.  Evolution in Twitter search seems to be moving at a snail's pace, though it is has been reported that company strategists are rethinking their approach.  But there is still a key place for Twitter, and nothing could be better for real-time search than the combining of the two in some fashion.  But don't start planning your strategy around real-time search just yet.  This is an innovation that is years in the making, and it is much bigger than Twitter alone.