• by Shankar Gupta , December 14, 2005

A decision announced Monday by Amazon's Alexa to open its Web indexes and allow developers to create their own search algorithms and Web services could bolster innovation in search technology, experts and developers said.

The new service, which launched into beta on Monday, allows developers to access the entirety of Alexa's 100-terabyte index, and use their server time at the cost of $1 CPU hour used, $1 per gigabyte of storage, $1 per 50 gigabytes of data processed, and $1 per gigabyte of data uploaded.

Jupiter Research Analyst Gary Stein said the opening of Alexa's index of roughly five billion documents could be a boon to developers who have ideas on how search should be done, but don't have the money for the massive server farms and crawling infrastructure needed. "There's a lot of innovation that could happen in search, but the barrier to entry is too high," he said. "Turning that over to the world is an interesting opportunity, and it also says that maybe the index isn't the important thing, but the interface is the important thing in terms of launching a new site or service."

Dubbed Alexa Web Platform search, the feature aims to be a one-stop service for launching a search-based Web service, providing index, server time, and hosting. "Just imagine all the talented entrepreneurs who have been stymied by a lack of Web-scale tools and data. Now, for less than the cost of an iPod, they can get into the search field and begin inventing and creating," wrote Alexa Project Manager Geoffrey Mack on the Alexa blog. "You as a consumer can begin using these new search services, and all of us can begin reaping the benefits of an expanded search space with hundreds of wholly new search services being created by anybody with an idea and a credit card."

Paul Forrester, CEO of Indeed.com, a vertical search engine created using search engine APIs, said the decision to open Alexa's index follows a trend that the release of APIs began. "It very much dovetails into the whole thing that's taking place--of search engines opening up their APIs and making their data more available, more accessible," he said. "I think the difference here is that you get access to the raw data, whereas with an API, it's mediated through the search. If you've got the raw data, you can create your own selection, your own [algorithm]--and it gives you a lot more flexibility as a developer."

Search expert John Battelle wrote on his blog that Alexa's move may force major search companies like Yahoo! and Google to re-examine their own APIs. "What has been a jealously guarded secret--the contents of the entire index--is now available to anyone who wants it," he wrote. "I am quite sure this means that Yahoo! and Google will have to stare hard at their own (somewhat limited) search services and APIs, and think what they might do to compete, that much is certain."

Stein, however, said that the threat of competition comes not from forcing Google and Yahoo! to open their indexes in response, but to what search services might emerge from Alexa's offering. "They don't have to worry too much about opening up their index--they have to worry about whether someone's going to do something novel that will suddenly pop onto the marketplace," he said. "Your biggest competition isn't necessarily the known competition, but the unknown."