• by Todd Friesen , June 1, 2007

Dynamic features can make your site lost in (search) space

Wandering through an expo hall at a recent conference for Web 2.0, I see company after company with oddly spelled names and unusual logos. I have one question for these companies: Other than flashy booths at conferences attended by other 2.0 folks, how do you think people will find you?

Web 2.0 is largely composed of new applications for existing data and most of these great new applications are not search-engine friendly, because they are based in AJAX. But the problem with AJAX is that it is rendered client-side. That means it's your Web browser that does all the heavy lifting to bring you these pretty Web sites. Essentially, search engine spiders are very basic Web browsers. They don't accept cookies, they don't understand session ids, and definitely don't execute AJAX. If you want to get an idea of what a search engine sees when it crawls your site, go and download a copy of the old Lynx text-based Web browser. I recommend a shot of whiskey before and after this exercise.

What do I care if a bunch of Web sites are simply repurposing existing data through AJAX-based mashups and widgets anyway? In some instances, I truly do not care. But there are a handful that aggregate, while also adding interesting and useful user-generated content. For the well-being of these, I care pretty deeply. These companies are producing great applications I could potentially use, but I will never find them in Google or Yahoo. If a search engine specialist like myself can't find these great 2.0 companies, what chance does the general public have?

You may wonder why I'm bashing AJAX. After all, AJAX does make for a user experience that is easy and fluid, what with its dynamic updates and all. In truth, I will never advocate sacrificing user experience for the benefit of search. However, I will always reiterate that you can't ignore search. You need to implement a solution to provide your content to search engines so they can help bring you new users. They want to index your content and drive you new visitors, subscribers, buyers, etc. If you give them a hand, your results become much more efficient.

Here are the basics of how you can satisfy the needs of your users and the needs of search engines simultaneously: Create two Web sites, one for your users and one for the search engines. For your users, you can serve up the slick, AJAX-laden, user-friendly site, but for search engine spiders, you provide a basic HTML version. The content is the same, but the search engines can read and index it. This is accomplished by identifying the visitor on entry to your Web site based on their IP or User Agent and deciding which version of the requested page to display.

Keep in mind that there's a limit to these techniques. And over-optimizing the HTML version of your site by stuffing keywords, hiding links or other foolish and unsavory tactics will lead to disastrous results. Your site will get banned, and then you'll be left with nothing but your conference booth.