Welcome | View My Profile | Sign Out
MediaPost Home About MediaPost Privacy/Terms Media Kit Sitemap
Publications Home News
Online Media Daily Media Daily News Marketing Daily Mobile Marketing Daily Search Marketing Daily
Daily Feed> Email Daily Feed> Video Daily Feed> Social
Online Blogs
Online Spin Email Insider Search Insider Behavioral Insider Online Publishing Insider Mobile Insider Video Insider Gaming Insider Performance Insider Metrics Insider Social Media Insider Just An Online Minute Daily Online Examiner Raw Blog
Media Blogs
Research Brief Diane Mermigas:On Media TV Watch TV Board Magazine Rack Media Creativity Notes From the Digital Frontier Digital Outsider Mad Blog Red White and Blog
Marketing Blogs
Engage:Hispanics Engage:Kids 6-11 Engage:Moms Engage:Boomers Engage:Gen Y Engage:Teens Marketing:Green Marketing:Sports
Magazines
OMMA Magazine Media Magazine
Subscribe
Feedback Loop RSS Feeds Archives Subscribe
Dec 2 Search Insider Summit (Utah) Dec 6 Email Insider Summit (Utah) Jan 11 OMMA Agency of the Year (NYC) Jan 12 MEDIA Agency of the Year (NYC) Jan 26 OMMA Social (San Francisco) Jan 27 OMMA Performance (SF) Feb 24 OMMA Metrics Measurement (NYC) Feb 25 OMMA Behavioral (NYC) Mar 15 OMMA Global (San Francisco) Apr 14 Search Insider Summit (FL) Apr 18 Email Insider Summit (FL)
Recently Concluded Events
Nov 3 OMMA Adnets (NYC) Oct 30 OMMA Video (LA) Oct 29 OMMA Mobile (LA) Oct 29 OMMA Mobile & Video (LA) Sep 23 Creative Media Awards (NYC) Sep 23 The Future Of Media (NYC) Sep 22 Online All Stars (NYC) Sep 21 OMMA Awards (NYC) Sep 21 MediaPost Live at Advertising Week All-Access (NYC) Sep 21 OMMA Global New York (NYC)
All MediaPost/OMMA Events Event Blogging Past Event Videos
Industry Events Calendar
2010 OMMA Agency of the Year 2010 MEDIA Agency of the Year
2009 Creative Media Awards 2009 OMMA Awards 2009 Digital Out-of-Home Awards 2009 Media Agency of the Year 2009 OMMA Agency of the Year
All Awards
Employment Situations Wanted Services Offered Post a Job
Briefs Reports Online
MediaPost Directories
Mobile Insiders Group
People Finder Edit My Profile View My Profile My Contacts My Calendar
HOME • MANAGE SUBSCRIPTIONS • MEDIA KIT
Are User-Generated Web Sites Breaking The Search Engines' Algorithms?
by Tim Daly, Friday, February 2, 2007, 10:00 AM

SHARE

TOOLS

RELATED ARTICLES

MOST READ

Everyone knows by now that user-generated content was the "in" thing of 2006 -- so much so that Time magazine named the user "Person of The Year." Nobody denies that user-generated Web sites have changed the face of the Web today, and will continue to do so in the future.

While Time looked at the trend of consumer acceptance of these new Internet vehicles, their impact has been both positive and negative. Some of their greatest influence lies in their effect on organic search results. No Web site epitomizes this better than Wikipedia. Try this: Go to your favorite search engine, be it Google, MSN, Yahoo, or Ask, and choose several one- or two-word searches at random. Once complete, I'll dare to guess that Wikipedia will have appeared on the first page of results for the majority of those searches.

I hope these results have startled you. Across every search engine today, Wikipedia stands tall as the authority on just about every subject matter underneath the sun. Whether you are in need of the definition of a word, a biography of a famous person, or you simply wish to determine the origin of something, Wikipedia stands out on search engines as a credible authority, regardless of the subject.

Wikipedia deserves kudos all around for an unbelievable business idea and for its rousing success in gaining search engine visibility. Its success at showing up so frequently on all fronts demonstrates the weakness that lies in search engines' algorithms that has perhaps rendered them obsolete. The shortcoming particularly lies in the overreaching importance given to link popularity. Wikipedia shows up in results so consistently due to the millions of internal and external links that it has generated across the Web. While search engine algorithms take this as a vote that the material on the page has substance, this is truly a "subjective" take, which is given "objective" weight in the algorithm. Obviously, Wikipedia cannot be the authority on everything. It simply shows up because it has tens of millions of links that suggest to computer algorithms that it is an expert, without further evaluation of this claim.

The question now lies in what this means to us as searchers. We can easily infer that if we keep getting non-relevant results, our interaction with organic results could plummet. This poses some interesting concerns:

Will search users begin interacting with and relying on sponsored listings for relevancy more frequently?

Will search engines run the risk of user abandonment due to irrelevant search engine query results?

Will the growth of general search engines be stunted, and will we subsequently see a shift to vertical search engines?

The next generation of search has arrived. Wikipedia has pioneered a new trend and will likely be boldly followed into the final frontier. Indeed, 2007 will be an interesting year as we watch how the first-tier search engines deal with this significant risk to their business models. The algorithms have been figured out -- and the emperor suddenly has no clothes.

1 person recommends this article. 

16 comments on "Are User-Generated Web Sites Breaking The Search Engines' Algorithms?"

  1. Caradoc Ehrenhalt from ValueClick
    commented on: February 05, 2007 at 7:00 PM
    Is it any wonder that with pay per click generating so much revenue for the search engines that non-commercial results will start appearing more often? The question is what is the searcher looking for when searching? Today the search engine guesses what the intent is. Will clever search engines eventually let the searcher say if they want results that are informational or commercial? This subject is one that probably keeps the search engines up at night: The balancing act between the searcher's best interest and their own. The engines that place the searcher first will find a way to match what the searcher is really looking for, and will ultimately win.

  2. Bruce Edwards from kmAgent Pty Ltd
    commented on: February 04, 2007 at 7:26 PM
    Let's distinguish between relevance and authority. All the boo-hoo about the authority issue regarding Wikipedia misses the point. The rise of Wikipedia and other social media is a very reasonable measure of its relevance to any search if we take a sufficiently broad view of relevance.

    There must be some real discomfort that a purely non-commercial vehicle like Wikipedia can quickly absorb so much traffic. At this level it disrupts commercial portals and even search itself.

    For example. I'm interested in a new subject. I often go FIRST to Wikipedia to take advantage of the social filtering that grows organically there. Wisdom of crowds notwithstanding, I can apply my own 'filter' and choose what's relevant quickly, including jump-off to pre-screened destinations that have the subject 'authority' I seek. This is tremendously useful to me every day.

    Brute force search just doesn't compete with this.

    Bruce Edwards, CEO bruce.edwards@kmAgent.com

  3. Tim Daly from SendTec
    commented on: February 02, 2007 at 10:28 PM
    I am glad that a lot of people got juiced up about the issue here, as it is important for us all to consider. The articles with MediaPost are restricted to 600 characters or less, so I had to cut some of my editorial.

    For those that think Wikipedia is ALWAYS relevant, the proof is in the data, not our opinions. People in the industry that know understand data is what drives me, not baseless opinion.

    The data I am referring to comes from Hitwise, whose Search Term Analysis tool consistently shows that Wikipedia is NOT as heavily clicked on by users as you may think. Despite its high positions, lower positioned organic and paid search ads get more search result click activity. The data from Hitwise shows that the average search engine user is finding Wikipedia to be less relevant than other listings and choosing not to interact with their listings at an expected rate.

    I encourage all that have a Hitwise license to take a look for themselves, as I found over and over that search results users passed on Wikipedia results and their "click index" was far below typical click rates for listings in those spots.

    We in the industry have different views on how we use the internet and particularly search. Its recommended to look through the eyes of a typical web user, not the sub-segment of search crazies like us.

  4. Dave Lang from TribecaTV
    commented on: February 02, 2007 at 5:58 PM
    Why is Wikipedia a bad inclusion? Must be a rarified search!

  5. Josh Lovison from MediaPost
    commented on: February 02, 2007 at 5:14 PM
    Wasn't there a study suggesting that Wikipedia had less informational errors than traditional print encyolopedias considering the ratio of volume to error?

    I wouldn't consider a library search to be faulty if for a general query it returned an encyclopedic entry giving me more information on a generic term. And if I do a more advanced and complex search on an engine, Wikipedia is no where to be found.

    And generally, when a wikipedia entry comes up for natural search, it does have a good deal of relevant information. There are wikipedia entries that don't have much info, but those don't generate much traffic and don't appear high in search results. And by the nature of wikipedia, highly trafficked pages are quite likely to have quality content by the very nature of their view count.

    I'm not sure where exactly you see the problem with wikipedia's dominance in natural search.

  6. Joe Chiffriller from Chiff Inc
    commented on: February 02, 2007 at 4:46 PM
    I laughed while reading this. Wikipedia comes up as relevant in many searches because it contains relevant material for many search queries.

    This is not an example of the search engine algo being broken.

    Now, when I try a search and come up with sites that offer nothing besides google ads linking to other sites, or a site that has three lines of content peeking out from under rich media ads, then I get peeved, but I have seldom been disappointed at being directed to the information contained in Wikipedia.

    If you think about the search engines delivering information that you are looking for and not as a giant shopping directory, your argument is comical.

    Most Internet users are looking for information - and a search for information on lung cancer should not lead to drug companies, law firms or anyone else trying to sell something. If the search actually turns up a site with good information instead of a site pushing a product - the search engine algo is working...not broken.

    You need to occasionally take your advertiser blinders off in order to understand the market that you serve. You missed the point here.

  7. Charles Ruggiero from SendTec
    commented on: February 02, 2007 at 4:03 PM
    It will be an interesting shake down! I for one love the fact that Google has something to worry about for a change.

    Chuck Ruggiero VP, Marketing eStudentNation

  8. Ken Dickens from 2080Ideas
    commented on: February 02, 2007 at 2:00 PM
    This is nonsense. First, user-generated-media means too many things. Is it Wickipedia? Is it MySpace. These are two different beasts. Second, Google knows that a lot of searchers are just looking for general information on a topic... it's the long tail effect... first I need information, then eventually I want to buy something. If Google includes Wickipedia high in its initial search rankings they have the whole tale covered because searchers are given an immediate choice of a "general information" ranking from a reasonably credible source on every topic, and a commercial choice (ADWORDS) for every credible (paying) buying source. Win-win for Google, and the consumer. That said, it wouldn't surprise me if the Google Algorythm's had something like this in their code:

    if QUERY and WICKEPEDIA TOPIC MATCH then display WICKEPEDIA PAGE @ RANDOMIZE POSITION 1 TO POSITION 5 else GOFISH and display ADWORDS @ topofpage in POSITION 1 to 5

    Ken Dickens 2080ideas.com

  9. Christopher West II from REVShare
    commented on: February 02, 2007 at 12:58 PM
    Your theory holds a lot of truth. I use Wikipedia numerous times, everyday. It not only comes up on the first page of the search, but is typically in the top 3-4 slots.

  10. Jordan Smith from globalcom
    commented on: February 02, 2007 at 12:38 PM
    How can u say wikipedia doesn't deserve top spot?!?! Most people use the internet for INFORMATION. Wikipedia is clear, concise and consistent, no longer does user have to scroll thru 20 webpages trying to find an answer to a simple question. Everyone is always picking on the top dog's of the internet for some reason, despite the spot being deserved. Wikipedia, like Google, does deserve what is has, it might have some flaws, but as of this moment, there's nothing better around.

  11. Sarah Chambers from The ManiaTV Network
    commented on: February 02, 2007 at 12:18 PM
    I think what you also have to consider is that it is so content rich. Not only are there a lot of links (which does that really make them irrelevant), but that there is very little imagery and graphics. The other question is, are the results really that bad? I would much rather see a wikipedia result than another site who happens to get good rankings via blackhat methods which in turn takes me to a page with more digging around to do in order to get the content I seek. I am much more frustrated with folks trying to trick the algorithms than with a site that does have content that relates to the topic that I searched on - at the very least it gives me information. My 2 cents.

  12. Mike Levin from Connors Communications
    commented on: February 02, 2007 at 12:08 PM
    Good point. But this is an issue the engines have been dealing with before Wikipedia. Nothing makes it quite so obvious, but the engines have to fairly represent both the head and the tail of the distribution curve. It can serve up a few picks from the head, such as Wikipedia, but it also has to take a smattering from the usually-under-represented tail.

    The ironic thing is that the ultimate "long tail site" is consistently served up in the head of search.

  13. Amy Smith from Searchles
    commented on: February 02, 2007 at 12:06 PM
    This is precisely why Searchles - a social search platform - is taking a different approach. Rather than relying on the wisdom of the masses, an impersonal algorithm, or an anonymous source, users discover content by tapping into the knowledge of trusted peers and are empowered to determine who influences their discovery efforts, when and how. i.e. search all posts, just your own posts, your friends, your friends' friends, or a particular topical group using those same filters. Or search for an expert and then search from that person's perspective.

  14. Heather Molina from Carat Fusion
    commented on: February 02, 2007 at 11:37 AM
    I diasgree with you. On most of what you wrote. Copmepletely diasgree with you.

  15. Martin Edic from Supernatual Agency
    commented on: February 02, 2007 at 11:35 AM
    There's only one problem with this: Wikipedia doesn't just show up based on links-it has tons of keywords in relevant context and, in most cases, it's going to look like the real thing to the crawlers. Personally, I think this validates the algorithms in that a highly informative page is returned high in the rankings. Expecting an algorithm to differentiate between legit information and sophisticated disinformation is highly unrealistic- most people have a very difficult time doing that!

  16. Alec Campbell from iChannel Strategies
    commented on: February 02, 2007 at 11:24 AM
    This article would have had far greater an impact if you had included some examples of irrelevant wikipedia search results. I've never seen an irrelevant wikipedia search result.

Leave a Comment

You must be signed in to comment. Sign In

Do you have strong opinions and inside knowledge about the topic of this article -- and do you want to share your insights, observations and points of view regularly with the readers of MediaPost? To be considered as a MediaPost contributing writer, please send pertinent info about your credentials, plus several column ideas and one example of your writing on the topic, to pfine@mediapost.com. Please see our editorial guidelines here first.

TIM DALY
  • Daly is president of ClearSight Interactive, a behavioral targeting solutions company. He has over 12 years of interactive marketing experience, having served both on the advertiser and agency while holding various senior management positions with Unreal Marketing, SendTec, Office Depot and Hanover Direct.


AUTHORS

ARCHIVES

RECENT VIDEOS
Recent Search Insider Articles
What's Going To Work? TEAMWORK   
If you have a child in the 18- to 36-month-old range, you may recognize the catchphrase...
Search Insider Sneak Peek: The Three-For-One Keynote   
Avinash Kaushik, Google's Analytics Evangelist, will be kicking off the Search Insider Summit in just two...
Even More On: Everything I Need to Know About Business I Learned From Google   
Today we close out the chapter on business lessons learned from Google. As much as I...
Search Is For The Drills; Social Is For The Holes   
How do people engage with your product or service? Do you sell something like kayaks, which...
Applied Video & Social Search   
So I am sitting around with some friends last weekend watching sports on TV. We get...
Rebranding Myself   
This past Saturday, I married the love of my life. Now begins the process of changing...
SIS Sneak Peek: Looking Backward AND Forward   
In about three weeks, we'll be gathering in Park City, Utah for another Search Insider Summit....
PPC: Commercial Real-Time Search (Almost) Realized    
For all of the focus on crawler and social layers, paid search has largely been ignored...
Finding That One Blue Marble   
In the months straddling 2000-2001, I had the good fortune to lead the ParentsConnected Nationwide Seminar...
The Failure To (Completely) Serve    
At Ad:Tech last week, one message I heard, over and over again, is that people seem...
>> Search Insider Archives 
ABOUT MEDIAPOST • MASTHEAD • MEDIA KIT • RSS FEEDS • PRIVACY/TERMS & CONDITIONS
©2009 MediaPost Communications. All rights reserved.
1140 Broadway, 4th Floor, New York, NY 10001
tel. 212-204-2000, fax 212-204-2038, feedback@mediapost.com