• by Mark Naples , December 10, 2004

Think about it - are there any major rankings that satisfy more people than not? College and university presidents hire, then fire their communications teams based on the US News & World Report rankings.

Any sports fan reading this probably knows that the team that won the strongest football conference in the country is the only major undefeated team among the top three that won't be playing for the national championship this year. Rankings are the business of displeasing their publics - not pleasing them.

Rankings primarily provoke people who treat them seriously or whose businesses can be impacted by them - that's why the news media loves them. Few other hard news hooks provide an implicit sense of quantification while also providing instant conflict. And conflict, as you may know, is the juice of any good news story.

When a credible source ranks members of a given segment, as JupiterResearch did this week in the search engine marketing (SEM) industry, arms will flail and teeth will gnash. That's just how it is. The main thrust of Jupiter's study was to reveal that the top-tiered SEMs were driving the bulk of spending in search, and to respond to continual questions from clients who pay for Jupiter's services - including some of these SEMs. These clients wanted to know which SEMs were in the top tier and discern how their clients felt about them.

Jupiter examined dozens of SEMs, spoke to hundreds of their clients, and came up with their top tier - 14 firms in all. Then they issued a "Constellation Report," which reviewed and rated these 14 across the criteria that many of Jupiter's clients had been asking about: market suitability, overall business value-adding, and size.

Jupiter made it clear in this report that these rankings were only to be used in consultation with Jupiter - essentially, as a lead-in to working toward a consultative business solution, which is - after all - what Jupiter is paid to provide.

But, what resulted was something of a firestorm. Why would anyone be surprised? It's a ranking, after all. And rankings of any kind will always provoke.

"There's been a lot of discussion about our report this week, but a lot of that discussion misses the nuance of this study," said Nate Elliott, the Jupiter analyst who conducted and published the survey and rankings.

"These 14 agencies are among the largest and best-respected in the space," he noted. "Just being included here is a good thing. But beyond that, we wanted to give our subscribers some guidance as to which agencies excel at which parts of the business. And we expect that anyone reading the report will talk to us about which agencies might meet their specific needs. We're trying to help our clients create a consideration set when they're choosing an agency, and this report is just one part of that conversation."

I asked Nate how he answered critics who decry any conflict of interest implicit in ranking companies, of whom, some pay Jupiter's subscription fees and some don't. "Yes, some of the companies we reviewed are clients. But most of them aren't," said Elliott. "And I can tell you that several non-clients did extremely well in the report, while several of our clients didn't do as well. Whether or not someone's a client never goes into my thinking when I'm writing."

Critics of the NCAA football ranking system, which is run by something called the Bowl Championship Series (BCS), think that the C should be removed from their acronym, and a playoff should be instituted just like in other sports. Such a merit-based system seems more the American way, no?

The corollary for any SEM rankings would be to institute a ranking like that done by Media magazine or other books, on the top 25 firms by revenue, with a mention of primary clients and services. I've worked for a firm that was in the top 14, but not singled out for favorable mention in the survey results. So, I know that the folks at that firm have been wondering how they could have doubled their client base and tripled their revenue this year, if they hadn't been pleasing these same clients in multiple ways.

"The truth of this study is all in the nuance," said Elliott. "None of these agencies are bad at what they do. What this study found is that some are better than others."

In other words, just like choosing a college for your kid or picking a winner in the BCS - you have to play it out. The college decision can only be made after multiple visits and discussions with experienced individuals, and the football title will only be winnowed after some more games are played. Neither is a perfect science, nor claims to be.

So, just like choosing any other kind of vendor, pick from among a few of the best, be sure to check the references and empirical work product they provide, and do your own research too. Don't be afraid to run a test campaign or two. You wouldn't choose your child's college based on the U.S. News Rankings, and nobody is handing USC or Oklahoma the Sears Trophy as national champion today. Rankings only provide a beginning.

The problem arrives when the people who regard them think of them as an end.