• by David L. Smith , August 12, 2008

With the introduction of more and more new information sets characterized as media research (Google Ad Planner) and other data sets that people are using as surrogates for research (log files/Google Analytics), it seems an appropriate time to talk about research and proper use of same.

Back in the "old days," media planners and buyers were trained in a number of different media and they had a perspective about the pros and cons of various media research sources. For example, while MRI reports on audiences for TV, you would not use it to estimate TV ratings. Why? MRI does not measure average program but whether you watch a show at all -- and how often you do. A very different thing than whether you watched the show specifically last night, which is what Nielsen measures.

Also, you would not take a comScore number from one site and compare it to a NetRatings number from another site. While they seem to be similar, take a look at any group of sites for either vendor and compare them to the other vendor, and you will see that the differences are significant.

That's why it seems strange to me that media people and clients often look at media data as absolutes. They are not. They are only measures of what the research source was trying to measure, with the limitation being the methodologies and best practices being employed

In line with the above, when a site says that they have a certain number of visitors or unique visitors a month, it is important to ask "according to whom?" Make sure that the data you are getting is always comparable to data sets you already have. If you are going to use a new data set, don't just take the data in for the publisher in question. Make sure that you also look at a similar cut of the data for sites you are already familiar with, and give the data a logic check.

Not that your due diligence should stop there. It is important that your resources disclose their methodologies so that you can make an educated decision as to whether they have done a complete job in gathering and/or projecting the data. Also, I hope you will join me in asking all sources based on sampling to reveal their respondent counts for any cross-tabs that are run. This again will help you in making an educated decision on the use of the data. You are right to ask what a source is trying to hide if their methodology is not transparent and/or their data is not obviously backed by a high enough respondent count.

Proprietary research is fine as far as it goes. In many cases, it is not comparable to other sets of research except for similar proprietary research. For example Erdos & Morgan does a lot of subscriber studies for magazines. It might be possible to compare one of these to another of these. But you cannot compare a subscriber study of a magazine to the MRI data of another without some serious machinations via prototyping.

Lastly, the majors (comScore and NetRatings), as well as one newbie (Quantcast), have at least started the auditing and certification process with the MRC. That should also be on your list, to ensure that down the road we are getting data that actually measures what the research source purports to measure, and that industry standards are being adhered to.