The impetus for the media industry’s new JIC (joint industry committee) is at least partly due to Nielsen’s inability to accurately measure TV viewing during the pandemic. For
reasons that seem obvious, Nielsen couldn’t keep sample homes in-tab and the decline in sample caused a reduction in TV viewing estimates.
The problem was related to sample
quality. At the same time, the JIC wants to open its certification to companies that are using less rigorous sample methodologies.
The data that comes in from TV distributors and
manufacturers that comprise the basis for these companies’ measurement are neither a census nor a random sample. Moreover, the companies have yet to even apply for accreditation to the Media
Rating Council (MRC), whose requirements which are tough even for Nielsen.
When you think about it, these facts don’t align.
If the “alt”
companies’ -- which is now what they’re starting to be called -- samples aren’t random or a census, then what are they? If they’re not random or a census, they’re not
representative of a population.
advertisement
advertisement
Their data samples are what some researchers might refer to as “convenience samples.” You get the name comes from the fact that their data
is conveniently available.
The researchers don’t need to create a sample frame, call or visit lots of homes to ask people to participate or send out a lot of surveys. They
simply collect it from TV set-top boxes or smart TVs.
There is no creation of a new sample frame with the data from each new census, no random choosing of census block groups or of
homes on a block group. There are no rules that must be followed to use another home on the same block if a home drops out. You simply get what you get.
This is not to say
convenience samples have no value. The sample sizes are enormous. They are fantastic for certain types of research, just not for traditional TV measurement. At least not as standalone services
unsupported by panels.
Back in the day when ErinMedia and Nielsen were attempting to make TV ratings out of newly available TV set-top box data, there was another startup that
didn’t believe the data was suitable for TV measurement. It instead believed that the best use of the data was for TV ad targeting and media return on investment research. The name of that
company was TRA, which was acquired by TV and is now called TiVo Research.
If you’re not familiar with the company, TRA was started by the legendary Bill Harvey and his
co-founder, Mark Lieberman.
Bill was a long-time Arbitron executive. He is often credited with having invented the area of dominant influence or ADI. He was the first winner of the
Advertising Research Foundation’s (ARF’s) Erwin Ephron Demystification Award. The ARF’s highest honor. He beat out the legendary David Poltrack, CBS’ then Chief Research
Officer, who won it the next year. It wouldn’t be a leap to say that he is the father of TV set-top box ad targeting. There is probably not a more respected researcher in all of media
research.
This isn’t to bash the “alt” research companies, or for that matter, the use of convenience samples. Convenience samples are phenomenal for ad targeting.
For this use, the data doesn’t need to be perfect. What you need is lots of highly granular data that enables the advertiser to send ads to homes with people who are most likely to respond to
the advertising. You’re not measuring anything.
They’re also amazing for gauging return-on-media-investment. As TRA and TiVo Research taught us with their product when it
first landed on the scene, you could double blind match large TV set-top box databases with purchase data from loyalty card programs and know exactly which homes were exposed to ads and which homes
bought the product. This was and is done with no modeling. In research, it’s the Holy Grail – it’s as good as it gets.
The issue is that the JIC isn’t looking
to use the “alt” research companies’ data in this way. It’s positioning it as a Nielsen replacement for currency measurement.
Unless the industry is planning
to change the basis for how it transacts business from cost-per-thousand (CPM) to some other metric based on conversion, convenience samples by themselves are not a good replacement for a proper,
randomly chosen, probability sample.