January 25, 2023
Commentary
Why Nielsen Is Forced To Model Even Though It Is Bad Research
- by Steve Sternberg , Featured Columnist, July 14, 2015
As viewing options and devices grow, samples become less capable of measuring them. The whole purpose of Nielsen’s national TV sample is that you and your cohorts watch the same thing, based on a dozen or so categories that Nielsen has determined impact TV viewing.
This worked quite well through the 1990s, when the average household could only receive about 30 or so channels and could only watch TV content on a television set. If I watched Seinfeld, and came into the office talking about being the master of my domain, virtually everyone (at least everyone Nielsen said was just like me) knew exactly what I was talking about.
advertisement
advertisement
Today, I recently noted to some of my friends that the woman in the new Terminator movie was one of the stars of Game of Thrones. None of them watch that show and didn’t know who she was. These days, simply because of the number of viewing options available, examples of you and your “cohorts” watching different things (or the same things at different times) is commonplace.
The problem with modeling is that it's not good research for measuring viewing behavior, no matter how good the modelers. And even though Nielsen does have some of the very best researchers, calling them “scientists” doesn’t make them better.
Should we accept that modeling viewing behavior from its national 25.000 people-meter sample to 13,000 local TV set meters will effectively double the national sample because Nielsen tells us the math works out? It’s not quite just making up numbers, but it’s close.
You can model overall reach of a network or group of networks, but not individual average program ratings. In other words, overall viewing is much more predictable, specific viewing much less so.
When I was head of research at ION, we would often ask Nielsen how it was possible that an hourlong show like “Criminal Minds” can possibly lose half of its adult 25-54 audience in the second half-hour and gain 40% among adults 18-24? The next day it might be the exact opposite.
They would diligently analyze the data and then tell us that 10 people in the sample switched to something else, and that resulted in a reported rating decline of 48%. This is something not unique to ION. If you really scrutinize the daily cable ratings of virtually any cable network, you’ll see many “illogical” fluctuations. If I were at a cable network now, I would favor almost anything that would significantly increase sample sizes.
I was one of the founding members of the Council for Research Excellence, which should be advising Nielsen on how to proceed here. This is the main purpose of the CRE. To quote from its website:
“Methodological research is concerned with the accuracy of audience measurements and the effects of possible changes in methods. It provides the foundation for valid, reliable and credible audience measurement. The Council for Research Excellence is intended to give Nielsen's client base greater voice in the design and execution of methodological research.”
I haven’t heard anything from the CRE on this yet, but a POV distributed to the industry would be a great idea. CRE members are from all sides of the industry, and its seal of approval would give Nielsen’s plan credibility, which, no matter how much “impact data” it releases, will otherwise be lacking.16 comments about "Why Nielsen Is Forced To Model Even Though It Is Bad Research".
Read more comments >
Next story loading 
Agree, Steve. I suspect that the desire to have numbers---any numbers---for every single minute of every telecast on every platform will eventually trump common sense and your ION experience will be repeated ad nauseum from now on.
The solution is obvious. Show data only when it can be presented with some degree of confidence. In other words, Nielsen should do what any good researcher would do and refuse to provide "granular" data for everything. For example, if cable channel X attains an average minute audience of 20,000, which is too small for reliable reporting, refuse to supply such data on a show by show basis, but only for a schedule of shows on Channel X on a cummulative basis. Sure, you can supply total GRPs so the CPM crunchers can do their calculations, but not for any show or telecast that has too small a sample.
A long time ago, Simmons and MRI supplied their respondent by respondent data to agency and media subscribers only to find buying decisions being made on "cells" with 3-6 viewing or reading respondents for contending media, which, of course, produced wildly unstable findings and many unjustified complaints about the accuracy of the studies.
One way to avoid this problem is to "smooth" the data by various "simulation" or "ascription" methods, in an attempt to supply granular data. I assume that Nielsen is heading down the "we must have data---any data" road, rather than drawing the line and saying, "if you guys want granular data on everything, it's going to cost you for a much larger, better sample". And Nielsen may well be correct in believing that many of its subscribers will refuse to pay for better research.
It's a sign of the times. Data rules.
I agree Ed. When I was at Magna, it was usually the agency researchers who had the most influence on what research mthodology was used. 10 years ago, Nielsen would have been laughed out of the room for proposing modeling demos (of course, 10 years ago they never would have considered it). I recall six or seven years ago at a national Nielsen client meeting, someone from Nielsen got up on stage and actually said that validating research was no longer its number one priority, rather getting products to market was. That said, if I was at a cable network now, I'd be in favor of anything that increases sample sizes (I'd probably be against it if I worked at a broadcast network).
I am sure John Grono from GAP Research will chime in but he is in Australia and asleep! I am guessing that he would agree that this is yet another argument for a TV JIC in the US (unequivocally quite legal in the US and long long overdue) and will repeat the fundmental question, when are "we" going to take control of our ratings measurement, its quality and cost levels? Of note the CRE has essentially been a self serving PR machine for Nielsen that apparently provides some solace to those involved on a strictly exclusive basis in terms of any research findings. In terms of an industry I love and have spent my entire career in, sadly it may be time to say, shut-up - you are getting what you deserve!
Unfortunately, I agree with you abot the CRE to a degree. At one time it was a great organization that truly cared mostly about producing good research and requiring the same of Nielsen (as a goal at least). We shuld keep in mind that the CRE was formed primarily to pre-empt government intervention in Nielsen's measurment processes. This seems like the perfect opportinity for the CRE to re-assert itself and live up to its original independent mission.
Excuse the typos please.
Excellent points and CRE did indeed accomplish some valuable 'research on research'. However, even in "re-asserting itself" how can the CRE ever be meaningfully independent when it is purely funded by and for the exclusive use of Nielsen? Government pre-emption on TV measurement would have been just the ticket for our industry in my opinion. So in hind sight we should ask, "Have those accepting an invitation to the CRE done the industry a huge disservice?" Its a rhetorical question!
On your solid point regarding sample sizes and broadcast network versus cable positions:
JICs are always mandated to do everything possible to provide fair and balanced audience ratings across all vehicles. In certain countries smaller audience vehicles can, within strict controls and overall approval, supplement the investment in the universal research approach via added samples, etc. to improve the rigour of their ratings.
The Visigoths are on the hill.
Dear Steve,
I trust you had nothing to do with this story's headline. It's obscene!
I respect you too much to believe it captures your real thoughts and intentions.
Like "The Decision to go to War in Iraq"(nsarchive.gwu.edu/NSAEBB/NSAEBB80/wmd34.pdf)
every one knows that no one is forced to do bad things unless one is the victim of overwhelming evil that effectively robs one of the capacity to choose and eliminates all good alternatives.
Hence, Bush, Cheney, Rumsfeld & Blair are not profiles in courage against Saddam Hussein, but examples of what happens when the lack of conscience meets pride, ignorance, lust and greed. Nielsen has better alternatives in this situation than they allow anyone to believe. And the benefits Nielsen claims are de minimis for Clients.
Nielsen only seeks its own benefit. They seem only to want a way to earn more by doing less. If they could fold in STB's, they'd be all set and knock out another competitor, which apparently has been their unchallenged stock in trade for decades.
As for the CRE, it should never have been started in the first place. It was a Faustian bargain brought about by the broadcast networks abdication of their CONTAM responsibilities proposed by Congress after the Harris Hearings. (And the rest of the our cowardly industry was MIA.)
Finally, Steve there are 2 profound errors in your piece.
One: The "whole purpose of Nielsen's national TV sample" is not what you wrote. Do over!
You know more and better, but I will not write it for you or Nielsen.
Two: All CRE Research is funded by Nielsen the last time I checked.
So, developing a CRE POV on NPX VAP is tantamount to calling "masturbation" love-making.
No Compass. No True North. No True Love.
In sum, the industry's last collective hopes are the real data and methodological scientists at E&Y and the MRC, as well as the "better angels of our nature."
Nielsen doesn't have to do bad things. Not now. Not ever.
Peace,
Nick
Yes Tony I was dropping some ZZZs. I will need to respond in two parts.
Modelling is a tricky one. For example, you could think of weighting a sample to a population is a type of modelling. That is, because it relies on an inherent assumption that the people in the sample have the same characteristics of those in the remainder of the population.
Enumeration generally relies on a two-stage process. First, there is an 'establishment survey' to enumerate data that is not available from a complete enumeration (e.g. a census). Second is then the sample or panel from which the empirical usage data is collected which is then extrapolated back to the 'known' population. Efficiencies in this process (such as stratification based on CHAID determinants of usage) can reduce the overall error rates.
But, this process relies on two major assumptions. First, that the establishment survey is robust. Back in the days when you could get a 75% response rate to an ES that was a pretty fair assumption. Now an ES rate of 25% is considered good. Second is the respondent selection bias. We have growing evidence that (especially with media consumption) that there are respondent propensity biases. We are seeing this in some readership studies done here. In round terms a person who 'likes' reading a newspaper or magazine is more likely to agree to participate. I am not a liberty to reveal the amount - but I can say it is enough to utilise 'propensity weighting', and that it is growing. I suggest the same is probably true for television (and indeed) any medium.
Having problems copy/pasting the second part. I was able to salvage what I typed into Word, but MediaPost won't allow me to post it as it adds 000s of characters of HTML code.
Nick, buddy, lighten up. There are space restrictions here. Obscene? Lol.
Relying totally on respondent level data in order to preserve granularity can have the undesired effect of relying on 'perfect data' from an 'imperfect subset'. My contention is that the subsets are becoming less and less representative in the increasingly complex media landscape, and that this too must also be addressed.
A lot of this came to light reconciling online panel usage data and online server traffic data (think RPD for TV). To cut a very long story short, we were able to establish that there are 'pockets' of usage that we can't empanel for tethered device usage - schools, universities, airline lounges, public places etc. Metering fixed devices in such places, while it could pick up the volume, could never pick up the demographic composition. In essence we had pockets of the universe that we had known traffic data on (by site) by no audience data. This is where the modelling came in. Again I am not at liberty to say how it was done, but simple re-weighting did not work, plus we had to preserve 'granular' data.
So, the question is, when you have evidence that you are NOT reporting the total audience do you say 'bad luck, can't measure it', or do you combine the existing projected panel data with modelled data that 'back-fills' a known quantum of usage. Of course I realise that the modelling assumes that the audience characteristics of the non-measured population are the same as for the measured population, but I can also assure you that the missing proportion was a lot lower.
This does not mean that I am in favour of modelling as a panacea or some form of research short-cut - heaven help us if that ensues. But I am also not in favour of ignoring audiences that don't fit in with the existing measurement paradigm. The more audiences fragment, the more we need to be flexible and become research bower-birds building a better nest.
If anyone else has a long post and wants to Copy/Paste from Word and you can't here's how you do it:
Dear Steve,
I've always meant every good thing I've said about you.
And if you want more deserved praise you can always get it here.
That said, I like you, have worked too long & hard at this practice to suffer the ignorant, foolish and lazy that appear to be almost in complete control of the circus now.
When that day arrives, I'm done.
Until then, I shall remain committed to doing "the right thing and doing things right" (Drucker). People like you deserve my best. Ed Papazian hasn't thrown in the towel
and he has more right than me.
I cannot accept rotten apples or tomatoes. And they spoil the rest of the basket.
Nielsen is selling rotting fruit at a premium price. That is neither right nor fair.
I never have and never will make excuses for negligence. That's above my pay grade.
Thank you for acknowledging me tonight. Please know that
you have spared the world my reply to the Australian Wizard of Oz
and his Canadian pen pal. Just kidding, Tony.
0ops. Need to drop some ZZZ's.
Peace,
Nick ... From Cut & Paste, Incorporated
The point, raised above, about the bias caused by allowing too many heavy users of a medium into a sample or panel measuring that medium, is one I have also raised before. It seems that this is invariably overlooked in sample/panel selection as the would be respondents are told, in many cases, that this is a TV or magazine survey or this is obvious by the way the proceedings are conducted. As a result, chronic light/non users tend not to cooperate, which has the effect of tilting the sample in the other direction--towards heavy users. This, in turn, affects the findings, inflating them to an unknown extent.
I wonder whether the MRC includes this factor in its evaluations or, if not, whether there's a way to alter the recruiting process---by disguising the purpose of the survey----as a way to deal with this issue. It seems to me that alternative ways to recruit samples and the effects regarding heavy/light user biases is an easy thing to test.
Exactly Ed. I don't have TV factors but with readership we've found that we are at the stage where we simply have to make adjustments.
We disguise the purpose of the survey (apart from that it is about magazine and newspaper reading) and ask 'propensity' questions right upfront. We are then able to quantify the propensity of people to start and complete the survey relative to their (claimed) upfront weight of reading (not by genre or title). This allows us to downweight the projected results because we basically know how many non-readers and light readers opt out. We don't alter the contact and recruitment process apart from the above.
From the data I am seeing (albeit not TV), the potential/actual distortion of the projected data is very much a real concern, yet the system would pass every research audit.