VAB Claims Nielsen Has Been Undercounting TV Usage Since Pandemic Began
- by Wayne Friedman , April 8, 2021
As crucial field agents servicing Nielsen’s TV panel stopped entering homes during the now year-long COVID-19 pandemic, the TV advertising trade group VAB says there has been sharp "undercounting of overall TV use."
“This really should have been avoided,” says Sean Cunningham, president-CEO of the VAB, in a Thursday press conference.VAB members were pushing Nielsen to have agents return to servicing the TV panel three months after the pandemic began. Cunningham: "TV networks were urging Nielsen to get back into homes to protect the sample in the summer [2020]."
Talks between TV networks, VAB and Nielsen have been ongoing since COVID-19 was declared a pandemic, around March 18, 2020 -- about the time
Nielsen suspended in-home field visits.
Nielsen field agents regularly service some 40,000 national TV households that participate in Nielsen People Meter sample.
This involves equipment repair and monitoring people-meter usage, as well as suggesting/replacing TV homes in the sample.
advertisement
advertisement
But when Nielsen field agents were unable to enter homes during the pandemic, the number of households or persons supplying “usable” information -- the so-called “intab”-- dropped from 36,957 average daily respondents in February 2020 to 29,456 in February 2021.
A Nielsen representative says the VAB's numbers are incorrect, that February 2020 respondents were more the 45,000, which then sank to a little over 35,000 in February 2021.
VAB's Cunningham also says there was a unexplained 120% increase in homes "watching no TV of any type" -- rising to 4,642 homes in February 2021 (5.6% of the sample) from 2,735 homes in February 2020 (2.5% of the sample). This data comes from Nielsen NPower.
"Basically empty homes were sitting in the sample.” he says. “The degraded panel led to progressively worse undercounting of TV usage.”
Cunningham says there has been an eye-opening drop in total weekly-reach TV use -- to an 87% average in the first three months of 2021, from 92% for all of 2019. “It’s an unprecedented drop,” he says.
All this comes at a crucial time as TV networks ready their upfront advertising market selling strategy -- in terms of inventory prices and viewer and ratings data estimates -- which begins in two months.
In particular, this has affected Black, Hispanic and Asian TV panel homes, as well as overall large homes in terms of residents, and homes with three or more TV sets.
There was a 23% decline in Hispanic TV panel homes, a 28% decline in Black TV panel homes, and a 14% fall in Asian TV panel homes.
In addition, there has been more data going against current media research trends, especially during the COVID-19 pandemic of the last year.
“Nielsen has streaming declining [emphasis added] among 18-34 in 2020 versus 2019,” says Cunningham. “How could this possibly be true?”
In response, a statement from Nielsen says:
"In early March [2021], we began our return to pre-COVID maintenance protocols and, in concert with local government guidance, resumed in-home field visits when it was safe to do so with the goal of returning to normal as quickly as possible.
While we have always been in the field, our return to in-home visits helps maintain our representative measurement panel and allows us to execute our Nielsen One vision for true, comparable cross-platform metrics.
"We have confidence in the fidelity of our ratings estimates and are focused on the continued quality of our panel."
Nielsen added that on Friday, it would be releasing research looking at both the "integrity of our panel data" and how viewership has shifted during the pandemic.
Nielsen says this research shows "the audience estimates are in line with trends observed via other data sources. We see no evidence to suggest that changes made during Covid to the panel have materially changed the audience estimates as reported."
Wayne, of all of the reasons why Nielsen reported no change in TV usage during the second quarter of 2020 compared to the same period in 2019--- when we all expected a sharp increase--- this is the most plausible . I'm not sure that I buy the "empty home" explaination but it certainly is possible that Nielsen didn't know when its peoplemeter system was malfunctionning in many homes ---which should have been excluded from the panel until the situation was corrected. If, in the absence of fieldmen determining that this was the case, a home was allowed to remain in the panel data base when its set usage could not be determined and thus was counted as not viewing, that, alone, might be sufficient explanation
Now, what do we do about it? One thing that nielsen might consider is tabulating other information it may have regarding TV usage for 2020 relative to 2019---perhaps from its local market surveys or even its PPMs---if these were operational for TV as well as radio. Do these reveal an increase in viewing while the national peoplemeter panel does not?If so, how much of a difference is there?
Wayne (and Ed). This is most interesting. I thought that Nielsen had systems in place that detected when there was a substantial change in viewing data coming from a home (e.g., the home went "dark"). A least this used to be the case. This alone would trigger a check of the home, which might involve a visit but could be as simple as a phone call. Shouldn't this have sent up warning flags, and even led to the removal of non or under-reporting homes from the sample?
Tim is correct, and Nielsen should be flagging and removing unoccupied homes from its sample, replacing these with occupied homes.
I wonder what percent of the resident population left their homes during the height of the pandemic scare. It's certainly possible that this happened to some degree---maybe more so in homes with older residents, who might also be heavier than average viewers, being taken in by younger residents. I also wonder about the degree to which Nielsen is aware when a malfunction takes place---or a set breaks down---or a new set, perhaps a smart set, is acquired and needs to be hooked up to the peoplemeter system---but isn't as the field men haven't been visiting as they usually do. The last point is one that might be checked in a survey of panel members---ask them if they bought a new set---or sets---during the time frame in question and compare their replies with the sets that were being monitored by the system at the time. It might be that 10% of the panel bought a new set but only one tenth of these were hooked up.
Don't need a survey here Ed, just proper sample maintenance. If a home shows zero viewing for a certain amount of time the home is supposed to be flagged and then contacted. There were times when Nielsen checked when sets stayed on one channel for days at a time. Nielsen should actively monitor its data. This is not acceptable.
I must admit to being a little baffled by the reporting of this story. With the inability for engineers to access homes for repairs/fault correction for meter homes, the primary impact should be a reduced in-tab/sample as Jack says. Yes, the panel management software should be picking up if a home has not reported or viewing looks anomalous. Assuming the fall off in the in-tab is consistent across demographic and weight of viewing groupings, one would not expect a reduction in viewng levels. There maybe greater bounce in reporting of specific spots for certain demos as the effective sample size has reduced. The concerns are threefold-fold: 1. are the non-reporting homes biased to ant demographic group? 2. non-reporting homes may have been left in the sample (which would damaged reach). 3. Senior Nielsen executives in the US and overseas have publicly lauded thier engineers and reassured clients of their ability to 100% remotely manage panels....seemingly this was not the case!
While this should be a concerning problem for all sides---time buyers and sellers----the rhetoric blowing in other venues is getting a bit out of control. Taking a look at Nielsen's published findings for the third quarter of 2020 versus the same time in 2019, the average adult TV home resident reduced his/her viewing by 15 minutes per day---or 6%. OK, so this is definitely surprising as I would have expected an increase of, say, 8-15%---not a decline. But even if that had been what was reported, is it correct to assume that viewing levels---even if measured "correctly" ----would continue at the pandemic highs during the 2021-22 season when more and more people are returning to the workplace or to school as well as outdoors activities, generally? Probably not, though the impact of the work- from- home movement has to be considered as this might stimulate more viewing for those involved.
In any event, it's probably best to consider this as a wake up call for Nielsen and its subscribers which will lead to corrective action once it is determined what actually happened and what effect it had. What Nielsen should do, as it gathers the facts, is be as transparent as possible and fix whatever needs fixing promptly---in which case it should be forgiven and we can all move on. What other solution is there?
A question.
Is there an independent industry-based auditor? We have had that in Australia since 1991. There are 4-weekly panel health reports (installed homes, in-tab home counts, panel composition, weighting factors, market-by-market data etc.) provided to the networks and various industry bodies.
And just for clarity, should a home not be reporting (i.e. a fault) it is (or should be) excluded with the relevant weighting factors adjusted to accountg for that. Persistent faults (or non-compliance) should result in removal (difficult to do during the pandemic) and in lieu of removal the home should be de-activated, which reduces the active panel count but is compensated by the increase in the weighting.
And just as a thought-starter, maybe the viewing is being done on unmetered devides (iPads, mobiles etc). In a pandemic the 'occupancy density' increases due to people staying indoors more often, so the potential active viewers at any one time increases maybe to a point that it exceeds the connected device account and viewing is done on a non-metered device. P.S. These are potential reasons - not excuses.
Agreed John. Your summary of the approriate panel treatment was much better described than mine :). In the UK the JIC (BARB), breaks out a separate contract for reviewing panel sizes and composition. I was not aware that there was a limit on the number of devices in a panel home? In my experience all devices in a panel home were registered and could be tracked. Any new device would be notified to the panel manager.
It seems to be about panel composition
...https://www.linkedin.com/feed/update/urn:li:activity:6786259561088196608/
Good points Andy.
The issue is not necessarily measuring the devices, but measuring the actual content on the devices can be problematic (along with the demogarphics).
For example, you may know that someone is watching (say) Netflix, but you don't know which movie or programme. Also, you may detect an active device in a panel home, but not have a registration for the device UID - for example a vistor's 'phone. Such unidentified content tends to end up in the "Other Viewing" bucket.
Some meaningful content and programme ID system would be a great advance. Free-to-air linear does it very well (albeit sometimes with some overnight glitches, subscription TV also does it will, but streamers etc. tends to have their own non-standard systems.
We need some guidance from the MRC. I'm not sure what Nielsen's rules are for temporarily unoccupied HHs- to deal with Nielsen panel homes who moved to a different location due to the pandemic, and whether they should have tried to replace that HH. But of course they couldn't due to the pandemic, so they might have removed those homes from their installed sample.
On the other hand, Nielsen should have reflected some instances of increased short term visitors- to reflect instances where families merged together- married kids moving in with their parents.
John, Netflix doesn't know if anyone was watching just because its content is on a screen. In this regard it has the same "empty room" issues as Nielsen or any onther measurement that is based mainly if not totally on device usage.
Regarding the size of the in-tab sample, of course the decline that is claimed is a signal that something is wrong. However, a smaller reporting base could easily result in inflated set usage and "viewing" results if those who remain in-tab and are supplying data include higher proportions of normally heavy viewers than should be the case.
What is disturbing are reports that various parties were urging Nielsen to make sure that its field men were monitoring all panel homes during the advent of the pandemic---and this advice may not have been heeded. Also curious, is that Nielsen put out its March 2021 "Total Audience Report", which showed the decline in viewing as opposed to the expected rise, as a matter of course and only responded when people took note of the seeminly inexplicable findings and raised a stink. Didn't anyone in the know at Nielsen review this report before it was released?If not, shouldn't this procedure be made mandatory for future reports as well as the ongoing rating findings---like on a weekly basis?.
Nielsen's and MRC's policy is to measure tuning in occupied homes. Unoccupied homes are not selected for Nielsen's panel and should not be included in the sample or viewing estimates unless an occupied household is on a short vacation. If unoccupied homes have gone from 2.5-5.6% there is a problem and as best guess (assuming average tuning viewing before) drop the reported ratings 2.5%. Nielsen reps can use the phone to call these out of spec homes -- those with zero tuning. If those folks say they are away they should be dropped from the denominator in any calculations. A decline of 28% in African American homes, which are normally heavy viewing homes is also a problem, but weighting should make up most, but probably not all, of that shortfall. The data for several months could and should be reprocessed removing unoccupied homes from any calculations. Hundreds of millions of dollars in make goods have been granted because of this, and will affect the numbers until it's addressed. Every day unoccupied homes are included in the estimates is another day costing Nielsen clients millions. This is not a difficult problem to fix.
Andy, I think that LinkedIn comment relates to local DMA ratings and not National ratings. Mind you, fair point made!
Howard, I agree that MRC has a very important role here.
I am not privvy to the US Nielsen system so I can only comment on the AU system run by OzTAM (but utilises Nielsen). With the AU weighting system, if a home becomes vacant it is generally withheld (i.e. its data is not used) and the weighting system takes into account the reduced active panel count so it still projects to the TV universe. In all active homes there is a "Visitor" button - though I can't vouch that all "Visitors" comply 100% of the time. Also, my understanding is that resetting the remote (for adding or removing a button for example) can be done electronically. In AU we conduct longitudinal usage patterns to pick up variations - e.g. holidays vs. non-occupancy, new HH members, HH members leaving, new equipment etc. I would be surprised if the US didn't do the same.
Agreed Ed.
The AU system reviews the usage by person and by minute looking for unusual viewing. For example, very long durations of a programme with absolutely zero changes in the TV set (e.g. zapping around in ad-breaks) signals that this is suspect viewing. If they exceed daily thresholds, that home or person's data is excluded. You'd be surprised how many people leave the TV on for their dogs when they leave the home - that tuning/viewing is excluded unless the dog knows how to push buttons.
Agreed Jack.
It's a tricky issue when to 'withhold' and when to 'remove'. If nationally, unoccupied homes increased from 2.5% to 5.6%, then shouldn't the panel reflect that? That is, process them as zero tuning? But the underlying assumption would be that those people stopped viewing TV completely, which is highy unlikely. If there was not a commensurate increase in 'visitor viewing' then there would be a strong (imperative?) case to withhold them.
While I don't have the US data, the results being reported don't make much sense. I'm at a loss to understand how such data could be correct if similar cross-checks and remedial actions are taken.
John, in the US Nielsen does not incorporate unoccupied housing units in its sample.
Thanks for the clarification Jack.
I was a tad confused by the comment that "If unoccupied homes have gone from 2.5-5.6% there is a problem and as best guess (assuming average tuning viewing before) drop the reported ratings 2.5%." as I misread that as there already being 2.5% unoccupied homes in the panel. My apologies.