Commentary

Reliability Vs. Validity

1. Ed Papazian from Media Dynamics Inc, May 1, 2023 at 10:05 a.m.

   Excellent article. Too many people confuse the two terms ---stability and validity. They also assume ---incorrectly---that the larger the sample the more accurate its findings will be. This is based on the  barrage of misinformation we get from statisticians and researchers as well as TV news which cites the results of poll after poll and as a matter of cautioning the viewers keeps telling them which findings fall within the "error" margin---based on sample size calculations which do nothing more than estimate the odds of getting the same answer if the same study was replicated. This has nothing to do with accuracy---only stability---but almost nobody, incuding many in the ad or media business, seems to understand this distinction. Folks---you can't determine a survey's accuracy by knowing its sample size. Period.
   
   As for TV audience measurement, what Nielsen has on a national level is a collection of panels---some with people meters; some without----which record set activity electronically. Being a panel---or collection of panels--- members drop out ---or are terminated---for a variety of reasons and even if replacements are sought in a  scientific manner---or even to replace those lost by new homes with similar characteristics---it remains questinable as to whether the resulting panel truly  representats the whole nation at any given point in time. But it's all we have and, in my opinion, it's far better than many of the alternatives being proposed as replacements---using big data ACR or STB panels.
   
   Stability and sample size are another issue---but that's largely a question of cost and whether the big audiece sellers are willing to pay substantially more just so the very  small audience sellers---who can't afford the added sample size---get more stable ratings. Why would they?
   
   The real tragedy is that we have forgotten what should be the real question---"What do we mean by audience"?What we are going to get will be basiclly a digital-style "audience" measurement--based on device usage and the assumption that the "viewer"---or  owner of the device---"watched" every bit of program and commercial content---so long as it remained on-screen. Which is nonsense and produces a hugely inflated picture of the ad reach as well as the frequency attained by most advertisers as well as misleading them about who was"watching" their commercials. But this is what then sellers want and as they do most of the funding while advertisers pay nothing, this is what we will get.

   Reply
John Grono from GAP Research, May 1, 2023 at 8:35 p.m.
Good article by the pair of Eds.

No matter what you do, you won't have 'THE truth.   Taking AU as an example, we have compulsory voting and compulsory completion of our quinquennial population census.

We're a pretty compliant mob generally getting around 95% compliance.   But does that mean that the population census, or even the election results are 100% correct and are 'the truth'?   No it doesn't, but in all probability it does no damage to the actual truth should it be achievable.

The US MRC is basically doing the same thing ... ensuring that the ratings systems is ensuring compliance to the acceptable rules as defined by the JIC.

Ed, while I agree that defining audience to include 'in-the-room' as adequate is wrong.   But the TV rating is not 100% based on "watched every bit of the program and commercial content so long as it remained on the screen".   In AU if a 'person' (well their button on the ratings remote) has not been active within thh prescribed time-frame (and I can't recall whayt it is) they are assumed that they have left the 'TV Room' and all 'assumed viewing' is removed.   Also, when you see an 'audience' to a program it is based on average minute (here in AU) and not on 'any assumed presence at any time during the program duration'.   But I certainly do take your point.   Just maybe if advertisers contributed financially we might be able to include 'attention' metrics.
Reply
Ed Papazian from Media Dynamics Inc, May 2, 2023 at 3:28 a.m.
John, the problem with the assumption that we are getting an indication of the program "viewer" being in the room---if not actually attentive---is that we aren't getting even that. Although the panel members---or household head---are instructed to press their buttons if they "stop viewing"---which can include leaving the room-----and there are periodic on-screen prompts to remind them of this need, few actually bother. Accordingly you may get 2-3% pressing their button per commercial break as your average result when the real figure ---per TVision and other observational studies---is 30-35% for commercials.

We have to remember that an average Nielsen panel member is counted as "watching" 4.5-5 hours of TV---'linear" and streaming---per day and is in the panel for years on end. That's a lot of viewing---many thousands of occasions----and it's simply unrealistic to expect most panel members to obey instructions about telling the system everytime they leave the room, then reaffirm their "viewing" when they return. Most of the time they just don't do it.

Before TVision came to the UK, it was assumed that their people meter system was findng a loss of only 2-3% per commercial due to not being present, as this is what the rating system was reporting---but when TVision finally set up a UK panel, they found exacty the same thing as we have long noted inthe U.S.---about a third of the presumed "audience" wasn't really there. I believe that the same result would obtain in Australia.
Reply

CommentRead more comments >

Next story loading