January 25, 2023
Commentary
Garbage In, Garbage Out
- by Joe Mandese @mp_joemandese, April 14, 2022
It’s one of the most fundamental principles of computer programming and data science, but based on some descriptions for the “test and learn” marketplace onboarding multiple, “alternative” currencies that advertising deals will be based on in the 2022-23 upfront marketplace, I think it is probably a good principle to keep in mind when it comes to media research -- especially audience estimates.
But that’s not what I heard during the opening session on day two of the Advertising Research Foundation’s (ARF’s) Audiencexscience conference, which after listening to Paramount execs explaining it, maybe should be renamed Audiencexprocess.
“We’re at this phase right now where, you know, the methodologies, I think have been roughly understood and accepted, and now it’s a question of operationalization," Paramount COO of Advertising Revenue John Halley explained during the day's opening keynote, "Perspectives On Measurement And Why It Matters."
advertisement
advertisement
Apparently, it doesn't, because Halley and Paramount Chief Research Officer Colleen Fahey Rush did not actually discuss any measurement issues, but focused instead on "process" and "operationalization" during the keynote, which included no one from an advertiser, an agency or a neutral industry research authority, but was moderated by an Ad Age reporter.
"What we're trying to do is make the point that alternative currency is not just a question of research and methodology," Halley asserted, "It's about operationalizing it."
I've spent the much of the past couple of months focusing on another supplier's similar claims -- NBCUniversal's -- mainly because that company has been incredibly transparent about its process and ambitions in pushing through its alternative currency supply chain, "certifying" its preferred audience metrics suppliers, and lobbying against Nielsen's long-standing role as the industry's "currency."
Personally, I've never understood the bum rush associated with the supply-side's aggressiveness, and why after decades of covering a thoughtful, methodical industry-wide approach to currency change -- you know, things like year-over-year comparative data before pushing the buttons or pulling the plugs -- why there is such urgency.
And yes, I've heard from and even quoted some smart people from the demand-side -- mainly agency execs who come largely from digital native backgrounds and say the reasons is that the industry's culture has shifted, and that they no longer have time for the kind of methodological consideration they've used in the past before modifying the audience measurement estimates that are the basis for billions of dollars in their client's ad spending.
I mean, I remember the conniptions the ad industry went through when Nielsen finally shifted to people meters from antiquated set meters and paper diaries for network TV ratings, and the Big 4 nets were estimated to lose $40 million in ratings shortfalls. And that was after years of methodological testing, and oodles of comparative data testing before that change was even implemented.
I also find it remarkable, that not one of the alternative currency suppliers -- neither NBCU's certified supplier, iSpot, nor Paramount's VideoAmp -- are currently accredited by the Media Rating Council, though at least iSpot is currently undergoing an audit.
I've never understood why the U.S. advertising industry doesn't operate under the same kind of joint industry committee (JIC) protocols used by their counterparts in other countries to ensure fair and equitable research inputs and outputs -- or why the demand-side in the U.S. always ends up capitulating to its suppliers on what their advertising buys are measured and accounted on. But hey, I've only covered this marketplace for 40 years, so what do I know?
I also don't understand why people representing billions of dollars in advertising spending just roll over when one of their sellers flippantly says the methodologies of the data they will input have been "roughly understood and accepted" and are actually ready for prime-time operationalization.
Whatever happened to garbage in, garbage out?
Lastly, there's another subplot in the 2022-23 alternative currency bum's rush I've never quite understood, but it was touched on by Paramount's Rush, which is why a marketplace comprised of multiple alternative currencies is actually a good thing for advertisers paying billions of dollars based on them?
“Every marketer has different needs, which is why I think it’s so necessary for us to be forging this much more innovative present in the future, where there is this array of choices of different data sets that can be harnessed to meet the advertisers’ needs," she asserted, adding, ironically, "That is sort of our North Star right now."
I say ironically, because having multiple, alternative metrics competing as an industry currency literally is the opposite meaning of a North Star, which is a universal point of reference.
I do understand the perfect storm of events that got us to this point, especially that Nielsen was its own worst enemy by dragging its feet and not moving faster to reboot its own antiquated measurement methodology and why other challengers hadn't accelerated their own vetting process to at least become MRC certified -- er, I mean accredited -- instead of by media suppliers.
But this is where we are. Those are the inputs. And it will certainly be interesting to cover their outputs.
But I do find the argument that having multiple potential currencies competing for dominance -- if not actual meaning -- side-by-side is a good thing to be incredibly stupid.
I mean, does anyone remember the days when buyers and sellers would haggle over not just the price of their advertising deals, but the currency values of side-by-side TV ratings from Arbitron and Nielsen, or print audience estimates from MRI and Simmons?
It seemed pretty inefficient back then, and I'm going to go out on a ledge and say it's not going to provide any more transparency today.
But maybe that's the real reason why the demand-side has been so complacent? Could it be that it doesn't actually like being held accountable to industry-standard metrics that everyone can see and agree on?
Whatever the reasons, it's about to be operationalized by the biggest media suppliers in the world.
"As much as it is a question of research methodology, it's actually more an exercise in understanding the workflow associated with doing things under a different denomination," Paramount's Halley explained. "It's about operationalizing it."
Joe Mandese is the Editor in Chief of MediaPost. You can reach Joe at joe@mediapost.com.
Interesting perspective, Joe. As you know I came from the era when the major ad agencies were right on top of methodological questions and were pushing ,first for viewing not set usage data and later for demos and product usage information for national TV shows. In fact, I was one of the pioneers in that movement along with some fine colleagues. All of us were primarily "we" oriented, meaning that we wanted the best measures for our industry, not just one agency trying to look good at the expsnse of its rivals.
While we made lots of progress on the magazine, radio and local TV fronts, we were stymied when it came to network TV buys as the buyers and sellers rejected virtually all efforts to employ more brand-relevant metrics and as the upfront process became entrenched, the buyers more or less went their separate way, agreeing with the sellers on a single, broad based "demo" as the currency for each upfront corporate negotiation. So that's where adults 18-49 and women 25-54 came from with many clients thinking that the use of such "curriencies" was a big improvement over household ratings---which it was---but why stop there?Yet this is exactly what happened and even now, in 2022, we are still using the same currencies as if they meant something.
To be fair to the "demand side" we should note that it's thinking is focused primarily on how the ad dollars are split between various sellers---and at what CPM--- not whether the audience surveys are right or wrong. In other words, even if we don't really know who is watching, so long as the research we use isn't greatly misleading us as regards share of audience, then we are making valid buys. If and when it can be demonstrated that national TV's rating currency is greatly distorted regarding how the aidience pie ---whatever it may be ---is split up differently than we have been led to believe, that's quite a different matter.
So far, none of the various competitors to Nielsen's national TV rating service over the years ----Arbitron, Burke, Audits & Surveys, R.D. Percy, AGB, SMART, etc.--- have been able to show that its share of auidience split is far off the mark. In fact, none of them even tried to make this critical point---so they all failed to win support.
Bravo Joe, and plus one Ed.
I've worked on multiple JICs in AU and I agree that "we" is the stongest requirement and most needed component.
I've had to push the media owners to invest in new methodologies, new data sources, new technology etc. (which all comes at a cost) because if they don't the media agencies would probably drop the research expensecompletely.
I've also had to reel in the media agencies/marketers expectations to be more realistic about ther outcomes they want .. unless they want to pay a ship-load of money towards the research via the advertising rates.
It feels like we're heading to a "tell me the answer you want, and I can get the research to show that". (Yes, that was the first research thing I learned at university on how you could produce a pre-ordained answer. We had to statistically prove that every husband is, or has been, a wife basher - which clearly is not true - but yes we could get a 100% wife-basher resukt by the way we wrote the questionnaire.).
We call that the Hot Chocolate approach when there is no currency ... Every One's A Winner, Baby ... but unfortunately it's not the truth.
Once again, Joe has the best 20-20 vision and insight. I was wondering when someone was going to remember when the presence of even two measurement companies drove Poltrack to observe that "one of them is right, or the other one is right, but they can't both be right!" It will indeed be an interesting time.
Jack, the only real comparisone between rival TV rating services occured at the local market level between Arbitron and Nielsen. Here we had very small samples and a mixture of methodologies---diaries only in most markets plus meters and diaries in some of the larger cities. While individual stations lived or died based on misicule rating differences between the two services, most of these---especially for independent stations---- were noted in markets where one service used meters plus diaries---the latter for viewer-per-set projections---- while the other used diaries entirely. So they weren't employing the same methodology and that plus the small sample sizes explained the differences.
However, in most cases where both services measured audiences in an identical manner the results were very similar---a few exceptions notwithstanding. So Dave was right---- under comparable circumstances they were both "right". Which is why one of them eventually lost out. There was no need for two services supplying essentially the same information. If they had produced really significant differences---that would have been a different matter---but it never got to that point.
Having been on the sales side, the differences even where methods were quite similar, would be troublesome enough to be vexing. Plus, agencies and advertisers declared themselves to be Arbitron or Nielsen "shops" so sellers had to keep two sets of books. It was hard enough when two suppliers had similar but not the same numbers. This time it's going to be more challenging. Way more.