Measurement That Makes Sense

Monday and Tuesday I was at the ARF's Audience Measurement 6.0, which used to be a symposium but is now a hash tag (#arfam6). I know at least 50 of you were there too, because I saw you. This is one of my very favorite conferences of the year, because at core, I consider myself to be an audience measurement professional (although, like the ARF's David Marans, I might opt for the larger umbrella term "media research," since the discipline -- as well as the scope of the conference -- is really broader than strictly measuring audiences.)  I won't attempt to run down the highlights, save to give a shout-out to a couple of colleagues, Joan FitzGerald and Alan Vaughn, who presented their best-paper-award-winning "Impact of Online and Television Advertising on Consumer Behavior."   

On Monday, it seemed like the theme of the day was getting GRPs for the Internet, a topic we've explored more than once in this very space.  Then on Tuesday, it seemed as if the theme was, moving beyond GRPs in digital. (When they said things move fast in digital, they weren't kidding!) 

The Making Measurement Make Sense (MMMS) initiative, spearheaded by the IAB but now sponsored too by the 4As and the ANA, has declared as one of its major intents the desire to develop a unifying metric for digital campaign evaluation; ideally one that can put digital on comparable footing with TV.  When I wrote about MMMS in April, I said that I appreciated the opportunity to participate in that industry dialogue.   

Earlier this week, the IAB put out the preliminary findings of Bain's work on MMMS.  Take a look.  This is important stuff, and the progress this initiative has made in articulating a way forward is impressive;  Bain did a great job of ingesting input from across the industry and laying out a construct for moving forward.  

For now, I want to offer some thoughts on the topics that were highlighted at #ARFAM6, and directly addressed by MMMS.

The MMMS principles call for moving to an audience-based currency that moves beyond gross impressions, and that focuses on "...the respective reach and frequency of such exposures."  My colleagues and I at comScore couldn't agree more. 

Gross Rating Points are essentially a known commodity if you can count your impressions against the target audience; and, you know the population size of that target.  So the question of whether or not we should have GRPs online is essentially moot.  We can count impressions served (more on that a little later), and we generally know the target population. Since most of us can perform division and multiply the result by 100, GRPs are thus known a priori.

The question isn't, are GRPs appropriate in digital; rather, the question that emerges is, are GRPs sufficient?  The answer is no. 

In analyzing about 600 hundred online campaigns at comScore covering 70 brands in 17 countries, we have come to understand that, far more so than in other media, it is essential for digital advertisers to disentangle the GRP into its component parts, reach and frequency.  It is possible that the same number of GRPs can be delivered online with wildly divergent mixes of reach and frequency.  It is axiomatic in media planning that desired frequency is between 3 and 10 exposures; three are thought to be required in order to register with the consumer, whereas incremental value is minimal after the tenth exposure. 

We've seen campaigns where, among all the persons who were exposed, almost two-thirds of them were only reached once or twice, while about 14% were reached 10 or more times. Worse, two-thirds of the impressions in the campaign were served to people in that 10+ bucket (in other words, we've seen 14% of the reach absorb two thirds of the impressions.) Looking just at the GRPs in this case might suggest that the campaign delivered as planned; but clearly, when looking at how the impressions actually distributed among consumers, such a campaign leaves much to be desired. One key driver for the differences between planned and actual delivery is the fact that ad server cookies are being deleted by about 30% of all Internet users, at a rate approaching 5 times per month. As a result, the ad server is often not able to distinguish a computer that has received impressions already versus one that hasn't. This causes reach to be under-delivered and frequency over-delivered relative to plan.  

I've written in the past that reach tells us how many, and frequency tells us how much, but that we have to add a third foundational metric: how good?  What was the quality of that impression? How do different ad formats compare in terms of impact and effectiveness? How does the environment in which the ad resides affect the impact of the message?  In short, what actually transpires between consumer and ad during that exposure? All this is sometimes called engagement. 

The very first published principle of MMMS is to move to the standard of a viewable impression.  Again, we couldn't agree more.  There can be a big difference between the number of impressions the server sends, and the number that actually make their way onto the user's screen, fully loaded and "above the fold."  We've observed that at the campaign level, the share of impressions that never load in a viewable section of screen real estate can run as high as 25% or 30%. This will vary based on the nature of the buy; not surprisingly, premium inventory is more likely to be visible, remnant inventory more likely to remain unseen. 

Of course, this is just about exposure; moving from exposure to impact, there are ways to measure effectiveness and ROI.   

MMMS says, "The industry must identify and define the specific metrics most valuable to brand marketers and define and implement reliable standards for existing metrics."  Here too, we agree.  But that's another column.

Before getting to those branding metrics, though, you have to get the messaging in front of consumers.  To do so, the fundamentals of media math haven't really changed; you need to think about how many of your target audience were reached; how the impressions among that reach were distributed, and whether the impressions were ever actually seen.  These three core building blocks of media effectiveness hold true across all media.   

What do you think of MMMS?  Let me know in the comments section.



Editor's Note: Dying to dissect data? We're looking for more Metrics Insiders -- folks who can write clearly about trends/issues and/or provide guidance to marketers who want to use metrics more effectively. To be considered for a Metrics Insider spot, please include your industry credentials, as well as one or two article ideas. Writing samples are great, too. Please email Thanks!

4 comments about "Measurement That Makes Sense".
Check to receive email when comments are posted.
  1. Bruce May from Bizperity, June 17, 2011 at 4:40 p.m.

    If we are going to look at meaningful metrics then we have to go beyond engagement and look at how engagement fulfills the underlying business objectives. That is defined at the business level, not at the media level. Engagement means different things to different businesses. That will always be difficult to capture with a generic metric of any kind.

  2. Nick Drew from Yahoo Canada, June 21, 2011 at 10:30 a.m.

    Another interesting post on the topic Josh, thanks. You're right that GRPs (or at least R&F) are appropriate for the industry in its current state, and it seems that there is agreement that they're not sufficient in their own right. Some metric of 'quality' (if R&F is 'quantity') is definitely in order, but regarding your points about being above or below the fold, you may be missing the mark a little. When a campaign is bought in print, advertisers are buying an *opportunity to see*, and they appreciate that a R/H page, first half of the title etc has a higher opportunity to see; with online placements it's a similar story.
    Actually with our quality metric we probably want to focus on the interaction between the consumer and the ad. A rich media in-banner minigame, for example, inherently has more value than a small gif, which in turn has more value than a non-targeted/ non-specific text ad. From the point of view of the publisher, being able to prove that a richer advertising experience adds value is important; at present, as with impression-based buying, it's mainly taken as received wisdom.
    It's not an easy question to answer, partly because there isn't much that's directly comparable offline, but IMHO it is crucial.

  3. Ned Greenberg from Media and Marketing Analysis Consulting, June 23, 2011 at 11:29 a.m.


    Your crisp commentary is quite helpful. Yes, reach, frequency and quality combine to create effective exposure. With ad-avoidance, I suspect that quality and context of the ad message within and across platforms is more important than ever. The faster we get a handle on the qualitative dimensions, the clearer our media plans and executions will be.

    Thanks so much for your postings.

  4. Francisco Vasquez from Grupo Trebol, June 26, 2011 at 3:31 a.m.

    Interesting article, Josh. I was particularly intrigued by the 3 to 10 exposures to measure advertising to have an impact, to be noticed, to have an effect...or whatever you wanted to describe So what do we tell a client? The effectiveness of a campaign may cost 3 or 10 million dollars? That's when we lose credibility.

Next story loading loading..