Ad Effectiveness Methodologies: Comparing 'Recall' With 'Opportunity To See'

• Comment (4)
• Recommend (4)
• Tweet
• 

The Evolution of Advertising Effectiveness Research

Over the past decade, cross-media research has become increasingly important to advertisers.  Today's marketers utilize multiple media channels to reach their target audience, and advertising research methodologies have also evolved to compare those channels on their ability to educate and persuade.  InsightExpress recently examined two popular methodologies to determine if and how their results differ.

Dueling Approaches

The first methodology is based on consumer recall of advertising. After a campaign launches, a respondent is asked a series of survey questions where one shows an advertisement that is part of the campaign being measured.  Those who recall seeing the ad are classified as “exposed,” while those who do not are considered “unexposed.”  Comparing the two groups leads to a statistical determination of the effectiveness of the campaign at changing awareness and perceptions towards the brand.

The second methodology is based on the “opportunity to see” an ad  (OTS).  Here, respondents are asked about recent media consumption habits including specific TV shows, channels, and magazines.  In the digital world, cookie data is collected on advertisements sent to respondents’ browsers.  This media consumption data is compared to the campaign's media plan or a TV post buy report to determine who had the opportunity to see the advertisements.  Respondents who consume the “right” media but not at the specific time that the advertisements ran are assigned to the “unexposed” or “control” group to provide a baseline for comparison.  As with recall, the two groups are statistically compared to determine if any changes in attitudinal measures resulted from advertising exposure.

The Contest: Recall vs. Opportunity To See

To understand the benefits and limitations of these two approaches, we examined a random sampling of studies that contained the ad recall questions and the OTS questions.  They ranged in complexity and included a single site online advertising effectiveness study, a multi-site online advertising effectiveness study, and a multi-channel cross-media study.  Results were calculated using the OTS and recall methodologies, with data averaged across the various studies to determine how the results differed.

The data showed considerable differences between the two methodologies.  The OTS aggregation revealed that, on average, the campaigns were moderately effective at boosting two of the exchange of information measures: awareness of advertising for the brand (an increase of 2 percentage points), and association between either messaging in the creative or sponsorship elements with the brand (an increase of 5 percentage points).

The recall-based approach led to very large increases in every measure, such as an increase of 10 percentage points for aided awareness  and 9 percentage points for association.  This also included the difficult to increase persuasion measures like favorability (an increase of 23 percentage points) and purchase intent (an increase of 12 percentage points).  But this was not surprising, since for Recall, the exposed group just includes people who remembered having seen the advertisement.   Clearly, this approach would only be correct if 100% of the people exposed recalled that exposure.

The next step in our comparison was to understand the accuracy of each at assigning people to the correct group (which can be difficult to do).  However, the digital channel tracking data provides information on which people were served the ads in the campaign, allowing us to see if people correctly recall their exposures.

Two digital campaigns were used for this second part of the analysis.  The cookie tracking data was compared to respondents’ recall of the ads shown again within the survey.  Between 16% and 25% of those who were not shown the ads incorrectly recalled exposure to the campaign.  Only 52% to 69% of those who were shown the ads remembered the exposure. 

Two campaigns may not provide enough data to definitely conclude what the false positive levels are across channels, but this analysis illustrates that people appear to have inaccurate memories for advertising recall. 

While there may never be a perfect methodology for assigning people to cross-media exposure groups, using recall likely overstates the effect of campaigns considerably.  It may be tempting to use it because the results are probably going to be very positive; however, marketers and researchers looking for accurate findings should be cautious about the recall-based approach.

• Comment (4)
• Recommend (4)
• Tweet
• 

4 comments on "Ad Effectiveness Methodologies: Comparing 'Recall' With 'Opportunity To See'".

1. Nick D from ___
   

   commented on: April 4, 2012 at 3:03 p.m.
   But both of these approaches are functions of an inexact research methodology. Since we're measuring actual exposure to online ads, it's methodologically unsound to compare anything except *actual* exposure to offline ads - simply put, with anything else we're not comparing like with like.
   While combining PPMs with cookie data is not especially easy or cheap, it's the comprehensive way to approach it; anything less is going to mis-state the effect of one or more media.
2. Tony Jarvis from Olympic Media Consultancy
   

   commented on: April 4, 2012 at 3:15 p.m.
   Recall, even "aided" recall used in this experiment, has long been dissmissed by the ad research cognoscenti as generally we can only recall around 5% of what we are exposed to. As you demonstrated even aided recall produces wild and biased swings. Of course, we will not mention the biases that accompany internet research or the use of panels which no doubt effected your results or the privacy issues regarding cookies! Your conclusions underline why some of us have been pushing so hard for so long for all media to provide ad exposure or known ad contact metrics versus OTS. Network TV with C3 is somewhat close, OOH with Eyes-On is right on and iab's 3MS initiative should ultimately provided viewable ad exposures for internet & mobile. Arbitron's PPM could provide minute by minute commercial audiences but the radio industry cannot stomach "lower" but much more valuable ratings. The Bottom line for cross media evaluation? Start with Ad or Commercial Exposure ratings for the target group across all channels! Tony Jarvis
3. Chris Borchert from Millward Brown - Dynamic Logic
   

   commented on: April 4, 2012 at 4:04 p.m.
   One has to remember that this test is comparing two methodologies that are meant to look at different aspects of ad effectiveness. OTS is measuring the strength of the medium/its efficiency at reaching and conveying a message to consumers. Recall is a measure of creative strength and breakthrough within that medium (essentially a subset of those that would fall into an OTS definition and you would expect to see higher scores within this group). To truly understand ad effectiveness, marketers need to look at both measures - until there is a source of passive data for all touch points.
4. Haren Ghosh from Symphony Advanced Media
   

   commented on: April 4, 2012 at 5:43 p.m.
   This is a very interesting and important topic in the advertising effectiveness measurement field. The validity of branded ad recognition as a proxy of advertising exposure has been debated for many years. For example, check out “Reconsidering Recall and Emotion in Advertising” by Mehta and Purvis (Journal of Advertising Research, 2006); you’ll see that “Advertising Research Foundation Copy Validity Research Project (ARF CRVP; Haley and Baldinger, 1991) found recall to be a valid measure of advertising effectiveness, second only to advertising liking.” Although the citation is dated, the fact remains unchanged. There is no empirical evidence yet that suggests otherwise. There is a very good book written by Erik Du Plessis on this topic, The Advertised Mind, which discusses ad recognition, ad recall, and persuasion in length. I do agree with Chris Borchert that Ad Recognition and OTS are meant to look at two different things, but there is a way to use the former one to get to the latter one. Repeatedly across varied product categories, we (at Symphony Advanced Media) found that the persuasive power (either active or passive) of advertising messaging (within respective media context) is the key driver of advertising effectiveness. For example, when clients run the same creative on TV and streaming media, we found that the effectiveness of one as very different from the other one. However, it’s crucial to differentiate passive memory from active memory during the measurement. As Tony correctly mentioned, a fraction (this could vary by number of channels where the ad running, the quality of ad, messaging attributes, type of products, etc.) of consumers could recall correctly. Therefore, one needs to capture the passive memory as well. We have done many studies, where we have noticed consumers say that they are unsure about their ad exposure but their responses are higher than that of those who had definitely not seen the ad. Another interesting outcome we have found - those who definitely don’t recognize the branded ad behave statistically similarly to those who ‘truly’ had no OTS to see the ad. Please feel free to reach out to me, if you need any detailed research information on this topic, Haren@SymphonyAM.com

Leave a Comment