Ad Effectiveness Methodologies: Comparing 'Recall' With 'Opportunity To See'
The Evolution of Advertising Effectiveness Research
Over the past decade, cross-media research has become increasingly important to advertisers. Today's marketers utilize multiple media channels to reach their target audience, and advertising research methodologies have also evolved to compare those channels on their ability to educate and persuade. InsightExpress recently examined two popular methodologies to determine if and how their results differ.
The first methodology is based on consumer recall of advertising. After a campaign launches, a respondent is asked a series of survey questions where one shows an advertisement that is part of the campaign being measured. Those who recall seeing the ad are classified as “exposed,” while those who do not are considered “unexposed.” Comparing the two groups leads to a statistical determination of the effectiveness of the campaign at changing awareness and perceptions towards the brand.
The second methodology is based on the “opportunity to see” an ad (OTS). Here, respondents are asked about recent media consumption habits including specific TV shows, channels, and magazines. In the digital world, cookie data is collected on advertisements sent to respondents’ browsers. This media consumption data is compared to the campaign's media plan or a TV post buy report to determine who had the opportunity to see the advertisements. Respondents who consume the “right” media but not at the specific time that the advertisements ran are assigned to the “unexposed” or “control” group to provide a baseline for comparison. As with recall, the two groups are statistically compared to determine if any changes in attitudinal measures resulted from advertising exposure.
The Contest: Recall vs. Opportunity To See
To understand the benefits and limitations of these two approaches, we examined a random sampling of studies that contained the ad recall questions and the OTS questions. They ranged in complexity and included a single site online advertising effectiveness study, a multi-site online advertising effectiveness study, and a multi-channel cross-media study. Results were calculated using the OTS and recall methodologies, with data averaged across the various studies to determine how the results differed.
The data showed considerable differences between the two methodologies. The OTS aggregation revealed that, on average, the campaigns were moderately effective at boosting two of the exchange of information measures: awareness of advertising for the brand (an increase of 2 percentage points), and association between either messaging in the creative or sponsorship elements with the brand (an increase of 5 percentage points).
The recall-based approach led to very large increases in every measure, such as an increase of 10 percentage points for aided awareness and 9 percentage points for association. This also included the difficult to increase persuasion measures like favorability (an increase of 23 percentage points) and purchase intent (an increase of 12 percentage points). But this was not surprising, since for Recall, the exposed group just includes people who remembered having seen the advertisement. Clearly, this approach would only be correct if 100% of the people exposed recalled that exposure.
The next step in our comparison was to understand the accuracy of each at assigning people to the correct group (which can be difficult to do). However, the digital channel tracking data provides information on which people were served the ads in the campaign, allowing us to see if people correctly recall their exposures.
Two digital campaigns were used for this second part of the analysis. The cookie tracking data was compared to respondents’ recall of the ads shown again within the survey. Between 16% and 25% of those who were not shown the ads incorrectly recalled exposure to the campaign. Only 52% to 69% of those who were shown the ads remembered the exposure.
Two campaigns may not provide enough data to definitely conclude what the false positive levels are across channels, but this analysis illustrates that people appear to have inaccurate memories for advertising recall.
While there may never be a perfect methodology for assigning people to cross-media exposure groups, using recall likely overstates the effect of campaigns considerably. It may be tempting to use it because the results are probably going to be very positive; however, marketers and researchers looking for accurate findings should be cautious about the recall-based approach.