Watch out, research firms! The Interactive Advertising Bureau has embarked on a broad initiative to improve online brand effectiveness research, and its initial findings aren't pretty.
What's wrong with most research that attempts to measure ad effectiveness? Small respondent size and low response rates for starters, according to an initial report from the IAB.
Above all else,
the validity of such research is threatened "by the extremely low response rates achieved in most IAE studies," according to Paul Lavrakas, Ph.D., the report's author, and former chief research
methodologist for the Nielsen Company.
Average research is also "threatened by the near-exclusive use of quasi-experimental research designs rather than classic experimental designs," in the
words of Lavrakas, author of "Telephone Survey Methods: Sampling, Selection, and Supervision."
Worse still, industry research is often compromised by "a lack of valid empirical evidence that the
statistical weighting adjustments ... adequately correct for the biasing effects," Lavrakas attests.
"In instances where the sample size is at the lower end of this range [less than 800
participants] and the clients want subsample analyses to be conducted ... these subsamples may not have enough members in them to provide precise analyses," Lavrakas concludes. "Thus, subsample
analyses based on small sized subsamples [fewer than 100 participants in the subsample] will have relative large sampling errors."
To accompany Lavrakas' findings, the IAB is launching a
cross-industry task force to help the interactive ad industry better understand how most studies impact the broader supply chain, and then suggest ways to minimize potential inefficiencies they may
create.
"If the phrase 'marketing science' is to have any meaning, participants in the ecosystem must demand that their vendors employ rigorous, tested research methodologies, even if doing so
costs more," said Sherrill Mane, SVP of Industry Services at the IAB.
The IAB also plans to create a set of U.S. best practices for online brand impact ad effectiveness studies, including
recommendations for marketers, ad agencies, research vendors and publishers.
The IAB gave research vendors the opportunity to respond to Lavrakas' report.
ComScore, for one, said Lavrakas'
"report evaluates two methods for assessing online attitudinal brand studies: 'site intercept studies that sample persons in real-time as they are using the Internet and studies that sample members of
existing online panels.' Neither of the methodologies discussed in this report is fully representative of the methodologies in use today by comScore."