Watch out, research firms! The Interactive Advertising Bureau has embarked on a broad initiative to improve online brand effectiveness research, and its initial findings aren't pretty.
What's wrong with most research that attempts to measure ad effectiveness? Small respondent size and low response rates for starters, according to an initial report from the IAB.
Above all else, the validity of such research is threatened "by the extremely low response rates achieved in most IAE studies," according to Paul Lavrakas, Ph.D., the report's author, and former chief research methodologist for the Nielsen Company.
Average research is also "threatened by the near-exclusive use of quasi-experimental research designs rather than classic experimental designs," in the words of Lavrakas, author of "Telephone Survey Methods: Sampling, Selection, and Supervision."
Worse still, industry research is often compromised by "a lack of valid empirical evidence that the statistical weighting adjustments ... adequately correct for the biasing effects," Lavrakas attests.
"In instances where the sample size is at the lower end of this range [less than 800 participants] and the clients want subsample analyses to be conducted ... these subsamples may not have enough members in them to provide precise analyses," Lavrakas concludes. "Thus, subsample analyses based on small sized subsamples [fewer than 100 participants in the subsample] will have relative large sampling errors."
To accompany Lavrakas' findings, the IAB is launching a cross-industry task force to help the interactive ad industry better understand how most studies impact the broader supply chain, and then suggest ways to minimize potential inefficiencies they may create.
"If the phrase 'marketing science' is to have any meaning, participants in the ecosystem must demand that their vendors employ rigorous, tested research methodologies, even if doing so costs more," said Sherrill Mane, SVP of Industry Services at the IAB.
The IAB also plans to create a set of U.S. best practices for online brand impact ad effectiveness studies, including recommendations for marketers, ad agencies, research vendors and publishers.
The IAB gave research vendors the opportunity to respond to Lavrakas' report.
ComScore, for one, said Lavrakas' "report evaluates two methods for assessing online attitudinal brand studies: 'site intercept studies that sample persons in real-time as they are using the Internet and studies that sample members of existing online panels.' Neither of the methodologies discussed in this report is fully representative of the methodologies in use today by comScore."