Commentary

The Science of Testing Online

Back to the issue of advertising being art vs. science for a just a moment…

Traditional scientific testing requires the establishment of a control, isolation of variables, development of a hypothesis and a statistically significant sample size for definition of a confidence level that is high enough to warrant the analysis of an outcome. But how much is enough?

Over the last few years, I have heard many different theories as to what is a statistically significant sample size, and I have formulated some of my own opinions regarding these requirements but they depend rather heavily on past experience for a campaign, past conversion and response rates, and a few other variables. No one seems able to define a minimum level for testing, which leads me to believe one of the tenants of online advertising is that directionally relevant data is enough for testing purposes.

One of the advantages for online advertising is immediate accountability. This translates to a reduced risk for creative testing in this environment. The costs to develop an online ad are much less than a print or television ad and the performance data is available almost immediately compared to print which requires 3 months for pass-along readership to be fulfilled or for television which can require just as long. This means that initial data on performance can be attained almost immediately and optimization can happen just as quickly. The issue seems to be whether the desire for statistically significant data is a crutch to hold off immediate campaign revisions or is it grounded in actual theory that translates over from traditional advertising?

advertisement

advertisement

Traditional advertising testing was based on some fundamental assumptions and I think that the change in these assumptions may also affect the way that we perform tests in the online space.

For example, it is very clear that the attention span of the average consumer has decreased significantly. Generation X has led to Generation Y, which has led to Generation “Why are you bothering me?” As the attention span of the average consumer has decreased, the clutter of advertising has increased. There are more messages battling for an ever-decreasing slice of your attention. As these factors have entered into the equation, the data has become deeper and more readily accessible. If a user is going to react to an ad, they are most likely to take this action quickly, whether based on a click or a view-through conversion. It seems that most data suggests that the reaction to an ad, especially online, will occur within a 14-day period of exposure, and much shorter if they clicked on the ad when initially exposed. This implies that the most significant data tracking a consumer’s reaction happens in a much shorter period in the online environment than it may in other forms of media (full circulation is still a factor in print, but we may even see a similar trend occurring in television or radio advertising based on the accessibility of follow-up information in the digital age).

These factors tend to support the rationale that testing in the online environment is based less on a sample size as it is on a sample period, but I would also argue that directional data can still be acted on and will most likely be indicative of the final outcome of a campaign. If this is the case, then optimization of an online campaign can occur in just the first few days of launch depending on the projected response time for a consumer. More informed purchase decision products might require a longer period of initial data collection whereas impulse products may require significantly less.

Directionally relevant data can be acted on, optimization changes made, and results analyzed the following day. If the directional change is a positive one, then your hypothesis was correct. If the change caused a negative reaction, then the hypothesis was incorrect and you should make changes back to the format of the original campaign. This results in what should amount to a more effective campaign in a fraction of the time that it would take in other forms of media. In fact, message testing for a campaign can be established in this environment and applied to other forms of media, thereby reducing the risk associated with the launch of a new campaign in other media as well. This coupled with effective use of focus groups for initial campaign development assists in the development of an overall more effective campaign.

Now, the accountability factor here is good and bad as it means that the initial launch stages of a campaign are very hectic and require much more work than any other form of media, but the immediate “pain” does turn into a long-term “gain”, so in all in all it would appear as though the end justifies the means, don’t you agree?

Next story loading loading..