• by Cory Treffiletti , Featured Contributor, October 29, 2003

Everyone knows the Internet is a great place for testing messages, but how do you correctly go about implementing a test that can be truly indicative of future performance? I get asked this question often, and I figured I'd give some top-line answers here, just to get started.

First of all, you must isolate the variables. This is the most important element of any test matrix you're putting together. If you're going to be testing a message or an execution of a strategy, you need to be able to point to one specific variable that is the focus of your test and will have an effect on the results. For example, you can test a message by keeping the execution and design the same but rotating in different copy. You can test an offer if you keep all variables the same and change the offer itself. You can test a strategy itself if you keep the placements the same, the flighting the same, and the color-palette the same. No one can truly lay-out what your test matrix will be for you until they are completely aware of all the variables you have to work with.

One of the problems that many advertisers will run into is they do not isolate to just one single variable in their test. If you run two different messages in two different months, this is not a test, this is a comparison. In such a case, you have a variable of the message but you also have a variable of the flight period. You may have built frequency among your target audience during this period, negating the actual effectiveness of your test, even though directionally you will see a difference. This isolation of the variable refers to the need for a Control. The Control is necessary for a baseline to determine if the new message (or whatever variable you are hoping to test) is the true driver of the change in performance.

Second of all you need to keep in mind whether or not you will be able to achieve a large enough sample size to provide a high enough confidence level in the outcome of the test (most people shoot for a 95% confidence level but this does seem rather subjective). One common mistake is to make an optimization decision off a very small sample size that may not be indicative of the total population of your target audience. This mistake can be costly. The typical rule of thumb that I have heard is you should shoot for a sample size of 100 actions. This means that you put your test together based on the goal of 100 sales/customers/registrations or whatever action your campaign is based on achieving. Then you back into an impression number based on projections for performance that are realistic. If you have past performance, you should use these. Otherwise, you can base your projections off of the industry averages.

Third, you must formulate a decision tree that will be followed depending on the outcome. Anticipation of the outcome is important in any test, as these will affect your budgets and strategic direction. You should develop a hypothesis and determine what actions you will take if your hypothesis is correct or if it is proven incorrect. If it is correct, you may need to develop more executions of a creative strategy or maybe you will look to further isolate a second variable for a new round of testing. If it is incorrect, then you need to determine what steps you will take next to further improve upon your effectiveness of the campaign. Just because the test was not successful does NOT mean the medium is at fault. It simply means "back to the drawing board."

Do you think the Wright Brothers flew on their first test of a flying machine? Don't think so.

There are, of course, many other elements to consider in a test, and I recommend picking up any of a number of Direct Response books to get much deeper information on testing hypotheses and other methods of improving your campaign effectiveness, but this should help for now.

What do you think?