Test Big, Test Often

by daniel.c , Melinda Krueger, Dec 13, 2005, 2:00 AM

• Comment
• Recommend (2)
• Tweet
• 

"If you're not falling," we told our kids when they were beginning skiers, "you're not learning." By encouraging them to take risks rather than avoid an embarrassing tumble, we helped them develop their skills.

Likewise, with e-mail, you'll learn more and learn faster if you try new things and "test big."

"Testing big" means testing something radically different. Many times when I have tested small, I ended up with results that were dead even, statistically insignificant or contradictory when retested. This happened both when we tried to arrive at the optimal number of products for an e-mail (Is 6 better than 8? Is 9 better than 7?), and when we tested different subheads for product categories. Every best-day-to-mail and color test I've done was "too small" as well.

A big test yielded a big difference in results. We tested two versions of an e-mail, one leading with the advertising message and the other leading with the consumer benefit. Of course the consumer benefit approach won, but what was surprising was the magnitude of the win: it beat the advertising approach by 140 percent!

In direct mail, there is typically a control package that you seek to beat by developing a radically different approach. Once a winner is identified, you begin to test smaller in order to either increase response further or reduce costs without dampening response. When a package wins, you develop theories about what it is that made it a winner and test those theories as well. But it all begins with a "big test."

Sometimes the test idea develops during the creative process, when opinions about the best approach differ. Embrace this as a great opportunity. Another way to develop a different approach is to allow two different teams to develop an e-mail. (The extra horsepower generated by the desire to win is a side benefit.) E-mails from your competitors and others with a comparable business model are also a good source of test ideas.

Force yourself to take some risks in your approach to e-mail. You'll fall occasionally, but you'll learn more--and learn faster--than those taking the safe-and-easy approach.

A correction: my extremely talented editor sought to clarify my last article by changing "mail" to "snail mail." I apologize for this characterization and the implied insult to the U.S. Postal Service that, far more often than not, does a great job for us all, particularly during this time of year.

• Comment
• Recommend (2)
• Tweet
• 

Be the first to comment on "Test Big, Test Often"

Leave a Comment