• by editor , David Baker , July 23, 2007

If you are like most marketers, you are hard-pressed simply to plan and execute your marketing programs, much less come up with advanced testing scenarios that yield a blend of qualitative and quantitative data. There has been a lot written about testing in the email channel, and I've warned many times about not testing just for its own sake. Yet I'm still amazed to see companies that can't find time to structure testing into their game plan.

Marketing is like learning to balance on a bicycle. I can give you a how-to book or show you a video. I can demonstrate it for you, and even talk you through it while you are on the bike. But I can not teach you to balance. You'll have to find out for yourself the art of over-correcting and testing new leans in each direction until you get the proper balance and weight distribution. We call this tacit learning. Testing is very much a part of this process and unique to each program.

W. Gregory Dowling and Emily Riley of Jupiter did a nice piece on "Multivariate Testing and Site Optimization" back in August 2006. I liked how they laid out simple ways to structure testing. While the focus of the article was site optimization, there were key principles that apply to email marketing, as well as marketing in general. The art of testing isn't about hoping to find out some new conclusion, it is about validating your assumption or hypothesis and acting on it.

As is the case with site testing/optimization and email marketing, there are innumerable variables you can test and as many testing methods. The Jupiter article simplified the process by listing three keys to maximizing the effectiveness of site testing and optimization efforts. I feel these are genuinely applicable to email marketing.

Create a hypothesis for testing

Understand that testing is about creating a user scenario or behavior, not a specific element within your email. Most people in email do subject line testing, but is it really about putting "Free" in the subject line or "Free shipping" or creating a sense of urgency with "Last day of the sale"? Or is testing about understanding the correlation of customer segments to buying patterns and conversion, based on the timing, seasonality and product offer? How can we apply a testing design that incorporates design, message, voice, timing and offer into a viable testing plan? Can you use customer surveys to help substantiate this testing? If you don't have a hypothesis, testing and results will always be linear. Consumer behavior is a spider web of insight that is cobbled together to help you draw some conclusions about how they will interact with your brand.

Establish multiple objectives

In a busy marketer's world, objectives are often too simple to be of any value. While it is justifiable to list "increasing sales" as an objective of a program or the reason you are testing, is that helping you evolve what you know about your customer, program and efforts? This is why establishing qualitative and quantitative Key Performance Indicators (KPIs) is critical to answering a broad array of questions and interpretation. Testing takes on so many dimensions of how you view a customer, how you reach, how you build stimuli and support this behavior. Yet it all must correlate to objectives that can continually help you answer your short- and long-term questions. I often counsel people on the differences between goals and objectives. The goal may be increased sales, but the objective may be about gauging the effectiveness of a test in driving changes in KPIs.

Segment your testing audience

There are many schools of thought regarding testing and choosing your sampling and audience, but remember you are testing scenarios. So it is less valuable to you to test your entire universe. Test within core segments, new segments or unknown segments. This is how we model preferred behaviors. Since we typically know more about our best customers, modeling top prospects and creating test audiences within known segments, it helps in finding predictable variables that can be used to identify future best customers.

Remember, testing keeps us in motion ... and objects in motion typically stay in motion.