We are constantly told that testing is the path to better email response rates, but we also know how time consuming and expensive it is to test all the variables that define an email campaign. What Sir John and his colleague, Joseph Henry Gilbert, did was invent a better way to test that is absolutely perfect for email.
Traditional testing is called A/B or split run. Only one variable (or “factor”) is changed, such as the offer. There might be two possible levels, such as “$10 off a $50 purchase” vs. “$20 off a $100 purchase.” Changing only one factor at a time (OFAT) makes interpreting the results easy, but there are usually many factors we want to test, such as subject line, targeting, imagery or typography. If a separate email campaign was needed to test each factor, it would be a long time before the actual, optimized campaign was sent and many potential customers would be used in the process.
John Lawes faced the same challenge back in Victorian England. How much of his new, artificial manure should be applied? To which crops? When in the growing cycle? How much phosphate should he use? How much sulphuric acid? In what proportions? To compound the problem, the results would not be known until the end of the growing season, and the next growing season was a year away. At least with email we get the answer by the next day.
Sir John realized that he needed to test several factors simultaneously if he wanted to get an answer in his lifetime. He turned to mathematics and discovered the formulas that enabled him to vary multiple factors within one test and still uncover the effect of each of the results. Thus was born factorial analysis, which is also sometimes referred to as factorial design or multivariate testing.
Computers were not standard equipment at the Rothamsted Experimental Station in the late 1800s, so Sir John did his calculations manually. The math behind factorial analysis was beyond most experimenters, so over the next hundred years there was little demand for the technique. Marketers who might make good use of factorial analysis usually had no knowledge of it, due to their relatively low level of math skills. After all, they were marketers, not math-savvy farmers.
Today, those barriers no longer exist. The necessary formulas are found in many statistics textbooks, such as “Statistics for Experimenters” by Box, Hunter and Hunter. If the math is challenging, competent marketing services firms will do the work for you, including designing tests and interpreting results.
Regardless of who is designing or interpreting, you should be using these techniques. The best targeting in the world can be ineffective if coupled with poor collateral. The most attractive offers could fall on deaf ears if directed to prospects not yet ready to buy or unopened by recipients not tempted by the subject line. Because factorial analysis can test multiple factors together, fewer subjects are needed and the testing is completed sooner, reducing the cost. Effects due to interactions between the factors can also be measured. For example, a proper factorial analysis could reveal whether using or not using photographs in a brochure works better with bigger or smaller text headlines.
So give thanks to John Lawes and his followers -- not only for your abundance of vegetables, but also for a methodology that can really improve your email campaigns. And that’s not artificial manure.