• by Stephanie Miller , December 8, 2009

In the "Email Top Chefs" session - artfully organized and presented by Lisa Harmon and Aaron Smith of Smith-Harmon, now part of Responsys included four smart marketers sharing testing results. Great session with Orbitz, REI, General Mills and ESPN.com each generously sharing test structure and results.

Yet, every one of them expressed frustration that even well produced tests generate results that are difficult to read and measure. For example, Arti Parikh of Orbitz had to add in a second measure in order to break a tie. Christine Scott of General Mills found that the version that had the higher clicks (in one test 95% increase) broke even on interactions.

Another great lesson came from Carolyn Ude of ESPN.com, who found that those who received three messages actually responded less than those who received two. This was a real learning around lifecycle, and ensuring that all the messages reach subscribers when they are truly "in market," not just when we hope they are in market.