• by Morgan Stewart , July 23, 2008

Recently the EEC posted the results  from a survey asking, "What is the primary metric by which you measure the success of your subject line A/B tests?"

The results:

42% --> Open rate
13% --> Click rate
18% --> Click-to-open rate
26% --> Conversion rate

There is no "right" answer. It varies by business, by situation, and by intent of the test being conducted.  Each of these metrics measures something different and it is critically important to understand what is-in fact-being measured when each of these metrics is used to evaluate subject line A/B split tests.

Open rate: Here, I am referring to the unique number of emails that were opened as tracked from a tracking pixel divided by emails delivered. Open rate is the only way to answer the question "Which subject line will get the most people to open the email?" However, it says nothing about the how well the subject line leads into the content of the email itself.

Click rate: Taken alone, click rates measure the combined success of the subject line and the content contained within the email. In the evaluation of subject lines, it answers the question, "Which subject line works best with the content contained in my email to get subscribes to my site?"

Click-to-open rate: This metric measures activity post open. At this point, the subject line has performed the job of getting the subscriber to open, but did it set the right expectations for the content in the email?  

Conversion rate: When conversion is the goal, this metric is always the bottom line. However, it really measures how all the parts of the campaign work together-the subject line, email creative, landing pages, site flow, and checkout processes. In evaluating subject lines, it measures, "Which subject line does the best job of setting subscribers on the path toward a transaction?"

What's the point?

More often than not, they are run at the last minute in an attempt to squeeze the last few dollars out of a campaign after everything else is set in stone -- the email creative is approved and the site cannot be changed. In this case, if conversion is the goal, then conversion rates are the logical measure of success. Alternatively, publications focused on driving CPM sale of ad units would look at open rates, and those focused on maximizing site traffic would evaluate click rates.

To suggest that one metric is right, and implying that others are wrong, for evaluating subject line performance is simply foolish. Each metric serves a distinct purpose. In the recent backlash against the open rate as a viable metric, I believe we are missing a big part of the picture. Namely, evaluation of subject lines using open rates provides unique insight into what our customers WANT to hear from us. Let us not forget that an open remains the first level of engagement with your brand and getting people to open is an important step. Every other metric used in the evaluation process simply tells us if we delivered the goods or not. If there is a discrepancy between the "best" subject line per open rates versus click or conversion rates there may be a larger problem to address.
 
Imagine a finance company has an email about investment options. They have two different messaging tactics they want to test, but the email content itself is the same. Subject line A yields a higher open rate than subject line B. However, subject line B yields a higher click rate -- more people get to the site. In this instance, subject line B clearly does a better job of setting expectations for the content in the email. Does that mean subject line B is the winner? It depends how you look at it.

Given time to make adjustments, I would argue that subject line A remains the winner. The issue now is changing the content of the email so that is does a better job of delivering on the expectations set by the subject line. Of course, if the email must be sent in short order use Subject Line B. Just recognize that you have settled for second best to meet a timeline. Companies looking to drive long-term value will gravitate toward subject line A and figure out how to optimize email creative to deliver on the initial interest generated.

Due to the granular level of measurement we have in email, we have the opportunity to make sure each component is doing its job -- from subject lines, to creative and copy, to landing pages, site flow, and the checkout process. Optimizing email performance involves multiple levels. Instead of arguing about the relative value of these metrics, we should leverage all of the tools we have at our disposal and determine what they are telling us about performance at each stage of the process.
  ;