| ||||||||||||
PRINT
COMMENT
SUBSCRIBE
RSSThe results:
42% --> Open rate
13% --> Click rate
18% --> Click-to-open rate
26% --> Conversion rate
There is no "right" answer. It varies by business, by situation, and by intent of the test being conducted. Each of these metrics measures something different and it is critically important to understand what is-in fact-being measured when each of these metrics is used to evaluate subject line A/B split tests.
Open rate: Here, I am referring to the unique number of emails that were opened as tracked from a tracking pixel divided by emails delivered. Open rate is the only way to answer the question "Which subject line will get the most people to open the email?" However, it says nothing about the how well the subject line leads into the content of the email itself.
Click rate: Taken alone, click rates measure the combined success of the subject line and the content contained within the email. In the evaluation of subject lines, it answers the question, "Which subject line works best with the content contained in my email to get subscribes to my site?"
Click-to-open rate: This metric measures activity post open. At this point, the subject line has performed the job of getting the subscriber to open, but did it set the right expectations for the content in the email?
Conversion rate: When conversion is the goal, this metric is always the bottom line. However, it really measures how all the parts of the campaign work together-the subject line, email creative, landing pages, site flow, and checkout processes. In evaluating subject lines, it measures, "Which subject line does the best job of setting subscribers on the path toward a transaction?"
What's the point?
More often than not, they are run at the last minute in an attempt to squeeze the last few dollars out of a campaign after everything else is set in stone -- the email creative is approved and the site cannot be changed. In this case, if conversion is the goal, then conversion rates are the logical measure of success. Alternatively, publications focused on driving CPM sale of ad units would look at open rates, and those focused on maximizing site traffic would evaluate click rates.
To suggest that one metric is right, and implying that others are wrong, for evaluating subject line performance is simply foolish. Each metric serves a distinct purpose. In the recent backlash against the open rate as a viable metric, I believe we are missing a big part of the picture. Namely, evaluation of subject lines using open rates provides unique insight into what our customers WANT to hear from us. Let us not forget that an open remains the first level of engagement with your brand and getting people to open is an important step. Every other metric used in the evaluation process simply tells us if we delivered the goods or not. If there is a discrepancy between the "best" subject line per open rates versus click or conversion rates there may be a larger problem to address.
Imagine a finance company has an email about investment options. They have two different messaging tactics they want to test, but the email content itself is the same. Subject line A yields a higher open rate than subject line B. However, subject line B yields a higher click rate -- more people get to the site. In this instance, subject line B clearly does a better job of setting expectations for the content in the email. Does that mean subject line B is the winner? It depends how you look at it.
Given time to make adjustments, I would argue that subject line A remains the winner. The issue now is changing the content of the email so that is does a better job of delivering on the expectations set by the subject line. Of course, if the email must be sent in short order use Subject Line B. Just recognize that you have settled for second best to meet a timeline. Companies looking to drive long-term value will gravitate toward subject line A and figure out how to optimize email creative to deliver on the initial interest generated.
Due to the granular level of measurement we have in email, we have the opportunity to make sure each component is doing its job -- from subject lines, to creative and copy, to landing pages, site flow, and the checkout process. Optimizing email performance involves multiple levels. Instead of arguing about the relative value of these metrics, we should leverage all of the tools we have at our disposal and determine what they are telling us about performance at each stage of the process.
Meet Morgan Stewart at Email Insider Summit Utah!
Morgan Stewart will be there speaking during "Mobile Marketing of the Future" on December 08 at 10:30 AM. Top executives will be there. Will you?
Register today and save.
Register today and save.
1 person recommends this article.
RSS FEEDS
There is a big difference between being driven by deadlines (my email goes out at noon every Tuesday) or product availability (We have just run out of - or received a big shipment of MP3 payers) and rushing out poorly thought out campaign to squeeze a few more dollars out of your customers. Either way, even if they wanted to most companies just do not have enough resource on standby to produce offers write copy and QA an email campaign at such short notice.
The research on subject lines by Alchemy Worx that Chad referred to, indicates that when it comes to subject line length, more opens = less clicks, but it does not advocate giving up on subject lines that optimize opens in favour of those that optimize clicks or any other metric.
What I understand from the study is that open rates are just a small part of the story; your tests should assess the impact of subject lines on opens, clicks, CTO rate and conversions, as well as sales.
In fact my greatest fear is that the people reading the whitepaper will be looking for a simple answer such as “when it comes to email subject lines, short is best�, when in fact the central message is keep searching, keep optimizing and keep on challenging assumptions.
You can download the free whitepaper here http://www.alchemyworx.com/news_subject_line_study.php
Perhaps in the part of most of your world (I assume UK and US), digital is indeed far superseding traditional media ad spend, in my part of the world, yes, Asia, who happens to be a market that had received much focus, marketers are still yet to question the accountability of traditional media.
To a certain extent, they have been educated, perhaps too well and too loop-sided, that when it comes to digital, they want everything to be accountable and yet when it comes to traditional media, they are still splurging huge amounts of marketing dollars w/o questioning the accountability.
Using open rates to measure the outcome of A/B split tests of whether one subject line (or style, attributes, etc.) absolutely measures the effectiveness of the subject line (putting aside all of the inaccuracies I've written about in Email Insider columns - http://www.mediapost.com/blogs/email_insider/?author=23 - to motivate recipients to open. Testing "Free Shipping" versus "10% Off" will provide insight into consumer motivations. But it is only one input, and perhaps the least important. As an example, "10% Off" might generate a higher open rate and more orders; but the "Free Shipping" subject line might result in higher average order value and better margins. The point: I'm not just going to change my email content because a particular offer or approach beat out another subject line. I'll absolutely take the increased initial interest in the "10% Off" offer in mind when designing future emails and offers, but the only thing that matters is the affect on the bottom line.
In summary, Morgan: I have to disagree with your financial company example. Interest in the subject line is just that: interest. And while I like to suggest that you have a subject line in mind when developing your email creative, the reality is the best subject lines are designed to motivate recipients to take the actions you want when you designed the body of the email. So, the best subject lines reflect what is in the email body and should in turn be modified and optimized over time, based on what body content converts customers - not the other way around as you suggest.
A preemptive strategy and defined metrics will make a world of difference.
Joe Reis
www.androidsagency.com
I am only arguing that open rate (or "render rate") is a valid way to compare subject lines. It is not always the right choice, but answering the question about which subject line gets more people to view your email (whether on a mobile device, with or without images on, etc.) can have material impact on your business. The alternative metrics used to evaluate subject line performance do not answer this important question.
To Chad’s, point, there is the chance of chasing your tail if the winning subject line is vague and Alchemy Worx study does a great job of highlighting the issues. This is why we need to look use all of the tools at out disposal to figure out what is going on at each step.
Alchemy Worx' recent research showing that shorter subject lines generate higher opens but lower clicks is worth serious consideration by everyone. At the core of that research is the issue of whether you're being subscriber-centric and subscriber-friendly: Are your subject lines explicit enough about what subscribers can expect when they open the email? Or are you being vague and mysterious in the hopes of luring more subscribers of opening the email? Being vague and mysterious can certainly generate higher open rates, but then you've done a rotten job of setting expectations.
Ideally, you shouldn't be sending messaging with the same SL to the entire list. Instead, you should identify the best formats for each of your segments after comprehensive SL testing and then on an ongoing basis you should test different variables in these formats.
1. Open rates are the least reliable metric. What's measured vs what actually happens are very different. Having said that, one can always assume actual open rates are slightly higher than measured open rates.
2. Conversion is what it's all about. Business owners aren't handing out promotions and bonuses to marketers who demonstrate increased open rates or CTRs. At the end of the day, it's all about increasing conversion and ultimately revenue.
I think conversion should be the metric that matters most. The other metrics are just nice to know. I know this is what matters most to our customers. If you're getting good conversion, everything is working - your subject line straight through to the call to action and associated web forms...
Carolyn Gardner Sitebrand
The problem with the open rate is that it may not accurately report how many people are actually looking at your content. I often click on links from emails from the preview pane without removing the image suppression in my client. So I would contribute to the click count but not the open count.
Perhaps it would be a more beneficial to determine which Subject Line causes more clicks?
All the best
Tom www.messagingtimes.com