• by Daniella Koren , February 24, 2017

Medical research relies on tests with reproducible results, and the same maxim applies to healthcare marketing and education. Marketers need to understand what works, what doesn’t, and the methods of delivery that resonate with patients. To that end, communications should be assessed using A/B testing.

What is A/B testing?

A/B testing is a method for comparing the performance of two communications in a statistically significant manner. Applied to healthcare marketing, the items that can be compared are diverse — you could test a direct mail piece sent to a random sample of individuals (A) vs. another one (B) or the number of visits to one web page vs. another. Another way to think about it: A is your test and B is your control.

The keys to successful A/B testing:

1. The variables tested (e.g., a web page with certain copy vs. another) need to be isolated, so you know what’s influencing the results.

2. The sample size should be large enough to gauge the results. Comparing a page with 8 visits vs. one with 6 is not reliable. 

3. Test populations need to be randomized. Sending out half a mailing to individuals who are under 30 years of age and half to older individuals will skew the results.

4. Formulate a hypothesis. This is as simple as “A will outperform B.” 

Understanding confidence

Confidence, in statistical terms, represents how confident you are that a result (“A outperformed B”) is not due to chance. To obtain a high confidence level, you either need wildly different outcomes (also known as “effect size”) or a large sample size — ideally both.

There are many online calculators that allow you to input sample size and response rates to determine the confidence of your hypothesis. A sample test:

Hypothesis: Email A, with unique copy, will outperform Email B 

Sample size: 1,000 opt-in subscribers randomly sent Email A, 1,000 sent Email B

Results: Email A generated 26 clicks to a website, whereas B resulted in 12

Analysis: The hypothesis is validated. Email A converted 117% more than B, at a 99% confidence interval. This means you are 99% confident that the results are not due to chance. And if you ran the same test again, there is only a 1% chance that Email B would outperform A.

Let’s say the results were different: Email A generated 26 clicks, whereas B generated 24. In this case, Email A still outperformed B in raw numbers, but only by 8%. In relation to your sample size, this knocks your confidence level down to 61%, meaning there is a 39% chance that the next time you run the test, Email B will come out on top. 

This is not a confident result – typically, you should aim for 90, 95, or 98% confidence. Try it again with a larger sample size, or go back to the drawing board for a new test.

How A/B testing can be applied in healthcare marketing 

A test can be applied to multiple components of a healthcare campaign, including the medium, messaging, and target population. 

Crucially, it can also be applied to the types of information sent to patient groups. For example, an analysis of web inquiries to The Genetic and Rare Disease Information Center (GARD) in 2011 found that most individuals looking for information sought basic disease information (75%) and information about treatment (33.3%). These results might form the basis of an education campaign sent to opt-in subscribers:

Test A: A downloadable resource with basic disease information

Test B: A downloadable resource about treatment options

Tests could also be designed around a series covering different information. In addition, the content, from email subject line to design and copy, could also be tested. 

The possibilities are varied and seemingly endless — but research about a condition, what patients want to know, and how they want to receive the information can inform test design. From there, it’s an iterative process that allows you to find the best ways to communicate.

In this sense, healthcare marketers can break new ground – by improving communications, we can create effective campaigns and impart the right information, further educating the right patients, with the right message at the right time.