ab testing email marketing


5 Best Practices for A/B Testing Your Email Marketing

By Ben Kruger

A/B, or split, testing is a strategy that is applicable to almost every discipline of marketing, especially email. It is a great way to determine which variations of a marketing message will improve conversion rates and thus improve your brand’s sales and revenue. Identifying ways to increase conversion rates by even the slightest percentage can have a significant impact to your bottom line and ROI. The most common mistake email marketers make, however, is becoming comfortable with average, or even good results. The most effective email marketers are always looking for new ways to improve their strategies and never fall into complacency.

That said, A/B testing in email marketing can take a lot of time and energy. It’s essentially like performing mini behavioral science experiments on a continuous basis. If you’re just getting started or consider yourself a veteran, here are a few best practices to keep in mind to make testing your email marketing program a little easier:

1. Isolate your test variables

In order to complete a successful A/B test, you should only test one variable at a time. Doing this is the only way to truly determine how effective that variable is. Let’s say you’re looking to increase clicks. In your single test, you try a few different call-to-action button designs and different images in your email body. Even if you succeed in seeing an increase in clicks, how will you know what actually drove that behavior? Short answer, you won’t be able to. Isolate variables for every test you run so you can know for certain which variables deliver your results.

2. Always use a control version to test against

A “control” or default is the original version of the email you would have sent anyway, as if you weren’t testing anything. This will provide you with a reliable baseline to compare your results with. The reason that having a control version is so important, is because there are always “confounding variables” or variables that you can’t control that impact the validity of your test. For example, a confounding variable could be something like one of your email recipients being on vacation without internet access during your test. By testing against a control version, you are cutting down on as many confounding variables as possible in order to make your results accurate. A control version will also serve as an easy variable to gauge results against. Without a baseline to measure against, it becomes difficult to see the actual lift the test version has driven.

3. Test simultaneously

Timing is everything, especially in ecommerce marketing. Throughout the year, retailers experience seasonal highs and lows. In order to account for any seasonality, changes in your customer behavior or changes to your product catalog, it is best to run your tests in parallel of one another. Bluecore will take care of this for our customers by splitting audiences and delivering tests randomly.

4. Check if results are statistically significant before declaring a “winner”

Going back to comparing A/B tests to science experiments, you want to make sure the results you’re finding actually mean something before you move forward and implement them into your email marketing strategy. In statistics, in order for results to mean something, it needs to be “statistically significant.”

To determine statistical significance, we use a “p-value.” This represents the probability that random chance or error could explain the result you find. In general, a 5% or lower p-value is considered to be statistically significant. Depending on the number of emails your triggered email program sends, it will take a few weeks until your results will be achieve a p-value of 5% or less.

5. Continuously challenge through new tests

Just about every aspect of an email can be A/B tested for optimization. Get creative with your variables and always be trying to think of new aspects to test. Take your subject line as an example. Several variables can be tested within that single aspect of an email, such as length, urgency, mention of a promotion or use of the recipient’s name, among others.

ben kruger headshot

Ben Kruger

As the Senior Manager of Data & Insights, Ben helps retailers maximize their value of the Bluecore platform by identifying trends and opportunities within their datasets.

ab test email subject lines

How to Start A/B Testing Emails and Subject Lines

Perhaps the biggest opportunity to impact email performance is within the subject line. Here are five thought-starter ideas to approach A/B testing this.

Read More

How Tommy Hilfiger Optimized Triggered Emails Through A/B Testing

How Tommy Hilfiger transformed a limping marketing program to its top performing channel with triggered email A/B testing.

Read More
7 tricks to improve email

7 Easy Tricks to Immediately Improve Every Email

Some of the most impactful ways to improve your emails are easy (and free) enhancements you can implement right away.

Read More

Sign up today to get the latest content from Bluecore delivered to your inbox.

Thanks for subscribing