How to Start A/B Testing Emails and Subject Lines

Although there are several best practices marketers should follow when it comes to A/B testing email marketing campaigns, sometimes choosing where to start can feel overwhelming. That’s because even if you know that A/B testing can help conversion rates skyrocket, there are a lot of potential variables to test. 

But there’s no need to feel lost or confused — A/B testing is a process, and recognizing that is an important first step. If you aren’t sure where to begin, this post will help you not only understand the benefits of A/B testing but also understand how to start testing your email marketing campaigns today.

Download Bluecore’s Email Marketer Playbook

What is A/B Testing and Why is it Valuable?

A/B testing is a fairly simple process once you understand exactly what it is, but there are some important details marketers must consider before using it to optimize campaigns.

For email campaigns, A/B testing, also known as split testing, describes the process of sending two versions of an email to two similar — but separate — audiences. Ultimately, the goal of these tests is to find which variation best meets your marketing priorities. Once you’ve found the most successful way to present your email, you may choose to send it to all subscribers within an audience.

A/B testing provides a simple method to help meet email marketing goals, whether that’s improving open rates or click through rates. By sending two variations of the same email to different customers, you will see how customers engage with your emails before your campaign is complete. This allows you to optimize campaigns while they’re inflight.

However, the process of setting up an A/B test requires some forethought. While the right email marketing platform can simplify A/B testing, your marketing team first needs to be familiar with the strategic elements behind testing emails.

Types of A/B Tests for Email

An email may seem like a simple piece of marketing outreach, but there are many elements that impact a customer’s likelihood to take a certain action. As part of an A/B test for email campaigns, your retail marketing team may want to test the following areas:

• Subject line
• Call to action
• Template design
• Personalization
• Email copy
• Images
• Offers or coupons

Any of these variables could provide an excellent starting point for your campaign — but for now, you need to pick just one in order to follow best practices.

Tips for A/B Testing Emails

Before you start making changes, it’s important to familiarize yourself with a few tips and best practices. This will help ensure you make the most headway possible with your campaigns.

• Test one variable: While you may want to test more than one aspect of your email, it’s important to prioritize and focus on one variable at a time. Juggling two or more changes in an A/B test is bound to obfuscate which change truly made a difference.
• Always use a control: Using the original, unaltered version of the email as a control will provide you with a baseline against which to measure your successes. Getting this baseline is absolutely essential to obtaining accurate results.
• Justify the variation: Inherently, A/B testing means some of your customers will get a less effective email than others. You should have reason to believe your change will pose a clear benefit – otherwise, you’re intentionally sending suboptimal outreach.
• Use consistent timing: Your control emails and adjusted emails need to be sent at the same time on the same date to avoid any seasonal highs and lows. Otherwise, one email could win based on timing, not content.
• Define recipients: Think about which audience should receive a given email and split them in a random way. Some advanced email marketing solutions help you deliver tests randomly to a specific audience.
• Check for statistical significance: Just sending a single batch of emails usually isn’t enough to rule out differences caused by random chance. When comparing test emails side by side, watch for a 5% or lower p-value. Some retail email marketing solutions provide this capability automatically.

These tips will give your marketing team the starting knowledge required to begin digging into prospective emails and making well-informed changes. Now, you should be ready to get started — but which variable should you focus on first? Think about the steps a shopper takes to engage with an email. 

Example: How to A/B Test Your Email Subject Lines

While there are a few different aspects that can affect your email marketing program’s open rate, perhaps the biggest opportunity to impact performance is in the subject line. After all, if consumers don’t even open your email, it’s hard to tell whether or not your email’s template or call to action is as impactful as it could be.

There are a handful of ways to approach testing subject lines, but here are four ideas to get you started:

1. Personalize with the shopper’s name

Dynamic subject lines are often used to provide a personalized touch before a shopper even clicks on the email, and what’s more personal than someone’s name? Including a shopper’s name in the subject line immediately grabs attention and can potentially earn your email more consideration to be opened. Teleflora tests this tactic in its cart abandonment triggered emails with the subject lines, “[First Name], Your Blooms are Waiting” or simply “Your Blooms are Waiting.”

2. Include a product description or attribute

Another way to use dynamic subject lines is to include product attributes, such as the product name, brand name or category viewed by a shopper. Including this information can help trigger the shopper’s memory about the recent session they’ve had on your site and the specific products they’ve viewed. 

A major furniture retailer tested dynamic subject lines with its cart abandonment triggered emails by including the product name. In the example above, a shopper can receive an email with the general subject line of “We Thought You Might Like…” or a dynamic subject line of “[Product Name] and more…” For triggers that engage shoppers higher in the purchase funnel, such as search abandonment, you should use broader attributes, such as product category or brand name.

3. Ask a question or make a statement

Using questions as headlines is a go-to tactic for attracting readers. This works well because people’s brains are hard-wired to solve problems, and phrasing headlines as questions tells the reader they must read the article to find the answer. 

The same logic applies to subject lines, too. As a best practice, be sure your subject line asks a question the reader cannot easily answer, otherwise they have no reason to open your email. In the example above, Dermstore tests cart abandonment subject lines with a statement, “Take Them Home…” or a question, “Did You Forget Something?”

4. Play with character count

The KISS principle (keep it simple, stupid) is usually a good methodology to avoid overcomplicating things. Email subject lines are no exception.

Since most email providers can only display up to 50 characters, it’s best to test below that range. In the example above, Pura Vida tests subject line character count in its back in stock triggered emails. The first subject line keeps things short and sweet with “It’s Back!” while the second option also reminds the shopper they had previously wanted to buy a product with “You wanted it and now it’s back!” Even though both subject lines are simple, slight variations can sometimes make all the difference.

Final Thoughts

Every email sent is an opportunity to test and optimize. A/B testing your email marketing subject lines can not only have a huge effect on your program’s open rates, but also have a waterfall effect on your other marketing metrics, which will ultimately impact your bottom line. Use the tactics above to get started and be sure to follow our best practices for email A/B testing to make the process as easy and effective as possible.