7 things we learned from A/B testing on emails

Leanne Feris is brand editor on the Ackermans account. To sign up for our newsletters click here.

The Ackermans team took email campaigns a level up after a series of A/B tests. Here’s what they learnt.

When we first started AB testing on the Ackermans account, it was a little daunting. Where do you even begin? How do you do it? Luckily we had some experts on hand (shout-out to Kamiel at our in-house digital agency Dialogue) to give us a few pointers.

So what is A/B testing?

It’s also called split testing – you compare two versions of an emailer/website/landing page to see which one does better in terms of whatever metric you use to gauge success.

We did A/B testing on the Ackermans account’s newsletters and product emailers that are sent to customers, from account holders and members of their rewards programme, to prospective customers.

While the mailers did well, there’s always room for improvement and we wanted to increase not just the open rate, but also the click rate and the click-to-open rate.

Here are the top seven things we learned.

  1. Just begin.

The easiest thing to start with is the subject line. Short or long? Straight-forward and to the point, or a little bit more clever and playful?

We found that for certain groupings, quirky subject lines work well, but in general straight-forward works best and shorter is definitely better. Having said that, it may not work for your audience, so test away.

Also, subject lines are hard! So get other people involved in impromptu subject line brainstorms. Sometimes you’ll get the most inspired subject lines from the most unexpected person in the team! #winningatwork

  1. Be creative when thinking about what to test

While you have those creative juices flowing, consider all the options available to test – there are many (see the list below), so test everything. But remember, test just one thing at a time.

Try these options:

  • Subject lines
  • Call-to-action copy
  • Call-to-action button colour
  • Links on images
  • Images
  • Copy or images above the fold vs. lower down
  • Which days to send
  • What time of the day to send
  • Personalisation
  • Location of social media buttons
  • Inbox preview copy


  1. Don’t assume. Test!

Some of the most clever calls to action simply did not resonate with our audience. We realised once again that the rule of ‘first be clear, then be clever’ still holds true. Testing will help you stay reader-focused.

In one emailer, we tested more creative calls to action against straight-forward ones. We tested “Here’s how” vs. “Read more” and “Get your shirt” vs. “View range” or even “Yes please!” vs. “Join now” and found that though the straightforward calls to action can be boring, for us, they give a higher click-to-open rate.

  1. Make sure the sample is big enough.

If you’re going to test something on a sample that’s too small, you won’t know if it’s accurate. We found this clever calculator tool to check if the results show statistical significance. Your results can be dissapointingly insignificant – but in that case, keep testing.

  1. Remember the big picture

However, not every statistically significant result will look impressive – and that’s okay. Avoid the temptation to throw out the results – rather take the learnings (for example what types of wording doesn’t work for your audience) and keep tweaking. Over time, even small changes add up.

  1. Do the test before your actual send
    Before you send your newsletter to your entire database, choose a sample for the A/B test. Then send the one that does better with whatever metric you tested to the entire base.


  1. Keep a log as reference

Have a record of what you tested and when, the results and the recommendations. That log will be very useful when you report your findings. And then, start doing what you know works!

  • A/B testing helped us be featured in an Everlytic white paper on Awesome Emails – read about it here.

Have you tried A/B testing? What were your insights?

Comment below…

Written by Leanne Feris

Post a comment

Your email address will not be published. Required fields are marked *