What is A/B Testing?
A/B testing allows you to apply scientific principles to the digital user experience. With A/B testing, you’re able to create a process to show different versions of a piece of content to people and see which one better achieves your goals. With it you can validate whether any new design or change to an element/copy on your content is optimized for user consumption.
This helps marketers with:
- Knowing your audience beyond instinct
- Learning what makes your audience tick
- Driving audience engagement
What goals can you track?
- Email open/click rates
- Clicks on a specific web or email element (Click-thru rate)
- Overall engagement: total clicks, time on site/page, pages per visit
- Abandonment rate
- Subscription requests
- Form submissions
- Video views
Best practices to consider before you start an A/B Test
Start with a simple idea—and keep it simple. Start with small, carefully measured experiments to begin to get a gauge of what makes your audience tick. Start simple to minimize the potential complications and maintain the integrity of your experiment.
Know what you’re hoping to learn about your audience:
- What will a positive or negative result tell you about your audience?
- What are you hoping to fix or improve?
- Will you be able to apply what you’ve learned to future decisions?
Don’t give away the reins. When initiating A/B testing on e-mail newsletters, editors shouldn’t simply assign this task to their marketing or web department. Editors should be very involved with the process from start to finish so as not to lose sight of the editorial integrity of the experiment and goals.
How to develop a statistically significant A/B test
Statistical significance represents that likelihood that the difference in conversion rates between a given variation and the baseline is not due to chance. Your statistical significance level reflects your risk tolerance and the confidence level you can have in the validity of your test results.
Create a solid hypothesis: A solid test hypothesis goes a long way in keeping you on the right track and ensuring that you’re conducting valuable marketing experiments that are quantifiable and generate lifts as well as learning.
Make sure your experiment is broad enough: You want your experiment to deliver audience insight beyond that one-off test that you can apply to future decisions.
Identify your metrics of success: In digital, any action is quantifiable, but with testing the most difficult task is figuring exactly which metric actually tells you whether an experiment is successful. Make sure you have this determined before starting your test.
Most important thing to remember: Select ONE variable to test. Only one!