How to interpret A/B test results
A/B testing is a form of marketing research. It’s important to establish goals while planning a test, so you can evaluate the results, determine a winner, and personalize the rest of the campaign deployment to reflect the winning outcome. In many situations, an audience is presegmented with a holdout group that will receive the winning version of a message. In other cases, long-term testing of automated messages can allow marketers to establish statistical significance before finalizing a test.
The test results will indicate the success of one element over another based on what you’ve decided to measure, such as:
- number of visitors
- open rates
- click-through rates
- signups (for newsletters, etc.)
The two elements are monitored until a statistically sufficient measurement is achieved.
Conversion rates can also be measured in terms of revenue. You might consider sales numbers along with the impact of a change on actual sales revenue. Remember that conversion rates can be any measurable action and are not restricted to ecommerce sites and sales. They can include:
- sales made
- leads generated
- newsletter signups
- clicks on banner ads
- time spent on the site
What sort of metrics should you be paying attention to when it comes to A/B testing?
The answer to that question depends on your hypothesis and goals. However, you should note metrics that show how engaged your audience is with your marketing content.
If you are testing a web page, look at the number of unique visitors, return visitors, how much time they are spending on the page, as well as the bounce and exit rates. For an email, you will want to see who opens it and clicks through to your call to action (CTA).