How to run an A/B test
A/B testing
I've often seen small changes to a web page make big improvements to conversion rates. I wanted to share an example of small design changes that lead to big improvements. In this post, I'll show a couple of examples of A/B test that worked, one that didn't and what I learnt from them.
How to know what to test
This is one of the key parts of the process. When I start to think about setting up A/B tests, I start with the business goals or key performance indicators (KPIs). Ask the question "Which business goal do I want to improve?"
In this test, I was working on an e-commerce site, so the business goal I wanted to achieve was sales and driving revenue.
Having a clear idea of what you want to improve and why is vital.
Look at the data
Once you know what you want to improve and why, it's time to look at the data. The data helps you see where might be the best place to run a test.
The e-commerce site, a paint and wallpaper website, I was working on used Google Analytics for it's reporting. I started by looking at which pages were high value and had high conversion rates. When looking at the data, I notice that the paint primer and undercoat product pages had a particular high page value and average order value. Despite the high average order values and high page values, the page views for these products were low. After a bit of look around the site, it became clear that the products were not particularly promoted or easy to get to.
Create a hypothesis
Once the what and why had been decided it was time to come up with how to test it. The first part of this was to create a hypothesis.
My hypothesis for this test was "By making it easy to get to the correct primer and undercoat pages customers would be encouraged to purchase the product, which would increase the average order values."
To achieve this, I would add a link from the paint colours product details page to the corresponding primer and undercoat page. A seemingly obvious straightforward thing to do. The reasoning behind not linking to other products from the details pages had been not to distract customers from their current purchase journey.
Creating the test
To create this test I used Google optimize. It's free, it integrated seamlessly into Google analytics and was really simple to get set up on the site.
Control
The control page. This is the original page, no changes; this is what any changes to other versions would be measured against.
Variant
On the B page, I added the link to the different primer and undercoat. This was the only change made!
Set the test live and collect the results
I ran the test for two weeks to allow enough data to make a more informed decision.
The results were in
This small change had the following impact:
The average order value was up 11.7% on the page with the added link
Revenue was up 14%
The test showed that making it easy for customers to find an accompanying product helped increase average order value and revenue.
After this test had run, I wondered if including links to other complimentary colours in the product description would also increase revenue and average order value. The first test had worked well, why wouldn't this work too!
It turned out I was completely wrong! The additional links to other colours increased the choice the customer had to make, which made it harder for the customer to make a decision. Introducing the additional options by including links to other products reduced revenue and the average order value compared to the original page.
Lesson learnt! Don't get carried away, think about the theory behind the test first. The primer and undercoat test worked because this was a cross-sell. The additional suggested product accompanied the customers choice. In the original test, I had done the research to identify the opportunity. I planned the test and reasoning as to why it could work.
The second test was done in a slight hurry and with no research or no real theory behind it. Adding other colour options increased the decisions the customer had to make and complicated the experience. (If I had stopped to think about Hicks law I would have probably thought twice about running this test.)
Both tests ended. I implemented the 'primer and undercoat links' across the site, and the 'other colours links' stayed in Optimize, never to be implemented, but the lessons learned.
