You’ve heard the stories. eCommerce case studies where revenue skyrocketed by as much as 107%, or revenue per visitor jumped by 17%. Are these just fairy tales? Internet rumors? How are these companies achieving growth at such unprecedented levels?
The answer is A/B testing.
What is A/B testing?
Simply put, A/B testing, also referred to as split testing, is when you show a different version of your web page to a bucketed group of your visitors and compare the desired outcome of each. A/B testing allows you to make tweaks and changes to your website and then track which variation performed better for your store.
Why should I be A/B testing?
Let’s face it, your website isn’t perfect. As a matter of fact, it’s more likely filled with leaky holes that are hurting your conversion rate and in turn, your revenue. You probably already know you can make your site better, but the problem is, what areas do you need to fix? And how do you know when you’ve fixed them?
When you make changes to your site and go through redesign after redesign, there is no way to know scientifically (oh yeah, we’re talking science) whether or not the increase or decrease in sales is a result of the new changes that you’ve made. You work hard (and spend a lot of money) driving quality traffic to your site. A/B testing is the best way to make sure you are capitalizing on that traffic with actual sales.
For example, let’s say that you want to make a change to your website in an effort to improve your conversion rate. If you’re not A/B testing, this is probably how that discussion goes.
Without A/B Testing
You: “Team, I’ve been doing some research and I found that our competitors only offer a maximum of 30-days for a return policy, and yet we offer a full 60 days. I think we should add some messaging on our product page that lets customers know that we have a 60-day return policy. I believe this change will increase our sales.”
Your Designer: “I’m not sure about that idea. It will totally ruin the white space of the layout.”
Your Boss: “I think this is a great idea, but the messaging should say, ‘Satisfaction Guaranteed!!!!!’ People love guarantees!”
At this point everyone starts to plead their case and perhaps things get a little tense. In the end, the HiPPO (Highest Paid Person’s Opinion) wins out, and you change the product page to have the added messaging, “Satisfaction Guaranteed!!!”
The changes go live and you go back to working on driving quality traffic to your site. A week goes by and your boss says, “Sales look great this week! See, I told you people LOVE guarantees!”
There are quite a few problems with this scenario, but let’s just look at a few:
- Things started off great. You brought research to the team and made a hypothesis that adding messaging would increase sales. The problem is that you had no way to prove your hypothesis.
- When this is the case, people tend to resort to their own agenda (i.e. your designer does not want to add messaging because he or she would like the page to be as clean as possible).
- Your boss wants to make an impact, so he or she decides to go with the idea, but not support the research about return policy messaging.
- When sales are up the next week it is assumed that the sales increase is due to the added messaging, but in reality the sales increase could have resulted for many different reasons (an email newsletter was sent out, your ad spending increased, you ran a sale on a popular item, etc).
Now, let’s breakdown this example with A/B testing.
With A/B Testing
You: “Team, I’ve been doing some research and I found that our competitors only offer a maximum of 30-days for a return policy, and yet we offer a full 60 days. I think we should A/B Test some messaging on our product page that lets customers know that we have a 60-day return policy. I believe this variation will increase our sales.”
Your Designer: “I’m not sure about that idea. It will totally ruin the white space of the layout.”
Your Boss: “I think this is a great idea, but the messaging should say, ‘Satisfaction Guaranteed!!!!!’ People love guarantees!”
You: “At the end of the day, we all want to see more sales. So we will test the control (no messaging, clean layout) vs. 60-day returns messaging and Satisfaction Guaranteed messaging. Can we agree that we will implement whichever version results in the most sales?
Your Team: YES!
Now you can implement the following A/B Test:
Control: No messaging
Variation #1: “Don’t like it? Return it for up to 60 days.”
Variation #2: “Satisfaction Guaranteed.”
There are tons of great case studies about eCommerce stores increasing their revenue with A/B testing. Check out Optimizely’s Customer Stories Page for a detailed look at a few successful tests.
This is Part 1 of a series of blogs on A/B Testing. Check back soon for Part 2 from Josh Frank, Human Element’s very own A/B Testing expert at Test Triggers.
