A/B Testing a Square Peg into a Round Hole
Ever walked into a shoe store and asked the sales person to help you find a shoe with the exact fitting. Did you get exactly what you were looking for – or did you emerge from the store with shoes that were a bit too tight but the sales person convinced you that they would loosen up over time. The most important thing, as the sales person put it, was that the shoes had air soles that were perfect for absorbing the impact of running on your knees. So do you run a lot? Not really, but you figured that it would be great to avoid the impact if you did decide to run.
I call this “selling a square peg into a round hole” and it is far more prevalent than you expect. Car dealers employ this tactic all the time convincing you to buy a different car from the one you want. The car they want to sell you comes with fancy features such as a high priced navigation system when you can easily pick up a portable GPS device for half the price. The idea is not to force the square peg into the round hole, but rather to make it just fit in and then hard-sell features with high value perception. We need to convince the consumer that is worth compromising on the original requirements since the alternative is a much bigger value.
Can your website sell square pegs into round holes? It would depend on two things – is the product close enough to what the consumer wants and are you offering any extras that could seal the deal. The answer to achieving these two objectives is A/B testing and you can carry it out using one of the many website optimization tools that test, record, and measure consumer preferences.
A/B testing is a technique that involves creating an experiment where we put a baseline product in front of consumers and a version of the product with a single significantly different component from the baseline. Measuring which version consumers favor the most provides insight into the relevance of the tested component. You can conduct several such A/B tests with variations in other components. This technique helps identify which of the product attribute(s) have the most influence on response rates.
Finding results for any optimization objective requires you to create a series of experiments, set up manual or automated tests to run them, measure the results, and apply the learning. This is not as hard as it sounds. Free tools such as Google Website Optimizer make A/B tests easy to execute and provide reasonably good results for most websites.
As an example, a website may offer a product for $75 and charge $6 in shipping costs – let us call this baseline as version A. Now we run an experiment where we split the website traffic into two and offer one-half of site visitors the version A, and offer the other half the same product for $75 but with free shipping – call this version B. Which version do you think will have more purchases? Next, run another experiment against version A, but this time offer the other half the same product for $81 with free shipping – call this version C.
Here are sample statistics collected from these two experiments.
A vs B
A vs C
Version B seems like the winner with the highest purchase rate. However, it is also a version in which our business takes a hit on the profit margin by absorbing the shipping costs. Version C has mixed results with a lower add to cart rate, but a higher purchase rate than A. However, it does ensure a higher margin despite the free shipping incentive. We would need to run several such experiments with different price points across a lot of traffic to determine the impact of the shipping cost and find the version driving the highest number of consumers willing to complete the purchase.
Amazon.com is a familiar example of this particular case. They likely conducted a large number of A/B tests to conclude that offering free shipping on orders greater than $25 led to significantly higher purchases while retaining sufficient margins to absorb the costs.
You will also notice that if your order is lower than $25, Amazon.com will suggest you purchase additional items to avail of free shipping – a clear indicator that they identified shipping costs as a big hurdle to converting visitors into purchasing customers. Free shipping is thus one of the key incentives that Amazon uses to sell its square pegs.
Keep in mind that the results of A/B tests will only converge toward the winners and are not absolute. What this means is that the winning version you find is a winner only in the timeframe in which you ran the experiments and the variations that you tested. If you keep running such experiments, you will keep finding new winners since consumer preferences will evolve and change. So, do not look upon A/B testing as a fixed time activity. The more you test, the more you will be able to optimize your results and get closer to filling that round hole.
