A/B Testing

A/B testing is a way of comparing two versions of a website or application in order to determine which one performs better. The comparison is done by showing users the two variants (let’s call them A and B) at the same time. Some users get version A and others get version B. The winner is the one with a better conversion rate over the same time (to reduce time dependent factors).

All websites on the Internet have a reason to perfom A/B tests

• eCommerce websites want visitors to buy their products
• SaaS web applications want visitors to sign up for a trial and convert them to paid visitors
• News and media websites want readers to click on ads or sign up for paid subscriptions

Every single business website wants visitors to convert from just visitors to something more. A key indicator for that is the "conversion rate". By applying A/B test to a website we essentially optimise this rate that converts visitors to goal achievers.

Incremental Testing vs. Multivariant Testing

The incremental approach has an advantage over multivariate approach because it detects which variant has a better conversion rate as soon as a credible result is received.

For example, you would like to test four ideas of where a newsletter subscription popup can be placed. In order to decide you start to test them. In a multivariate approach you set up four experiments at the same time and let them run simultaneously. The traffic is therefore split into four streams and you do not improve until you get a winner. On the other hand, using the incremental approach, you set up an experiment between first and second versions of a popup. You conduct an A/B Testing and the winner is going to be tested against the third version. Then this winner is going to be tested against the fourth version. That way the traffic is always split into two streams and you are able to improve your website along the way to the final winner. That way you will save time and money!

We strongly advise you to go with the incremental rather than the multivariant approach.

Example