Website Live (Again!) - this time with WIX for A/B...C/D/E testing?
Just as my attention turned to product development we realised that while building a website with GoDaddy website builder was quick because of the handy templates it was just too limited for our needs 🤦 (Details Below). So no sooner than we had created a website, we'd need to recreate it again🤦🤦! Luckily, with the content already written this wasn't such a chore and a couple of days later it's back up, better than ever. 💥
So what couldn't we do? Well, unlike a physical and localised businesses like restaurants or hotels, our sales are digital and international and we're trying to raise awareness of these amazing toilets through purely digital channels**. This means A/B testing everything we do to be as effective (efficient) as possible in reaching the right people and making them understand why these toilets are amazing! GoDaddy didn't have the capability to do different things for our different visitors and keep track of where each subscriber had come from. The simplest example was to allow some visitors to access a content upgrade (downloadable pdf guide) by subscribing, but not everyone.
If we spend money on adverts, we will undoubtedly get more clicks (by accident even if our ad was terrible 😖). But there are a lot of variables even with an advert: Who you show it to, what the ad says and what message you see if you click (eloquently summarised by Erica). So how do you know if something can be improved? Simple - try to improve it and find out: A/B Testing.
Lets say we had 2 adverts, we show both 1000 times randomly (1k impressions each). We then conclude that the advert with the highest click-through-rate (ctr) is higher. Great, but 2k impressions is a lot. So when do we know one advert is better than the other - Statistical Significance. In common sense terms if you toss a coin 10 times and get 10 heads - you'd be a mug to bet on tails for the 11th toss. But what about if you only had 2 or 3 or 4 or 5 heads tosses (observations) to make a decision? If i can find out one advert is better quickly I can save a lot of beer tokens🍺... Luckily clever mathematician types have come up with lots of tools to test the 'statistical significance'.
Enter the multi arm bandit problem. Let's say we have 5 different adverts, we could show each 1k times and see which has the highest ctr. But that means at the end we showed the bad adverts as many times as the best and wasted good money that could be spent on beer! 🍻
Showing an advert is like a coin toss where the chance of getting tails (winning) is very low - People just don't click adverts that often. When you're trying to choose between 5 coins with 2.5%, 3%, 3.5%, 4% and 4.5% chance of a win respectively you're going to need a lot of trials just to see a win. You're going to need lots and lots of trials to have a chance of guessing which coin is the better one! But it can be done more efficiently than A/B testing and for a startup like ours with little traffic and low advertising budgets this matters
For early startups with insufficient user traffic, multi-armed bandit experiment works better because it requires a smaller sample size, terminates earlier, and is more agile than A/B testing.
Source here with lots of great animations!
So the aim is to explore each option enough to know which is the best while exploiting he best option as much as possible. This is the exploration vs exploitation payoff - You know you like strawberry ice cream but have you tried mint choc chip? maybe you'll like it better?
Long story short - each ad is a Bernoulli process which is modelled as a beta distribution. As each advert is presented / clicked (or most likely not) we update our estimate of the beta distribution. We choose which ad to show randomly based on how likely it is to be clicked - the more likely it is to be clicked the more likely it is to be picked. Poorer ads are still picked but with increasingly lower likelihood. We can also test whether each ad is statistically better than another and remove it altogether.
Anyway - I love statistics and could go on all day 🤓, but this is likely to be my most-bounced blog post ever as I'm pretty alone in my love of stats! If you really want to hear more please leave a comment and I'll follow up with more details...
**Arguably, Hotels and restaurants may be doing lots of online advertising too so may want to sharpen their website as well...