As an online marketer, you are probably familiar with AxB testing. The trend is tests based on “multi-armed bandit” systems. It looks like it is a more intelligent way then AxB testing.


We tend to stumble upon articles that go on about whether alternative intelligent A/B testing systems are actually useful, what you can use them for and what they are based on. First an introduction.

What are the so-called “multi-armed bandit” systems?

The simplest way to describe them is as an analytic and statistical mind that observes and does up to date corrections to our test results. When testing two, three or more options in this way, you can always be sure that the one that generates the most success at a given time is the option to which most of the objects that we test are directed to – in other words, potential clients. Thanks to this method, theoretically speaking, we should lose less clients during testing because, most of them are directed to the solution that gives more conversions (according to the experiment). Everything sounds simple enough, however, make a note that there are advanced statistical analysis going on in the background. They calculate the probability of a certain occurrence (i.e. the purchase of a product) in current conditions (i.e. red “buy” button). The idea of my statement isn’t to go through complex statistical formulas. In order to simplify the idea we can illustrate it like this (using 2 landing pages as an example): when we separate tested objects into 2 equal parts in A/B testing, you place them in their corresponding landing pages and wait for final results. In the multi-armed bandit, a given landing page’s number of visits is directly proportional to how much better it is than the other landing page that is being tested. If one generates 60% of conversions, then 60% of all users will be directed to this landing page. Thanks to this we can have a peaceful mind, while our system decides on its own which solution is better and positions it accordingly.

While analyzing multi-armed bandit type tests you can notice two things. Firstly, during  the test, we lose less potential clients. Secondly, in order to achieve interchangeable results that allow you to observe which of the tested solutions are better you will have to test them longer than in the case of A/B testing. That is why it is best to figure out in which situations A/B testing is more suitable and in which the multi-armed bandit. AxB testing be applied when doing single tests once in a while, whereas the multi-armed bandit should evoke better results when making many ongoing changes while changing our testing options.

If you would like to have more information about this article then please contact me by clicking on this link.

Leave a Reply

Your email address will not be published.