A/B testing & Machine Learning

Concept days: every Tuesday, we explain in our own words a new trendy concept around one of our favorite four themes – start-ups, customer behavior, e-commerce and technology.

Tuesday’s concept #2 : « A/B Testing & Machine Learning »
What to use when ?
themes : consumer behavior & technology

A/B testing is a well known method to check what pleases your customers and identify user-experience improvements. In short, you separate your users in two groups, one sees the same old page, the second sees the experimental page, you gather statistics, and after a while you decide whether the user interface improvement is worthwhile. This approach is great as it does not leave your user interface decisions to random, rather it helps you build arguments for every change.

You can’t A/B test everything!

User interface is a good example of something you can A/B test, because you have time to gather statistics and build your arguments for or against. However, it does not a good job in helping you decide the ordering of products to show to a customer. First, there is an opportunity cost to step back and wait to see the aggregated statistics. Second, your inventory changes constantly and new products are coming in. Finally, why choose only one ordering for all customers unvariably?

Use machine learning

Work instead 1-to-1, giving each customer what he came looking for. Machine learning is here to help you tune your product display algorithms and makes them intelligent enough to learn from each customer individually. It will take into account product view, putting a product in a shopping basket, preferred brands or catégories. In other words: the algorithms are trained to understand a customer, to learn from any changes in his behavior and adapt continuously.

How to choose your technology?

Do you have time to test on a small group, sit-back, analyze the results and launch on a large scale? Then you should use A/B testing. A good example is emailing: you can send two different e-mails with two different catch phrases to two samples, you see which one actually convert more, and then choose the right email to send to everyone. You have to decide constantly on what to show? You should use machine learning, to learn from the past and give each customer a unique experience. A good example is showing a list of product: when deciding the order of products to show, products that appeal more to the customer will be shown on top.

What next?

You can also run experiments continuously and take advantages of both worlds, running Multi-Armed Bandits experiments. The original problem statement comes from deciding which slot machine to play from a pool of machines in a casino. The principle is simple, you run multiple experiments at the same time, and you use the statistics gathered to feed directly into the decision process of what experiment to promote or demote, until one experiment wins. The whole point is that you limit your opportunity cost of running bad experiments and maximize your gains quickly.

Share your opinions or questions with us, we are interested!
Or send us a mail at : commerce@ezako.com

See you next Tuesday !

You would like us to write about something? Send us a mail: commerce@ezako.com or contact us via our facebook page EZAKO, or twitter account @BigDataEzako