Experimentation Works
The Surprising Power of Business Experiments
Experimentation Works: The Surprising Power of Business Experiments – a book by Stefan H Thomke is a wonderful book about the power of experiments for businesses. People familiar with IT and digital marketing are aware of A/B testing, which means testing two options to find out which works better. After a long time, finally i was able to finish a book.
A/B testing is the heart and soul of any tech organization. Companies like Google, Amazon, Booking.com, Micrsoft, Apple all use A/B testing extensively to introduce new features and make informed business decision.
Experimentation Works mentions few famous examples that brought millions of revenues to the firms with very simple changes suggested by employee. The book quotes an example of Microsoft Bing (content taken from HBR site.
“In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert. Usually, such alerts signal a bug, but not in this case. An analysis showed that the change had increased revenue by an astonishing 12%—which on an annual basis would come to more than $100 million in the United States alone—without hurting key user-experience metrics. It was the best revenue-generating idea in Bing’s history, but until the test its value was underappreciated.”
You can look at the minor change in the picture below.
Experimentation Works offers framework and step-by-step approach on how to imbibe an experiment driven culture in any organization. Perhaps, the best part of the book is the use of vivid and live examples. Some of the findings are similar to books on Nudge, Habit, and derive from psychology.
Experimentation Works devotes one chapter to breaking the seven common myths about experimentation.
Myth 1: “Experimentation-driven innovation will kill intuition and judgment.” – The two are complements rather than substitutes. In fact, testing is a fast and cheap way to test one’s intuition.
Myth 2: “Online experiments will lead to incremental innovation but not breakthrough performance changes” – In fact, many breakthrough performance changes are motivated by findings from online experiments.
Myth 3: “We don’t have enough hypotheses for large-scale experimentation.” – The more you experiment, the more hypotheses you generate from your experiments’ findings. The largest experimenters started by running only a few experiments per year and most companies still only run very few experiments.
Myth 4: “Brick-and-mortar companies don’t have enough transactions to run experiments.” – While sample size is definitely a valid concern, the author advises such companies to focus on running bigger and riskier experiments. (remember that the larger the treatment effect, the less sample size you need to find statistically significant differences).
Myth 5: “We tried A/B testing, but it had a modest impact on our business performance.” – Experimentation is a highly long-term strategy and many failures are to be expected. In addition, one should make sure to properly account for interaction effects (i.e. if two tests each increase performance by 1%, together they may increase performance by 3%).
Myth 6: “Understanding causality is no longer needed in the age of big data and business analytics. Why waste time on experiments?” – Correlation is not causation, regardless whether you have big data or not. “A superficial understanding of why things happen can be costly, or in the case of medicine, even dangerous”.
Myth 7: “Running experiments on customers without advance consent is always unethical.” – Informing users that they are participating in an experiment may alter behavior. The author restates the “A/B illusion” from chapter 4. Finally he says that “People seem unconcerned with the current practice of being emotionally manipulated through advertising and other means, although the harmful effects of these media may have never been rigorously tested.”
Who should read this book?
Practically anyone who is interested in experiment method and want to improve – be it organization policy, or the color of the Pay now button. In essence, experiments offer a practical way to test the assumptions and hypothesis about what will work and most importantly, what will not. The experiments can run on small data and big data. Most of the experiments return a failed result, which shows that many ideas may not bring substantial value. Hence, incremental changes over a period of time will bring a sustainable revenue growth.
Experimentation Works may help in deciding whether
- an organization should introduce a four days week working culture.
- which communication to use to influence users
- which color attracts more clicks
- where to put that checkout button on screen, and many more trivial decisions.