Controlled experiments for decision-making in e-Commerce search

With the explosion of big data, companies both small and large are increasingly motivated to make data-driven decisions. For web-based companies in particular online controlled experiments or A/B tests have become essential scientific tools for decision-making. Large scale organizations like Google, Amazon, eBay, Facebook, LinkedIn, Yahoo, and Microsoft have built mature systems and support for controlled experiments and have helped to popularize the methodology of A/B testing for guiding product development. In e-Commerce, A/B tests are used extensively to understand how customers respond to new features and to use statistics at scale to make decisions rather than relying on the highest paid person´s opinion. However, unlike other web-based businesses, e-Commerce exhibits particular qualities and behaviors where generalized controlled experimentation guidelines are not nuanced enough to provide meaningful insight into decision-making. Specific functions of an e-Commerce website present further complexity in testing, especially in essential and highly visible functions like search. Per week, more than 260 million customers visit Walmart´s retail units and e-Commerce websites. A single decision can result in not only large financial repercussions but also cascading effects across such an expansive customer base that traverses between digital and physical purchase points. For the retail sector at large, e-Commerce is a new business as most operations are still primarily physical. The transformation to web-focused purchase points is burgeoning but the application of data-driven decision-making remains difficult. Generalized guidelines from technology companies have not been able to fully serve the problems specific to e-Commerce. In this paper, we discuss our experiences in running controlled experiments at WalmartLabs Search and specific guidelines in bias, test design, measurement, analysis, financial constraint and decision-making for e-Commerce. We share examples and key lessons from our work in each of these focal areas where we believe the reader will benefit from interpreting real results. Our work provides the growing e-Commerce analytics community with guidelines for running experiments within their own retailers and highlights sector specific challenges in the application of controlled experiments.