DocumentCode :
1119460
Title :
Online Experiments: Lessons Learned
Author :
Kohavi, Ron ; Longbotham, Roger
Author_Institution :
Microsoft´´s Exp. Platform, Redmond
Volume :
40
Issue :
9
fYear :
2007
Firstpage :
103
Lastpage :
105
Abstract :
The Web provides an unprecedented opportunity to evaluate proposed changes or new features quickly using controlled experiments. The simplest experiments randomly assign live users to one of two variants: the control, which is commonly the existing version, and the treatment, which is usually a new version being evaluated. The overall evaluation criterion can be a simple metric that summarizes important business goals or a weighted combination of metrics, as is often used in credit scores. Randomization is too important to be left to chance. A common way to maintain user experience consistency is to employ a hashing function on a user ID stored in a cookie. Cryptographic hashes such as MD5 are generally the best. Failure to randomize properly can confound results when running multiple tests simultaneously. The other common mistakes in online experiments are launching a feature that is statistically significantly different but has little business value and launching a feature because it doesn´t negatively impact users. Online experiments, whether they fail or succeed, generate insights that can bring a quick return on investment and promote innovation.
Keywords :
Internet; cryptography; program debugging; program testing; MD5; cryptographic hashing function; online experiments; program debugging; program testing; randomization; user ID; Crawlers; Cryptography; Optimized production technology; Size control; Switches; Technological innovation; Testing; Web technologies; online experiments;
fLanguage :
English
Journal_Title :
Computer
Publisher :
ieee
ISSN :
0018-9162
Type :
jour
DOI :
10.1109/MC.2007.328
Filename :
4302627
Link To Document :
بازگشت