Using statistical power analysis to tune-up a research experiment: a case study

This work presents a case study of the use of statistical power analysis in a research study. When the University of Washington researchers ran a pilot study to investigate the effect of link wording on Web site browsing behavior and comprehension, they obtained results that were not significant on most dependent measures. To analyze the results and discover whether link wording really had no effect, they first turned to statistical power analysis to see whether they might be committing a type II error (accepting a false hypothesis). They did in fact find that the power of the study was too low and the number of participants too few. This work explains how they used the results of the power analysis to redesign the study and increase its power and the likelihood of obtaining significant results if true between-group differences did in fact exist.