Setting a standard for electricity pilot studies

In-home displays, dynamic pricing, and automated devices aim to reduce residential electricity use—overall and during peak hours. We present a meta-analysis of 32 studies of the impacts of these interventions, conducted in the US or Canada. We find that methodological problems are common in the design of these studies, leading to artificially inflated results relative to what one would expect if the interventions were implemented in the general population. Particular problems include having volunteer participants who may have been especially motivated to reduce their electricity use, letting participants choose their preferred intervention, and having high attrition rates. Using estimates of bias from medical clinical trials as a guide, we recalculate impact estimates to adjust for bias, resulting in values that are often less than half of those reported in the reviewed studies. We estimate that in-home displays were the most effective intervention for reducing overall electricity use (~4% using reported data; ~3% after adjusting for bias), while dynamic pricing significantly reduced peak demand (~11% reported; ~6% adjusted), especially when used in conjunction with home automation (~25% reported; ~14% adjusted). We conclude with recommendations that can improve pilot studies and the soundness of decisions based on their results.