Stochastic Control for Smart Grid Users With Flexible Demand

In this paper, we study the optimal control problem for the demand-side of the smart grid under time-varying prices with general structures. We assume that users are equipped with smart appliances that allow delay in satisfying demands, and one central controller that makes energy usage decisions on when and how to satisfy the scheduled demands. We formulate a dynamic programming model for the control problem. The model deals with stochastic demand arrivals and schedules the demands based on their own allowable delays, which are specified by users. However, the dynamic programming model encounters the “curses of dimensionality” and some other difficulties, thus is hard to solve. We develop a decentralization-based heuristic first, and also propose an approximation approach based on Q-learning. Finally, we conduct numerical studies on a testing problem. The simulation results show that both the Q-learning and the decentralization based heuristic approaches work well, but they have their own advantages and disadvantages under different scenarios. Lastly, we conclude the paper with some discussions on future extension directions.