Entropy Estimation: Simulation, Theory and a Case Study

We consider the statistical problem of estimating the entropy of finite-alphabet data generated from an unknown stationary process. We examine a series of estimators, including: (1) The standard maximum-likelihood or "plug-in" estimator; (2) Four different estimators based on the family of Lempel-Ziv compression algorithms; (3) A different plug-in estimator especially tailored to renewal processes; and (4) The natural estimator derived from the Context-Tree Weighting method (CTW). Some of these estimators are well-known, and some are new. We first summarize numerous theoretical properties of these estimators: Conditions for consistency, estimates of their bias and variance, methods for approximating the estimation error and for obtaining confidence intervals. Several new theoretical results are developed. We show how the theory offers preliminary indications results offer guidelines for tuning the parameters involved in the estimation process. Then we present an extensive simulation study on various types of synthetic data and under various conditions. We compare their performance and comment on the strengths and weaknesses of the various methods. For each estimator, we develop a precise method for calculating the estimation error based on any specific data set. Finally we report the performance of these entropy estimators on the (binary) spike trains of 28 neurons recorded simultaneously for a one-hour period from the primary motor and dorsal premotor cortices of a quietly seated monkey not engaged in a task behavior. Based on joint work with Yun Gao and Elie Bienenstock.