Stochastic Processes With Random Contexts: A Characterization and Adaptive Estimators for the Transition Probabilities

This paper introduces the concept of random context representations for the transition probabilities of a finite-alphabet stochastic process. Processes with these representations generalize context tree processes (also known as variable length Markov chains), and are proved to coincide with processes whose transition probabilities are almost surely continuous functions of the (infinite) past. This is similar to a classical result by Kalikow about continuous transition probabilities. Existence and uniqueness of a minimal random context representation are shown, in the sense that there exists a unique representation that looks into the past as little as possible in order to determine the next symbol. Both this representation and the transition probabilities can be consistently estimated from data, and some finite sample adaptivity properties are also obtained (including an oracle inequality). In particular, the estimator achieves minimax performance, up to logarithmic factors, for the class of binary renewal processes whose arrival distributions have bounded moments of order 2 + γ.