Information theory for atypical sequences

One characteristic of the information age is the exponential growth of information, and the ready availability of this information through networks, including the internet - “Big Data.” The question is what to do with this enormous amount of information. One possibility is to characterize it through statistics - think averages. The perspective in this paper is the opposite, namely that most of the value in the information is in the parts that deviate from the average, that are unusual, atypical. Think of art: the valuable paintings or writings are those that deviate from the norms, that are atypical. The same could be true for venture development and scientific research. The paper first discusses what exactly should be understood by “atypical.” This is by no means straightforward. It has to be a well defined theoretical concept corresponding to some intuitive idea of atypicality, which when applied gives useful results. This is followed by a simple example of iid binary sequences. This example is simple enough that complete algorithms can be developed and analyzed, which give insights into atypicality. We finally develop a more general algorithm based on the Context Tree Weighing algorithm and apply that to heart rate variability.