Entropy and the law of small numbers

Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when S_n=Σ_i=1ⁿX_i is the sum of the (possibly dependent) binary random variables X₁,X₂,...,X_n, with E(X_i)=p_i and E(S_n)=λ, then D(P(S_n)||Po(λ)) ≤Σ_i=1ⁿp_i²+[Σ_i=1ⁿH(X_i)-H(X₁,X₂,...,X_n)] where D(P(S_n)||Po(λ)) is the relative entropy between the distribution of S_n and the Poisson (λ) distribution. The first term in this bound measures the individual smallness of the X_i and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution. Second, in the particular case when the X_i are independent, the following sharper bound is established: D(P(S_n)||Po(λ))≤1/λ Σ_i=1ⁿ ((p_i³)/(1-p_i)) and it is also generalized to the case when the X_i are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.