Information Theory and the Central Limit Theorem

The following sections are included:

• Preface

• Contents

This chapter contains a review of some results from information theory, and defines fundamental quantities such as Kullback-Leibler distance and Fisher information, as well as giving the relationship between them. We review previous work in the proof of the Central Limit Theorem (CLT) using information-theoretic methods. It is possible to view the CLT as an analogue of the Second Law of Thermodynamics, in that convergence to the normal distribution will be seen as an entropy maximisation result.

In this chapter we show how a proof of convergence in Fisher information distance and relative entropy distance can be carried out. First we show how our techniques are motivated via the theory of projections, and consider the special case of normal distributions. We then generalise these techniques, using Poincaré inequalities to obtain a sandwich inequality. With care this allows us to show that convergence occurs at a rate of O(1/n), which is seen to be the best possible.

In this chapter we show how our methods extend to the case of non-identical variables under a Lindeberg-like condition and to the case of random vectors. Many of the same techniques can be applied here, since the type of projection inequalities previously discussed will also hold here, and we can define higher dimensional versions of Poincaré constants.

Having dealt with the independent case, we show how similar ideas can prove convergence in relative entropy in the dependent case, under Rosenblatt-style mixing conditions. The key is to work with random variables perturbed by the addition of a normal random variable, giving us good control of the joint and marginal densities and hence the mixing coefficient. We strengthen results of Takano and of Carlen and Soffer to provide entropy-theoretic, not weak convergence. It is important that such results hold, since the independent case is not physical enough to give a complete description of interesting problems.

In this chapter we discuss some aspects of convergence to stable distributions. These are hard problems, not least because explicit expressions for the densities are only known in a few special cases. We review some of the theory of stable distributions, and show how some of the techniques used in earlier chapters will carry forward, with potential to prove convergence to the Cauchy or other stable distributions. In particular, we will show how certain partial differential equations imply a new de Bruijn-like identity for these variables. Whilst the results in this chapter do not give a complete solution to the problem, we will discuss how our techniques have some relevance.

We investigate the behaviour of the entropy of convolutions of independent random variables on compact groups. We provide an explicit exponential bound on the rate of convergence of entropy to its maximum. Equivalently, this proves convergence in relative entropy to the uniform density. We prove that this convergence lies strictly between uniform convergence of densities (as investigated by Shlosman and Major), and weak convergence (the sense of the classical Ito-Kawada theorem). In fact it lies between convergence in L^1+ϵ and convergence in L¹.

In this chapter we describe how our methods can solve a different, though related problem, that of the 'law of small numbers' convergence to the Poisson. We define two analogues of Fisher information, with finite differences replacing derivatives, such that many of our results will go through. Although our bounds are not optimally sharp, we describe the parallels between Poisson and normal convergence, and see how they can be viewed in a very similar light.

In this chapter we discuss free probability, a particular model of non-commutative random variables. We show how free analogues of entropy and Fisher information can be defined, and how some of our previous methods and techniques will apply in this case. This offers an alternative view of the free Central Limit Theorem, avoiding the so-called R-transform.

The following sections are included:

• Appendix A Calculating Entropies
  • Gamma distribution
  • Stable distributions

• Appendix B Poincaré Inequalities
  • Standard Poincaré inequalities
  • Weighted Poincaré inequalities

• Appendix C de Bruijn Identity

• Appendix D Entropy Power Inequality

• Appendix E Relationships Between Different Forms of Convergence
  • Convergence in relative entropy to the Gaussian
  • Convergence to other variables
  • Convergence in Fisher information

• Bibliography

• Index