The Method of Maximum Entropy

The following sections are included:

• Preface

• Table of Contents

The Method of Maximum Entropy is an offspring of the Maximum Entropy Principle introduced in 1957 in statistical physics by E. Jaynes. That principle has the esthetic appeal of all variational principles in physics and its basic role is to characterize equilibrium states. It works as follows: a functional which is a Lyapunov function for the dynamics of the system is defined on the set of states. It is postulated that the equilibrium states are those yielding a maximum value of the functional, compatible with a set of given values of some extensive variables…

We will recall some basic concepts from measure theory and from probability theory. The purpose of this chapter is to provide applied and other scientists with some standard vocabulary. Most of the concepts and results are intuitive and obvious at times, even though the proper names are not widely known…

Statistical physics owes its birth to the inconvenience and impossibility of describing systems having very large numbers of particles by specifying the behavior of each individual. Loosely speaking, the aim of statistical physics is to describe the “collective” or “macroscopic” properties of a system of particles in terms of appropriate averages of its “microscopic” motions…

Here we follow Jaynes [3.1] and Papoulis [3.2] in developing some heuristics that sheds light on the concept of entropy and on the method of maximum entropy.

Let X be a discrete random variable taking n values x₁,x₂,…,x_n and let p₁ be the probabilities of observiong the events A₁ = {X = x_i}…

The following sections are included:

• Basics

• Entropy inequalities

• Axiomatic characterization of entropies

• REFERENCES

The following sections are included:

• Kullback's and Jaynes' reconstruction methods

• Czizar's results

• Borwein and Lewis's extensions

• Dacunha-Castelle and Gamboa's approach to level 2 M.E.M.

• APPENDIX

• REFERENCES

The following sections are included:

• Entropy maximization under quadratic constraints, or constraint relaxation

• Failure of maximum entropy methods for reconstruction in infinite systems

• Some finite dimensional, linear reconstruction problems

• Maxentropic approach to linear programming

• Entropy as Lyapunov functional. Further comments

• Solving matrix equations

• Estimation of transition probabilities

• Maxentropic reconstruction of velocity profiles

• Fragmentation in a nuclear reaction

• Maxentropic inversion of Laplace transforms

• Maxentropic inversion of Fourier transforms

• Maxentropic spectral estimation

• Maxentropic solution of integral equations

• Maxentropic image reconstruction

• An application in systems analysis

• Distributions with preassigned marginals and related problems

• Maxentropic approach to the moment problem

• Maxentropic taxation policies

• The Stieltjes moment problem

• The Hamburger moment problem

• Applications to data analysis

• REFERENCES

The following is taken almost literally from [7.1]. It comprises the very basic results in the theory of large deviations. In that reference you will find quite a lot about the subject and its applications to statistical mechanics. Also, check with [7.2] for more…

In this chapter we present a very interesting way of “understanding” how do maxentropic distributions arise. We go through the simplest situation, lifting the results from [8.1-3] but direct the reader to [6.41], [8.2], [8.3] and especially [8.4] for history, different and more comprehensive results. It should also become apparent that the results here and in the previous chapter are deeply connected…

The following sections are included:

• Gauss principle and minimum discrimination

• Sufficiency

• Some very elementary Bayesian statistics

• REFERENCES