The Statistical Foundations of Entropy

The following sections are included:

• DEDICATION
• PREFACE
• CONTENTS
• RELATIONAL NOTATION

Entropy occupies a secure and prominent position on any list of the most fundamental and important concepts of physics, but this trite observation does not fully do it justice. Entropy has a certain solitary mystique all its own, which both fascinates and frustrates those who aspire to comprehend its elusive and multifaceted subtleties. Unlike mass, momentum, and energy, the entropy of an isolated system is not conserved but has the peculiar property of spontaneously increasing to a maximum. A quantity which can only grow larger is curious if not unsettling to contemplate; it vaguely suggests instability, or even some inexorable impending catastrophe worthy of a science fiction movie, perhaps entitled The Heat Death of the Universe! It also seems curious that in spite of its thermodynamic origins, entropy cannot be fully understood in thermodynamic terms alone but requires statistical concepts for its complete elucidation. Indeed, the statistical interpretation of entropy is in many respects simpler and easier to comprehend than its thermodynamic aspects, and arguably provides the most transparent pedagogical approach to the subject. This of course is the rationale for treating thermodynamics and statistical mechanics together as a single discipline, an approach which was once considered heretical but has now become accepted and commonplace.

The following sections are included:

• Entropy as Uncertainty
• Systems and States
• The Boltzmann-Planck Entropy
• The Boltzmann-Gibbs-Shannon Entropy
• Relative Entropies
• Continuous States and Probability Densities
• Systems with an Infinite Number of States

At this point the task of defining the entropy has been completed, but its full significance has yet to be explored, and the task of actually evaluating it has barely begun. Equation (2.10) expresses the BGS entropy S of a system with discrete states as a function of their probabilities p_k, so S cannot be evaluated until values have been assigned to those probabilities. This evaluation is greatly simplified in the case when all the probabilities are known or presumed to be equal; i.e., when p_k = 1/W. The BGS entropy then reduces to the BP entropy of Eq. (2.8), which can be evaluated simply by counting the states. Similarly Eq. (2.19) expresses the BGS entropy of a system with continuous states as a functional of the probability density p(x), which likewise reduces to the BP entropy of Eq. (2.8) when the continuous states are equally probable; i.e., when p(x) = p₀(x) = (1/W) ρ(x), with W given by Eq. (2.23). In either case, the assumption that the discrete or continuous states of a particular system are equally probable (perhaps subject to certain conditions or restrictions) is traditionally referred to as the hypothesis of equal a priori probabilities (EAPP). This hypothesis clearly effects an enormous simplification when W is large. It also implies that the BGS entropy of Eq. (2.10) or (2.19) has its maximum possible value, as discussed in Chapter 2.

In most situations of physical interest, the number of accessible states W and their probabilities p_k are not fixed constants but rather depend on the values of a relatively small number of externally controlled parameters. Such parameters are typically associated with restrictions imposed on the system in order to (a) ensure that the number of accessible states is finite, as discussed in Sect. 2.7, and/or (b) confine the system to a subset of states for which the EAPP hypothesis is presumed to be valid, as discussed in Chapter 3. In this chapter we shall presume that conditions (a) and (b) are both satisfied, so that the number of accessible states W is finite and their probabilities p_k are equal with the common value p_k = 1/W. The entropy of the system is then the BP entropy S = log W. The functional dependence of W and S on whatever parameters are associated with conditions (a) and (b) is understood and must be remembered, but in this chapter it will be suppressed from the notation for simplicity. The purpose of this chapter is to develop a suitable mathematical framework to explore and describe the implications of imposing additional constraints on the system. The resulting constrained values of W and S then become functions of the parameters associated with those constraints. The leitmotiv of the discussion is that those functions conversely determine the probability distributions or densities of those constraint parameters and their fluctuations in the unconstrained system. The resulting expressions for the probability densities are generalizations of those first used by Einstein to describe thermodynamic fluctuations.

The following sections are included:

• Motivation
• Generalized Canonical Probability Distributions
  • Discrete States
  • Continuous States
• A Premonition of Thermodynamics
• Jaynes’ Proof of the Second Law

In the preceding chapters, the mathematical fundamentals of entropy have been deliberately developed in a general form in which the nature of the system and its states have been left unspecified, with only occasional intimations (mainly in Sect. 5.3) of the possible physical applications of the formalism. This approach was adopted in order to preserve maximum generality for as long as possible, thereby making it easier to clearly distinguish between general features and those which are specific or peculiar to a particular context. For example, the treatment of constraints in Chapter 4 shows that Einstein fluctuation formulae, which are commonly used to describe thermodynamic fluctuations, are not restricted to thermodynamic systems but are actually more general. We now proceed to systematically explore the application of the preceding general concepts to macroscopic thermodynamic systems. The brief prelude to such systems in Sect. 5.3 was merely an hors d’oeuvre based on the BGS entropy and the PME, which was made plausible but in a logical sense constitutes an additional postulate. In contrast, the more detailed treatment in this chapter is based entirely on the BP entropy and the EAPP hypothesis, and is thereby more fundamental with fewer assumptions. As we shall see in Chapter 7, however, both approaches ultimately lead to essentially the same results, a circumstance which provides compelling further evidence in support of the PME.

The terms microcanonical, canonical, and grand canonical were coined by Gibbs [17] to describe systems which are respectively either isolated, in thermal equilibrium, or in thermal and diffusional equilibrium with a reservoir. Gibbs applied those adjectives to the term ensemble, which denotes an imaginary collection of a very large number of mental replicas of a particular system of interest, in which the number of fictitious systems that occupy each state is proportional to its probability. The concept of an ensemble thenceforth became a conventional and well established feature of statistical mechanics, which however is unfortunate in some respects. Gibbs’ benevolent intention was merely to facilitate the visualization of probability distributions by those unfamiliar with statistical concepts [20, 37]. Nowadays, however, students are normally introduced to basic probability theory well before they encounter statistical mechanics, and the concept of an ensemble has become a historical anachronism which is more likely to create confusion than to confer enlightenment. It seems preferable to focus attention directly on the system of interest and the uncertainty as to which of its states it occupies [37]. To this end and in that spirit, we eschew the term “ensemble” and use the terms microcanonical, canonical, and grand canonical exclusively to refer to the probability densities and distributions that describe the particular system of interest.

Many of the most interesting and important applications of entropy are to composite systems composed of simpler subsystems which either are, or can be thought of as, various types of particles. It is therefore convenient to refer to them as such, with the understanding that in some cases it may be appropriate to interpret the term “particle” in a generalized sense. The number of particles N in such systems is typically very large, so they are commonly referred to as “many-particle” systems, but it is well to keep in mind that the concept of entropy (and consequently much of the general formalism developed prior to Chapter 6) is equally applicable to both simple and composite systems of any type or number of subsystems, including particulate systems of small N. In addition to being of intrinsic interest in their own right, systems of small N provide the clearest illustration of the additional concepts required to apply the general formalism of entropy to systems of arbitrary N, including macroscopic thermodynamic systems containing N ˜ 10²³ particles.

The preceding development has been specifically restricted to systems of identical indistinguishable particles, but no further conditions were imposed on the particles or their microstates. The relations derived in Chapter 8 are therefore equally applicable to systems of identical particles in both classical and quantum mechanics, provided the particles are regarded as indistinguishable. As discussed in the Preface and Introduction, it is our firm conviction that identical particles, by their very nature, are indeed inherently and unconditionally indistinguishable. If that proposition is accepted, the relations of Chapter 8 combine with the results of previous chapters to provide a unified general statistical formalism which applies to both classical and quantum particles. However, the appealing simplicity of this picture is undermined by the widespread misconception that identical classical particles are distinguishable, at least in principle, and must be treated as such. In this section we discuss the apparent origins and rationale for this unfortunate misconception in an attempt to reconstruct and rectify the fallacious notions on which it is based.

The following sections are included:

• BIBLIOGRAPHY
• REFERENCES
• INDEX