Nonlinear Mixture Models

The following sections are included:

• Dedication
• Contents
• List of Tables
• List of Figures
• Acknowledgments

In this book we study Bayesian analysis of nonlinear hierarchical mixture models with a finite but possibly unknown number of components. Our approach will be based on Markov chain Monte Carlo (MCMC) methods. Application of our methods will be directed to problems in gene expression analysis and population pharmacokinetics in which these nonlinear mixture models arise very naturally. For gene expression data, one application will be to determine which genes should be associated with the same component of the mixture (clustering problem). For population pharmacokinetics, the nonlinear mixture model, based on previous clinical data, becomes the prior distribution for individual therapy. Bayesian analysis of prediction and control can then be performed. From a mathematical and statistical point of view, these are the problems analyzed in this book:

• Theoretical and practical convergence problems of the MCMC method.
• Clustering problems.
• Determination of the number of components in the mixture.
• Computational problems associated with likelihood calculations.
• Comparison of parametric and nonparametric Bayesian approaches…

In this chapter, we will discuss methods for calculating the posterior distribution π(Φ|Y) for a parameter vector Φ conditional on data Y , as well as functionals of this distribution . We will concentrate on the extremely interesting and difficult case of nonlinear hierarchical mixture models. Until recently, there was no practical way to calculate π(Φ|Y) and F(π) for such complex models. Development of the Markov chain Monte Carlo (MCMC) methodology has drastically changed this situation. In this chapter, we introduce fundamental notions of MCMC theory that we are going to use in this book (Section 2.1). In Section 2.2 we introduce general nonlinear normal hierarchical models. Section 2.3 is devoted to the description of Gibbs sampling in the framework of MCMC. This chapter is concluded with a discussion of selection of prior distributions for linear and nonlinear models (Section 2.4). We will see that the model defined by a Dirichlet process in Chapter 6 has much in common with the finite mixture model with a bounded but unknown number of components.

We start this chapter with a description of the permutation invariance properties of mixture models as well as the label-switching effect (Section 3.1). Next we address the main convergence issues of Markov chains and the methods of convergence diagnostics (Section 3.2). Then we describe the random permutation sampler algorithm which utilizes the properties of label switching to determine convergence of the Markov chain (Section 3.3). However, label switching does not allow determination of component-specific parameters. Consequently, we then describe algorithms that try to minimize the influence of label switching: methods of re-parametrization (Section 3.4) and relabeling (Section 3.5).

In this chapter we discuss methods and problems related to the treatment of mixture models with an unknown number of components. The scope of the problem and terminology are outlined in Section 4.1. Next we describe a method of selection of the optimal number of components using weighted Kullback–Leibler distance (Section 4.2) and Stephens' birth–death MCMC (Markov chain Monte Carlo) algorithm (Section 4.3). Finally, we propose a novel method (Section 4.4) for finding the number of mixture components.

In this chapter, we present several examples of how birth–death MCMC (BDMCMC), Kullback–Leibler MCMC (KLMCMC), and random permutation sampler (RPS) deal with the problem of the selection of the optimal number of mixture components. As examples we consider: Galaxy dataset (Section 5.1), nonlinear simulated mixture dataset (Section 5.2), two-component Boys and Girls dental measurements dataset (Section 5.3), one-compartment pharmacokinetics model (Section 5.4), and seed development dataset (Section 5.5). In referring to the algorithms used we mean specifically: BDMCMC – Algorithm 4.1, KLMCMC – Algorithm 4.2, RPS – Algorithm 3.1, Stephens' relabeling – Algorithm 3.2 for linear normal mixture models, otherwise Algorithm 3.3. Our goal in each case is to determine the number of mixture components and to estimate the unknown values of component-specific parameters.

In the previous chapters we have studied parametric mixture models where the mixture components were continuous parameterized densities. In this chapter we consider the nonparametric case where the mixture model is a sum of delta functions. We consider both maximum likelihood and Bayesian cases…

In this chapter, we discuss the application of algorithms developed in the previous chapters to different datasets and present our Kullback–Leibler clustering (KLC) method for the treatment of data-rich problems. We start with a description of previously developed clustering methods and the motivation for the development of this approach (Section 7.1). We show the application of KLMCMC (KL Markov chain Monte Carlo) to gene expression data (Section 7.2). Next, we describe a novel algorithm for selection of the optimal number of components in Section 7.3, and demonstrate utility of this method (Section 7.4). We conclude this chapter with the application of the nonparametric Bayesian (NPB) method to the prediction of transcription start sites (Section 7.5).

The following sections are included:

• Appendix A: Standard Probability Distributions
• Appendix B: Full Conditional Distributions
  • Binomial/Beta
• Appendix C: Computation of the Weighted Kullback–Leibler Distance
  • Weighted Kullback–Leibler Distance for a Mixture of Univariate Normals
  • Weighted Kullback–Leibler Distance for a Mixture of Multivariate Normals
  • Weighted Kullback–Leibler Distance for a Mixture of Beta Distributions
• Appendix D: BUGS Codes
  • BUGS Code for the Eyes Model
  • BUGS Code for the Boys and Girls Example
  • BUGS Code for Thumbtack Data
  • BUGS Code for Eye-Tracking Data
  • BUGS Code for PK Example, Nonparametric Bayesian Approach with Stick-Breaking Priors
• Bibliography
• Index