ArticlesNo Access

GAUSSIAN PROCESSES FOR MACHINE LEARNING

MATTHIAS SEEGER

Department of EECS, University of California at Berkeley, 485 Soda Hall, Berkeley CA 94720-1776, USA

Previously at: Institute for Adaptive and Neural Computation, University of Edinburgh, UK.

https://doi.org/10.1142/S0129065704001899Cited by:890 (Source: Crossref)

Abstract

Gaussian processes (GPs) are natural generalisations of multivariate Gaussian random variables to infinite (countably or continuous) index sets. GPs have been applied in a large number of fields to a diverse range of ends, and very many deep theoretical analyses of various properties are available. This paper gives an introduction to Gaussian processes on a fairly elementary level with special emphasis on characteristics relevant in machine learning. It draws explicit connections to branches such as spline smoothing models and support vector machines in which similar ideas have been investigated.

Gaussian process models are routinely used to solve hard machine learning problems. They are attractive because of their flexible non-parametric nature and computational simplicity. Treated within a Bayesian framework, very powerful statistical methods can be implemented which offer valid estimates of uncertainties in our predictions and generic model selection procedures cast as nonlinear optimization problems. Their main drawback of heavy computational scaling has recently been alleviated by the introduction of generic sparse approximations.^13,78,31 The mathematical literature on GPs is large and often uses deep concepts which are not required to fully understand most machine learning applications. In this tutorial paper, we aim to present characteristics of GPs relevant to machine learning and to show up precise connections to other "kernel machines" popular in the community. Our focus is on a simple presentation, but references to more detailed sources are provided.

Keywords:

References

P. Abrahamsen, A review of Gaussian random fields and correlation functions, Technical Report (Norwegian Computing Centre, 1997) . Google Scholar
R. J. Adler , Geometry of Random Fields ( John Wiley & Sons , 1981 ) . Google Scholar
N. Aronszajn, Trans. Amer. Math. Soc. 68(3), 337 (1950). Crossref, Web of Science, Google Scholar
D. Barber and C. Bishop, Advances in Neural Information Processing Systems 10, eds. M. Jordan, M. Kearns and S. Solla (MIT Press, 1998) pp. 395–401. Google Scholar
S. Becker, S. Thrun and K. Obermayer (eds.), Advances in Neural Information Processing Systems 15 (MIT Press, 2003), to appear . Google Scholar
P. Billingsley , Probability and Measure 3 ( John Wiley & Sons , 1995 ) . Google Scholar
S. Boyd and L. Vandenberghe , Convex Optimization ( Cambridge University Press , 2002 ) , www.stanford.edu/~boyd/cvxbook.html . Google Scholar
C. Burges, Data Mining and Knowledge Discovery 2(2), 121 (1998). Crossref, Web of Science, Google Scholar
K. L. Chung , A Course in Probability Theory , 2nd edn. ( Academic Press , 1974 ) . Google Scholar
C. Cortes, P. Haffner and M. Mohri, Rational kernels, in Becker et al.,5 (To appear) . Google Scholar
C. Cressie , Statistics for Spatial Data , 2nd edn. ( John Wiley & Sons , 1993 ) . Crossref, Google Scholar
L. Csató, E. Fokoué, M. Opper, B. Schottky and O. Winther, Efficient approaches to Gaussian process classification, in Solla et al.,72 pp. 251–257 . Google Scholar
L. Csató and M. Opper, Neural Computation 14, 641 (2002). Crossref, Medline, Web of Science, Google Scholar
G. Cybenko and M. Berry, SIAM J. Matrix Anal. Appl. 11, 499 (1990). Crossref, Web of Science, Google Scholar
T. Dietterich , S. Becker and Z. Ghahramani (eds.) , Advances in Neural Information Processing Systems 14 ( MIT Press , 2002 ) . Crossref, Google Scholar
16. M. N. Gibbs, Bayesian Gaussian Processes for Regression and Classification, (PhD thesis, University of Cambridge, 1997) . Google Scholar
A. Girard, C. Rasmussen and R. Murray-Smith, Gaussian process priors with uncertain inputs — Application to multiple-step ahead time series forecasting, in Becker et al.,5 (To appear) . Google Scholar
P. J. Green and B. Silverman , Nonparametric Regression and Generalized Linear Models , Monographs on Statistics and Probability ( Chapman & Hall , 1994 ) . Crossref, Google Scholar
G. Grimmett and D. Stirzaker , Probability and Random Processes , 3rd edn. ( Oxford University Press , 2001 ) . Crossref, Google Scholar
P. Halmos , Introduction to Hilbert Space and the Theory of Spectral Multiplicity ( Chelsea, New York , 1957 ) . Google Scholar
D. Haussler, Convolution kernels on discrete structures, Technical Report UCSC-CRL-99-10, (University of California, Santa Cruz, July 1999). See http://www.cse.ucsc.edu/~haussler/pubs.html . Google Scholar
S. Ihara , Information Theory for Continuous Systems , 1st edn. ( World Scientific , 1993 ) . Link, Google Scholar
T. S. Jaakkola and D. Haussler, Exploiting generative models in discriminative classifiers, in Kearns et al.,26 pp. 487–493 . Google Scholar
T. Jaakkola and D. Haussler , Probabilistic kernel regression models , Workshop on Artificial Intelligence and Statistics ( ) , eds. D. Heckerman and J. Whittaker ( Morgan Kaufmann , 1999 ) . Google Scholar
T. Jaakkola, M. Meila and T. Jebara, Maximum entropy discrimination, in Solla et al.,72 pp. 470–476 . Google Scholar
M. Kearns , S. Solla and D. Cohn (eds.) , Advances in Neural Information Processing Systems 11 ( MIT Press , 1999 ) . Google Scholar
G. Kimeldorf and G. Wahba, Annals of Mathematical Statistics 41, 495 (1970). Crossref, Web of Science, Google Scholar
A. N. Kolmogorov , Foundations of the Theory of Probability , 2nd edn. ( Chelsea, New York , 1933 ) . Google Scholar
R. I. Kondor and J. Lafferty , Diffusion kernels on graphs and other discrete input spaces , Int. Conf. Machine Learning ( ) , eds. C. Sammut and A. Hofmann ( Morgan Kaufmann , 2002 ) . Google Scholar
D. Krige, J. Chemical, Metallurgical and Mining Society of South Africa 52, 119 (1951). Web of Science, Google Scholar
N. D. Lawrence, M. Seeger and R. Herbrich, Fast sparse Gaussian process methods: The informative vector machine, in Becker et al.,5 See www.cs.berkeley.edu/~mseeger . Google Scholar
T. Leen , T. Dietterich and V. Tresp (eds.) , Advances in Neural Information Processing Systems 13 ( MIT Press , 2001 ) . Google Scholar
D. MacKay, Bayesian Methods for Adaptive Models, (PhD thesis, California Institute of Technology, 1991) . Google Scholar
D. MacKay, ASHRAE Transactions 100, 1053 (1994). Google Scholar
G. Matheron, Economic Geology 58, 1246 (1963). Crossref, Google Scholar
G. Matheron, J. for Applied Probability 5, 439 (1973). Crossref, Web of Science, Google Scholar
P. McCullach and J. A. Nelder , Generalized Linear Models , 1st edn. , Monographs on Statistics and Applied Probability 37 ( Chapman & Hall , 1983 ) . Crossref, Google Scholar
T. Minka and R. Picard, Learning how to learn is learning with point sets, Unpublished manuscript (1997). Available at http://wwwwhite.media.mit.edu/~tpminka/papers/learning.html . Google Scholar
T. Minka A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, January 2001 . Google Scholar
E. H. Moore, Bull. Amer. Math. Soc. 23, 59 (1916). Google Scholar
R. M. Neal, Probabilistic inference using Markov chain Monte Carlo methods, (Technical Report CRG-TR-93-1, University of Toronto, 1993). See www.cs.toronto.edu/~radford . Google Scholar
R. M. Neal , Bayesian Learning for Neural Networks , Lecture Notes in Statistics 118 ( Springer , 1996 ) . Crossref, Google Scholar
R. M. Neal, Monte Carlo implementation of Gaussian process models for Bayesian classification and regression, (Technical Report 9702, Department of Statistics, University of Toronto, 1997) . Google Scholar
J. Nelder and R. Wedderburn, J. Roy. Stat. Soc. B 135, 370 (1972). Crossref, Google Scholar
A. O'Hagan, J. Roy. Stat. Soc. B 40(1), 1 (1978). Google Scholar
A. O'Hagan, Bayesian Statistics 4, eds. J. Bernardoet al. (Oxford University Press, 1992) pp. 345–363. Google Scholar
M. Opper and O. Winther, Gaussian process classification and SVM: Mean field results and leave-one-out estimator, in Smola et al.68 . Google Scholar
M. Opper and O. Winther, Neural Computation 12(11), 2655 (2000). Crossref, Medline, Web of Science, Google Scholar
C. Paciorek, Nonstationary Gaussian Processes for Regression and Spatial Modelling, (PhD thesis, Carnegie Mellon University, Pittsburg, 2003) . Google Scholar
J. Platt, Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods, in Smola et al.68 . Google Scholar
J. Platt, C. Burges, S. Swenson, C. Weare and A. Zheng, Learning a Gaussian process prior for automatically generating music playlists, in Dietterich et al.,15 pp. 1425–1432 . Google Scholar
J. C. Platt, Fast training of support vector machines using sequential minimal optimization, in Schölkopf et al.,58 pp. 185–208 . Google Scholar
T. Poggio and F. Girosi, Proc. IEEE 78(9), 1481 (1990). Crossref, Web of Science, Google Scholar
W. H. Press et al. , Numerical Recipes in C , 2nd edn. ( Cambridge University Press , 1992 ) . Google Scholar
F. Rosenblatt, Psychological Review 65(6), 386 (1958). Crossref, Medline, Web of Science, Google Scholar
S. Roweis and L. Saul, Science 290, 2323 (2000). Crossref, Medline, Web of Science, Google Scholar
P. Rujan, J. de Physique I 3, 277 (1993). Crossref, Google Scholar
B. Schölkopf , C. Burges and A. Smola (eds.) , Advances in Kernel Methods: Support Vector Learning ( MIT Press , 1998 ) . Google Scholar
B. Schölkopf and A. Smola , Learning with Kernels , 1st edn. ( MIT Press , 2002 ) . Google Scholar
I. Schönberg, Proc. Nat. Acad. Sci. 39, 811 (1938). Medline, Web of Science, Google Scholar
I. J. Schönberg, Annals of Mathematicals 52, 947 (1964). Google Scholar
M. Seeger, Covariance kernels from Bayesian generative models, in Dietterich et al.,15 pp. 905–912 . Google Scholar
M. Seeger, Bayesian Gaussian Process Models: PAC-Bayesian Generalisation Error Bounds and Sparse Approximations (PhD thesis, University of Edinburgh, 2003). See www.cs.berkeley.edu/~mseeger . Google Scholar
M. Seeger, Bayesian methods for support vector machines and Gaussian processes, (Master's thesis, University of Karlsruhe, Germany, 1999). See www.cs.berkeley.edu/~mseeger . Google Scholar
65. M. Seeger, Bayesian model selection for support vector machines, Gaussian processes and other kernel classifiers, in Solla et al.,72 pp. 603–609 . Google Scholar
M. Seeger, Covariance kernels from Bayesian generative models, (Technical report, Institute for ANC, Edinburgh, UK, 2000). See www.cs.berkeley.edu/~mseeger . Google Scholar
M. Seeger, J. Machine Learning Research 3, 233 (2002). Crossref, Web of Science, Google Scholar
A. Smola et al. (eds.) , Advances in Large Margin Classifiers ( MIT Press , 1999 ) . Google Scholar
A. Smola, Z.Óvári and R. Williamson, Regularization with dot-product kernels, in Leen et al.,32 pp . Google Scholar
A. Smola, B. Schölkopf and K.-R. Müller, Neural Networks 11, 637 (1998). Crossref, Medline, Web of Science, Google Scholar
E. Solak, R. Murray-Smith, W. Leithead and C. Rasmussen, Derivative observations in Gaussian process models of dynamic systems, in Becker et al.,5 (To appear) . Google Scholar
S. Solla , T. Leen and K.-R. Müller (eds.) , Advances in Neural Information Processing Systems 12 ( MIT Press , 2000 ) . Google Scholar
P. Sollich, Probabilistic methods for support vector machines, in Solla et al.,72 pp. 349–355 . Google Scholar
M. Stein , Interpolation of Spatial Data: Some Theory for Kriging ( Springer , 1999 ) . Crossref, Google Scholar
I. Steinwart, J. Machine Learning Research 2, 67 (2001). Web of Science, Google Scholar
M. Szummer and T. Jaakkola, Partially labeled classification with Markov random walks, in Dietterich et al.,15 pp. 945–952 . Google Scholar
J. Tenenbaum, A. de Silva and J. Langford, Science 290, 2319 (2000). Crossref, Medline, Web of Science, Google Scholar
M. Tipping, J. Machine Learning Research 1, 211 (2001). Web of Science, Google Scholar
F. Vivarelli and C. K. I. Williams, Discovering hidden features with Gaussian process regression, in Kearns et al.26 . Google Scholar
G. Wahba , Spline Models for Observational Data , CBMS-NSF Regional Conf. Series ( SIAM Society for Industrial and Applied Mathematics , 1990 ) . Crossref, Google Scholar
G. Wahba, Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV, in Schölkopf et al.,58 pp. 69–88 . Google Scholar
M. Wainwright, E. Sudderth and A. Willsky, Tree-based modeling and estimation of Gaussian processes on graphs with cycles, in Leen et al.,32 pp. 661–667 . Google Scholar
Y. Weiss and W. Freeman, Correctness of belief propagation in Gaussian graphical models of arbitrary topology, in Solla et al.,72 pp. 673–679 . Google Scholar
C. Williams, Neural Computation 10(5), 1203 (1998). Crossref, Web of Science, Google Scholar
C. K. I. Williams , Learning in Graphical Models , ed. M. I. Jordan ( Kluwer , 1997 ) . Google Scholar
C. K. I. Williams and D. Barber, IEEE Transactions on Pattern Analysis and Machine Intelligence 20(12), 1342 (1998). Crossref, Web of Science, Google Scholar
S. Wright, SIAM J. Optimization 9(4), 1159 (1999). Crossref, Web of Science, Google Scholar
A. Yaglom , Correlation Theory of Stationary and Related Random Functions I ( Springer , 1987 ) . Google Scholar
H. Zhu et al. , Neural Networks and Machine Learning , NATO ASI series 168 , ed. C. Bishop ( Springer , 1998 ) . Google Scholar