Machine LearningNo Access

An Efficient Method for Computing a Bayesian Mixture of Feature-Based Models

Amir Jalalirad

Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, 5600 MB, The Netherlands

E-mail Address: a.jalalirad@tue.nl

Corresponding author.

Search for more papers by this author

and

Tjalling Tjalkens

Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, 5600 MB, The Netherlands

E-mail Address: t.j.tjalkens@tue.nl

Search for more papers by this author

https://doi.org/10.1142/S0218001416510022Cited by:0 (Source: Crossref)

Abstract

We describe a computationally efficient method to produce a specific Bayesian mixture of all the models in a finite set of feature-based models that assign a probability to the observed data set. Special attention is given to the bound on the regret of using the mixture instead of the best model in the set. It is proven theoretically and verified through synthetic data that this bound is relatively tight. Comparing the workload of the proposed method with the direct implementation of the Bayesian mixture shows an almost exponential improvement of computing time.

Keywords:

References

1. C. M. Bishop, Pattern Recognition and Machine Learning (Springer Science, 2006). Google Scholar
2. N. G. de Bruijn, Asymptotic Methods in Analysis (Dover Publications Inc., New York, 1981). Google Scholar
3. W. L. Buntine, A guide to the literature on learning probabilistic networks from data, IEEE Trans. Knowl. Data Eng. 8 (1996) 195–210. Crossref, Web of Science, Google Scholar
4. H. Cai, K. Mikolajczyk and J. Matas, Learning linear discriminant projections for dimensionality reduction of image descriptors, IEEE Trans. Pattern Anal. Mach. Intell. 33 (2011) 338–352. Crossref, Web of Science, Google Scholar
5. G. C. Cawley and N. L. C. Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, J. Mach. Learn. Res. 11 (2010) 2079–2107. Web of Science, Google Scholar
6. I. J. Good, The Estimation of Probabilities: An Essay on Modern Bayesian Methods (MIT Press, 1965). Google Scholar
7. P. D. Grünwald, The Minimum Description Length Principle (MIT Press, 2007). Crossref, Google Scholar
8. I. Guyon, A. Saffari, G. Dror and G. Cawley, Model selection: Beyond the Bayesian/Frequentist Divide, J. Mach. Learn. Res. 11 (2010) 61–87. Web of Science, Google Scholar
9. F. Johansson and G. Falkman, Detection of vessel anomalies - a Bayesian network approach, Melbourne, Australia 3rd Int. Conf. Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP2007) 12 (2007) 395–400. Google Scholar
10. M. Kafai and B. Bhanu, Dynamic Bayesian networks for vehicle classification in video, IEEE Trans. Ind. Informat. 8 (2012) 100–109. Crossref, Web of Science, Google Scholar
11. R. E. Krichevsky and V. K. Trofimov, The performance of universal encoding, IEEE Trans. Inf. Theory 27 (1981) 199–207. Crossref, Web of Science, Google Scholar
12. U. Kuter and J. Golbeck, Using probabilistic confidence models for trust inference in Web-based social networks, ACM Trans. Internet Techn. 10 (2010) 8. Crossref, Web of Science, Google Scholar
13. H. Meng, K. Appiah, A. Hunter and P. Dickinson, FPGA implementation of Naïve Bayes classifier for visual object recognition, Colorado Springs, CO IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops (CVPRW) 06 (2011) 123–128. Google Scholar
14. A. Miguel, A. Ortega, L. Buera and E. Lleida, Bayesian networks for discrete observation distributions in speech recognition, IEEE Trans. Audio, Speech, Language Process. 19 (2011) 1476–1489. Crossref, Google Scholar
15. A. Ng and M. I. Jordan, On discriminative vs. generative classifiers: A comparison of logistic regression and naïve Bayes, Advances in Neural Information Processing Systems (NIPS14) (2002) 841–848. Google Scholar
16. C. Ordonez and S. K. Pitchaimalai, Bayesian classifiers programmed in SQL, IEEE Trans. Knowl. Data Eng. 22 (2009) 139–144. Crossref, Web of Science, Google Scholar
17. J. Riordan, An Introduction to Combinatorial Analysis (John Wiley & Sons, New York, 1958). Google Scholar
18. J. Rissanen, Universal coding, information prediction, and estimation, IEEE Trans. Inf. Theory 30 (1984) 629–636. Crossref, Web of Science, Google Scholar
19. T. Roos, H. Wetting, P. Grünwald, P. Myllymäki and H. Tirri, On discriminative Bayesian network classifiers and logistic regression, Mach. Learn. 59(3) (2005) 267–296. Web of Science, Google Scholar
20. R. Ruiz-Torrubiano and A. Suárez, Hybrid approaches and dimensionality reduction for portfolio selection with cardinality constraints, IEEE Comput. Intell. Mag. 5 (2010) 92–107. Google Scholar
21. K. Seyoung, P. Smyth and H. Stern, A Bayesian mixture approach to modeling spatial activation patterns in multisite fMRI data, IEEE Trans. Med. Imag. 29 (2010) 1260–1274. Crossref, Web of Science, Google Scholar
22. I. Shmulevich, E. R. Dougherty, S. Kim and W. Zhang, Probabilistic Boolean networks: A rule-based uncertainty model for gene regulatory networks, Bioinformatics 18 (2002) 261–274. Crossref, Web of Science, Google Scholar
23. Yu. M. Shtarkov, Universal sequential coding of single messages, Probl. Inf. Transm. 23 (1987) 3–17. Google Scholar
24. P. Trairatphisan, A. Mizera, J. Pang, A. A. Tantar, J. Schneider and T. Sauter, Recent development and biomedical applications of probabilistic Boolean networks, Cell Commun. Signal. 11 (2013) 46. Crossref, Web of Science, Google Scholar
25. F. M. J. Willems, Yu. M. Shtarkov and Tj. J. Tjalkens, The context-tree weighting method: basic properties, IEEE Trans. Inf. Theory 41 (1995) 653–664. Crossref, Web of Science, Google Scholar
26. L. Zhang and Q. Ji, Image segmentation with a unified graphical model, IEEE Trans. Pattern Anal. Mach. Intell. 32 (2010) 1406–1425. Crossref, Web of Science, Google Scholar
27. L. Zhang and Q. Ji, A Bayesian network model for automatic and interactive image segmentation, IEEE Trans. Image Process. 20 (2011) 2582–2593. Crossref, Web of Science, Google Scholar