Machine LearningNo Access

A New Sparse Restricted Boltzmann Machine

Jiangshu Wei

Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China

E-mail Address: weijiangshu66@163.com

Search for more papers by this author

Jiancheng Lv

http://orcid.org/0000-0001-6551-3884

Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China

E-mail Address: lvjiancheng@scu.edu.cn

Corresponding author.

Search for more papers by this author

, and

Zhang Yi

Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China

E-mail Address: zhangyi@scu.edu.cn

Search for more papers by this author

https://doi.org/10.1142/S0218001419510042Cited by:7 (Source: Crossref)

Abstract

Although existing sparse restricted Boltzmann machine (SRBM) can make some hidden units activated, the major disadvantage is that the sparseness of data distribution is usually overlooked and the reconstruction error becomes very large after the hidden unit variables become sparse. Different from the SRBMs which only incorporate a sparse constraint term in the energy function formula from the original restricted Boltzmann machine (RBM), an energy function constraint SRBM (ESRBM) is proposed in this paper. The proposed ESRBM takes into account the sparseness of the data distribution so that the learned features can better reflect the intrinsic features of data. Simulations show that compared with SRBM, ESRBM has smaller reconstruction error and lower computational complexity, and that for supervised learning classification, ESRBM obtains higher accuracy rates than SRBM, classification RBM, and Softmax classifier.

Keywords:

References

1. Y. Bengio and O. Delalleau, Justifying and generalizing contrastive divergence, Neural Comput. 21(6) (2009) 1601–1621. Web of Science, Google Scholar
2. Y. Bengio, P. Lamblin, D. Popovici and H. Larochelle, Greedy layer-wise training of deep networks, Adv. Neural Inf. Process. Syst. 19 (2007) 153–160. Google Scholar
3. N. Boulanger-Lewandowski, Y. Bengio and P. Vincent, Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription, arXiv:1206.6392. Google Scholar
4. A. M. Bruckstein, D. L. Donoho and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Rev. 51(1) (2009) 34–81. Web of Science, Google Scholar
5. M. A. Carreira-Perpinan and G. E. Hinton, On contrastive divergence learning, in Proc. 10th Int. Workshop on Artificial Intelligence and Statistics (Barbados, 2005), pp. 33–40. Google Scholar
6. S. Chandra, S. Kumar and C. Jawahar, Learning Multiple Non-Linear Sub-Spaces using K-RBMs, IEEE Conf. Computer Vision and Pattern Recognition (CVPR) (Portland, USA, 2013), pp. 2778–2785. Google Scholar
7. K. Cho, A. Ilin and T. Raiko, Improved learning of Gaussian-Bernoulli restricted Boltzmann machines, in Proc. 21st Int. Conf. Artificial Neural Networks and Machine Learning (ICANN) (Espoo, Finland, 2011), pp. 10–17. Google Scholar
8. A. Fischer and C. Igel, An introduction to restricted Boltzmann machines, in Proc. 17th Iberoamerican Congress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (CIARP) (Buenos Aires, Argentina, 2012), pp. 14–36. Google Scholar
9. A. Fischer and C. Igel, Training restricted Boltzmann machines: An introduction, Pattern Recognit. 47(1) (2014) 25–39. Web of Science, Google Scholar
10. X. Glorot, A. Borders and Y. Bengio, Deep sparse rectifier neural networks, in Proc. 14th Int. Conf. Artificial Intelligence and Statistics (AISTATS) (Fort Lauderdale, FL, USA, 2011), pp. 315–323. Google Scholar
11. G. E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Comput. 14(8) (2002) 1771–1800. Web of Science, Google Scholar
12. G. E. Hinton, A practical guide to training restricted Boltzmann machines, Neural Netw. 9(1) (2012) 599–619. Google Scholar
13. G. E. Hinton, S. Osindero and Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18(7) (2006) 1527–1554. Web of Science, Google Scholar
14. G. E. Hinton and R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313(5786) (2006) 504–507. Web of Science, Google Scholar
15. G. E. Hinton and T. Sejnowski, Learning and relearning in Boltzmann machines, in Parallel Distributed Processing: Explorations in Microstructures of Cognition, eds. D. E. Rumelhart and J. L. McClelland (MIT Press, Cambridge, 1986), pp. 282–317. Google Scholar
16. P. O. Hoyer, Non-negative matrix factorization with sparseness constraints, J. Mach. Learn. Res. 5 (2004) 1457–1469. Web of Science, Google Scholar
17. N. N. Ji, J. S. Zhang and C. X. Zhang, A sparse-response deep belief network based on rate distortion theory, Pattern Recognit. 47(9) (2014) 3179–3191. Web of Science, Google Scholar
18. N. N. Ji, J. S. Zhang, C. X. Zhang and Q. Y. Yin, Enhancing performance of restricted Boltzmann machines via log-sum regularization, Knowl. Based Syst. 63 (2014) 82–96. Web of Science, Google Scholar
19. O. Krause, A. Fischer, T. Glasmachers and C. Igel, Approximation properties of DBNs with binary hidden units and real-valued visible units, in Proc. 30th Int. Conf. Machine Learning (ICML-13) (Atlanta, USA, 2013), pp. 419–426. Google Scholar
20. H. Larochelle and Y. Bengio, Classification using discriminative restricted Boltzmann machines, in Proc. 25th Int. Conf. Machine Learning (ICML-08) (Helsinki, Finland, 2008), pp. 536–543. Google Scholar
21. H. Larochelle, M. Mandel, R. Pascanu and Y. Bengio, Learning algorithms for the classification restricted Boltzmann machine, J. Mach. Learn. Res. 13(1) (2012) 643–669. Google Scholar
22. H. Larochelle and I. Murray, The neural autoregressive distribution estimator, in Proc. 14th Int. Conf. Artificial Intelligence and Statistics (AISTATS) (Fort Lauderdale, FL, USA, 2011), pp. 29–37. Google Scholar
23. N. Le Roux, N. Heess, J. Shotton and J. Winn, Learning a generative model of images by factoring appearance and shape, Neural Comput. 23(3) (2011) 593–650. Web of Science, Google Scholar
24. H. Lee, C. Ekanadham and A. Y. Ng, Sparse deep belief net model for visual area V2, Adv. Neural Inf. Process. Syst. 20 (2007) 873–880. Google Scholar
25. K. Liu, Z. Limin and Z. Chao, New hybrid sparse penalty mechanism of restricted Boltzmann machine, J. Zhejiang Univ. Eng. Sci. 6 (2015) 1070–1078. Google Scholar
26. J. Louradour and H. Larochelle, Classification of sets using restricted Boltzmann machines, arXiv:1103.4896. Google Scholar
27. H. Luo, R. Shen and C. Niu, Sparse group restricted Boltzmann machines, arXiv:1008.4988. Google Scholar
28. J. C. Lv, Z. Yi and Y. Li, Non-divergence of stochastic discrete time algorithms for PCA neural networks, Non-Divergence of Stochastic Discrete Time Algorithms for PCA Neural Networks 26(2) (2015) 394–399. Google Scholar
29. J. Melchior, A. Fischer, N. Wang and L. Wiskott, How to center binary restricted Boltzmann machines, arXiv:1311.1354. Google Scholar
30. A.-R. Mohamed and G. Hinton, Phone recognition using restricted Boltzmann machines, in Proc. 2010 IEEE Int. Conf. Acoustics, Speech and Signal Processing (Dallas, Texas, USA, 2010), pp. 4354–4357. Google Scholar
31. T. Osogami and M. Otsuka, Restricted Boltzmann machines modeling human choice, Adv. Neural Inf. Process. Syst. 1 (2014) 73–81. Google Scholar
32. M. Ranzato, Y.-L. Boureau and Y. L. Cun, Sparse feature learning for deep belief networks, Adv. Neural Inf. Process. Syst. 20 (2007) 1185–1192. Google Scholar
33. M. Ranzato, C. Poultney, S. Chopra and Y. L. Cun, Efficient learning of sparse representations with an energy-based model, Adv. Neural Inf. Process. Syst. 19 (2006) 1137–1144. Google Scholar
34. R. Salakhutdinov and G. E. Hinton, Deep Boltzmann machines, in Proc. 12th Int. Conf. Artificial Intelligence and Statistics (AISTATS) (Clearwater Beach, Florida USA, 2009), pp. 448–455. Google Scholar
35. R. Salakhutdinov and H. Larochelle, Efficient learning of deep Boltzmann machines, in Proc. 13th Int. Conf. Artificial Intelligence and Statistics (AISTATS) (Clearwater Beach, Florida USA, 2010), pp. 693–700. Google Scholar
36. P. Smolensky, Information processing in dynamical systems: Foundations of harmony theory, Parallel Distributed Processing: Explorations in Microstructures of Cognition, eds. D. E. Rumelhart and J. L. McClelland (MIT Press, Cambridge, 1986), pp. 194–281. Google Scholar
37. I. Sutskever, G. E. Hinton and G. W. Taylor, The recurrent temporal restricted Boltzmann machine, Adv. Neural Inf. Process. Syst. 20 (2008) 1601–1608. Google Scholar
38. Y. Tang, N. Srivastava and R. R. Salakhutdinov, Learning generative models with visual attention, Adv. Neural Inf. Process. Syst. 2 (2014) 1808–1816. Google Scholar
39. Y. Tang and I. Sutskever, Data normalization in the learning of restricted Boltzmann machines, Technical Report UTML-TR-11-2, Department of Computer Science, University of Toronto (2011). Google Scholar
40. N. Wang, J. Melchior and L. Wiskott, Gaussian-binary restricted Boltzmann machines on modeling natural image statistics, arXiv:1401.5900. Google Scholar
41. N. Wang, J. Melchior and L. Wiskott, An analysis of Gaussian-binary restricted Boltzmann machines for natural images, 2012 European Sympo. Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) (Bruges, Belgium, 2012), pp. 287–292. Google Scholar
42. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry and Y. Ma, Robust face recognition via sparse representation, IEEE Trans. Pattern Anal. Mach. Intell. 31(2) (2009) 210–227. Web of Science, Google Scholar
43. T. Yamashita, M. Tanaka, E. Yoshida, Y. Yamauchi and H. Fujiyoshii, To be Bernoulli or to be Gaussian, for a Restricted Boltzmann Machine, in Proc. 22nd Int. Conf. Pattern Recognition (ICPR) (Stockholm, Sweden, 2014), pp. 1520–1525. Google Scholar