Research PaperNo Access

Enhancing performance of the back-propagation algorithm based on a novel regularization method of preserving inter-object-distance of data

Junying Hu

School of Mathematics, Northwest University, 1 Xuefu Avenue, Xi’an, 710010, P. R. China

E-mail Address: hujunying@nwu.edu.cn

Search for more papers by this author

Kai Sun

School of Mathematics and Statistics, Xi’an Jiaotong University, 28 Xianning West Road, Xi’an 710049, P. R. China

E-mail Address: kaisun@mail.xjtu.edu.cn

Search for more papers by this author

, and

Hai Zhang

https://orcid.org/0000-0003-1437-1729

School of Mathematics and Statistics, Xi’an Jiaotong University, 28 Xianning West Road, Xi’an 710049, P. R. China

Faculty of Information Technology & State Key Laboratory of Quality Research in Chinese Medicines, Macau University of Science and Technology, Weilong Road, Macau 519020, P. R. China

E-mail Address: zhanghai@nwu.edu.cn

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0219691320500812Cited by:0 (Source: Crossref)

Abstract

Artificial neural networks, consisting of many levels of nonlinearities, have been widely used to deal with various supervised learning tasks. At present, the most popular and effective training method is back-propagation algorithm (BP). Inspired by manifold regularization framework, we introduce a novel regularization framework, which aims at preserving the inter-object-distance of the data. Then a refined BP algorithm (IOD-BP) is proposed by imposing the proposed regularization framework into the objective function of BP algorithm. Comparative experiments on various benchmark classification tasks show that the new regularization BP method significantly improves the performance of BP algorithm in terms of classification accuracy.

Keywords:

AMSC: 22E46, 53C35, 57S20

References

1. M. Belkin, P. Niyogi and V. Sindhwani , On manifold regularization, in Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Bridgetown, Barbados: Society for Artificial Intelligence and Statistics (2005), pp. 1–8. Google Scholar
2. M. Belkin, P. Niyogi and V. Sindhwani , Manifold regularization: A geometric framework for learning from labeled and unlabeled examples, J. Mach. Learn. Res. 7 (2006) 2399–2434. Web of Science, Google Scholar
3. L. Cayton, Algorithms for manifold learning, Univ. of California at San Diego Tech. Rep. (2005), pp. 1–17. Google Scholar
4. Y. Chauvin , A backpropagation algorithm with optimal use of hidden units, in Advances in Neural Information Processing Systems, ed. D. S. Touretzky , Vol. 1 (Morgan Kaufmann, San Mateo, CA, 1989), pp. 519–526. Google Scholar
5. D. Ciresan, U. Meier, J. Masci et al., Multi-column deep neural network for traffic sign classification, Neural Netw. 32 (2012) 333–338. Crossref, Web of Science, Google Scholar
6. T. M. Cover and J. A. Thomas , Elements of Information Theory (John Wiley & Sons, 2012). Google Scholar
7. A. Fischer and C. Igel , An introduction to restricted Boltzmann machines, in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (Springer, Berlin, Heidelberg, 2012), pp. 14–36. Crossref, Google Scholar
8. A. Frank and A. Asuncion, (2011). UCI Machine Learning Repository. Available at http://archive.ics.uci.edu/ml. Google Scholar
9. G. E. Hinton, S. Osindero and Y. W. Teh , A fast learning algorithm for deep belief nets, Neural Comput. 18(7) (2006) 1527–1554. Crossref, Web of Science, Google Scholar
10. G. E. Hinton and R. R. Salakutdinov , Reducing the dimensionality of data with neural networks, Science 313(5786) (2006) 504–507. Crossref, Web of Science, Google Scholar
11. K. Hirasawa , Enhancing the generalization ability of neural networks through controlling the hidden layers, Appl. Soft Comput. 9 (2009) 401–414. Web of Science, Google Scholar
12. http://www.cs.toronto.edu/hinton/MatlabForSciencePaper.html. Google Scholar
13. S. Melacci and M. Belkin , Laplacian support vector machines trained in the primal, J. Mach. Learn. Res. 12 (2011) 1149–1184. Web of Science, Google Scholar
14. D. E. Rumelhart, G. E. Hinton and R. J. Williams , Learning representations by back-propagating errors, Nature 323 (1986) 533–536. Crossref, Web of Science, Google Scholar
15. R. R. Salakhutdinov and G. E. Hinton , Deep Boltzmann machines, in Proc. Int. Conf. Artificial Intelligence and Statistics, Vol. 12 (Florida, USA, 2009), pp. 448–455. Google Scholar
16. V. Sindhwani, P. Niyogi and M. Belkin , Beyond the point cloud: From transductive to semi-supervised learning, in Proc. 22nd Int. Conf. Machine Learning (Bonn, 2005), pp. 824–831. Crossref, Google Scholar
17. N. Srivastava, R. R. Salakhutdinov and G. E. Hinton, Modeling documents with deep Boltzmann machines, preprint (2013), arXiv:1309.6865 . Google Scholar
18. J. Zhang, N. Ji, J. Liu, J. Pan and D. Meng , Enhancing performance of the backpropagation algorithm via sparse response regularization, Neurocomputing 153 (2015) 20–40. Crossref, Web of Science, Google Scholar