Research PaperNo Access

Comparison theorems on large-margin learning

Amina Benabid

Department of Mathematics, Zhejiang Normal University, Jinhua, Zhejiang 321004, P. R. China

E-mail Address: 201810800016@zjnu.edu.cn

Search for more papers by this author

Jun Fan

Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong, P. R. China

E-mail Address: junfan@hkbu.edu.hk

Search for more papers by this author

, and

Dao-Hong Xiang

https://orcid.org/0000-0003-3793-4707

Department of Mathematics, Zhejiang Normal University, Jinhua, Zhejiang 321004, P. R. China

E-mail Address: daohongxiang@zjnu.cn

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0219691321500156Cited by:2 (Source: Crossref)

Abstract

This paper studies the binary classification problem associated with a family of Lipschitz convex loss functions called large-margin unified machines (LUMs), which offers a natural bridge between distribution-based likelihood approaches and margin-based approaches. LUMs can overcome the so-called data piling issue of support vector machine in the high-dimension and low-sample size setting, while their theoretical analysis from the perspective of learning theory is still lacking. In this paper, we establish some new comparison theorems for all LUM loss functions which play a key role in the error analysis of large-margin learning algorithms. Based on the obtained comparison theorems, we further derive learning rates for regularized LUMs schemes associated with varying Gaussian kernels, which maybe of independent interest.

Keywords:

AMSC: 68Q32, 68W40, 41A46

References

1. P. L. Bartlett, M. I. Jordan and J. D. McAuliffe , Convexity, classification, and risk bounds, J. Am. Stat. Assoc. 101(473) (2006) 138–156. Crossref, Web of Science, Google Scholar
2. B. E. Boser, I. M. Guyon and V. N. Vapnik , A training algorithm for optimal margin classifiers, in Proc. Fifth Annual Workshop on Computational Learning Theory (1992), Pittsburgh, ACM, pp. 144–152. Crossref, Google Scholar
3. A. Caponnetto and E. De Vito , Optimal rates for the regularized least-squares algorithm, Found. Comput. Math. 7(3) (2007) 331–368. Crossref, Web of Science, Google Scholar
4. D. R. Chen, Q. Wu, Y. Ying and D.-X. Zhou , Support vector machine soft margin classifiers: Error analysis, J. Mach. Learn. Res. 5 (2004) 1143–1175. Web of Science, Google Scholar
5. C. Cortes and V. Vapnik , Support-vector networks, Mach. Learn. 20(3) (1995) 273–297. Crossref, Web of Science, Google Scholar
6. Y. Feng, J. Fan and J. A. Suykens , A statistical learning approach to modal regression., J. Mach. Learn. Res. 21(2) (2020) 1–35. Google Scholar
7. T. Hastie, R. Tibshirani and J. Friedman , The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer-Verlag, New York, 2001). Crossref, Google Scholar
8. T. Hu, J. Fan and D. H. Xiang , Convergence analysis of distributed multi-penalty regularized pairwise learning, Anal. Appl. 18(1) (2020) 109–127. Link, Web of Science, Google Scholar
9. G. Kriukova, S. V. Pereverzyev and P. Tkachenko , On the convergence rate and some applications of regularized ranking algorithms, J. Complexity 33 (2016) 14–29. Crossref, Web of Science, Google Scholar
10. Y. Liu, H. H. Zhang and Y. Wu , Hard or soft classification? Large-margin unified machines, J. Am. Stat. Assoc. 106(493) (2011) 166–177. Crossref, Web of Science, Google Scholar
11. F. Lv and J. Fan , Optimal learning with Gaussians and correntropy loss, Anal. Appl. 19(1) (2021) 107–124. Link, Web of Science, Google Scholar
12. J. S. Marron, M. J. Todd and J. Ahn , Distance-weighted discrimination, J. Am. Stat. Assoc. 102(480) (2007) 1267–1271. Crossref, Web of Science, Google Scholar
13. I. Steinwart, C. Scovel et al., Fast rates for support vector machines using Gaussian kernels, Ann. Stat. 35(2) (2007) 575–607. Crossref, Web of Science, Google Scholar
14. A. B. Tsybakov et al., Optimal aggregation of classifiers in statistical learning, Ann. Stat. 32(1) (2004) 135–166. Crossref, Web of Science, Google Scholar
15. A. W. van der Vaart and J. H. van Zanten , Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth, Ann. Stat. 37(5B) (2009) 2655–2675. Crossref, Web of Science, Google Scholar
16. B. Wang and H. Zou , Another look at distance-weighted discrimination, J. R. Stat. Soc. Ser. B. Stat. Methodol. 80(1) (2018) 177–198. Crossref, Web of Science, Google Scholar
17. Q. Wu , Classification and Regularization in Learning Theory (VDM Verlag, 2009). Google Scholar
18. Q. Wu and D. X. Zhou , Svm soft margin classifiers: Linear programming versus quadratic programming, Neural Comput. 17(5) (2005) 1160–1187. Crossref, Web of Science, Google Scholar
19. D. H. Xiang , Classification with Gaussians and convex loss II: Improving error bounds by noise conditions, Sci. China Math. 54(1) (2011) 165–171. Crossref, Web of Science, Google Scholar
20. D. H. Xiang , Logistic classification with varying Gaussians, Comput. Math. Appl. 61(2) (2011) 397–407. Crossref, Web of Science, Google Scholar
21. D. H. Xiang , A new comparison theorem on conditional quantiles, Appl. Math. Lett. 25(1) (2012) 58–62. Crossref, Web of Science, Google Scholar
22. D. H. Xiang, T. Hu and D. X. Zhou , Learning with varying insensitive loss, Appl. Math. Lett. 24(12) (2011) 2107–2109. Crossref, Web of Science, Google Scholar
23. D. H. Xiang and D. X. Zhou , Classification with Gaussians and convex loss, J. Mach. Learn. Res. 10(7) (2009) 1447–1468. Google Scholar
24. T. Zhang et al., Statistical behavior and consistency of classification methods based on convex risk minimization, Ann. Stat. 32(1) (2004) 56–85. Crossref, Web of Science, Google Scholar
25. Y. Zhao, J. Fan and L. Shi , Learning rates for regularized least squares ranking algorithm, Anal. Appl. 15(6) (2017) 815–836. Link, Web of Science, Google Scholar