Research PaperNo Access

Error analysis of the kernel regularized regression based on refined convex losses and RKBSs

Baohuai Sheng

Department of Applied Statistics, Shaoxing University, Shaoxing 312000, P. R. China

E-mail Address: bhsheng@usx.edu.cn

Search for more papers by this author

and

Lan Zuo

Department of Applied Statistics, Shaoxing University, Shaoxing 312000, P. R. China

E-mail Address: zzuolan@163.com

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0219691321500120Cited by:6 (Source: Crossref)

Abstract

In this paper, we bound the errors of kernel regularized regressions associating with $2$ $2$ -uniformly convex two-sided RKBSs and differentiable $σ (t) = \frac{| t |^{p}}{p} (1 < p \leq 2)$ $σ (t) = \frac{| t |^{p}}{p} (1 < p \leq 2)$ uniformly smooth losses. In particular, we give learning rates for the learning algorithm with loss $V_{p} (t) = \frac{| t |^{p}}{p} (1 < p \leq 2)$ $V_{p} (t) = \frac{| t |^{p}}{p} (1 < p \leq 2)$ . Also, we show a probability inequality and with which provide the error bounds for kernel regularized regression with loss $V_{q} (t) = \frac{| t |^{q}}{q} (q > 2) .$ $V_{q} (t) = \frac{| t |^{q}}{q} (q > 2) .$ The discussions are comprehensive applications of the uniformly smooth function theory, the uniformly convex function theory and uniformly convex space theory.

Keywords:

AMSC: 90C25, 68Q32, 68T40, 41A25

References

1. V. A. Abilov, F. V. Abilova and M. K. Kerimov , Some issues concerning approximation of functions by Fourier–Bessel sums, Comput. Math. Math. Phys. 53(7) (2013) 867–873. Crossref, Web of Science, Google Scholar
2. R. Askey and S. Wainger , A convolution structure for Jacobi series, Amer. J. Math. 91 (1969) 463–485. Crossref, Web of Science, Google Scholar
3. F. Bauer, S. Pereverzey and L. Rosasco , On regularization algorithms in learning theory, J. Complexity 23 (2007) 52–72. Crossref, Web of Science, Google Scholar
4. H. H. Bauschke and P. L. Combettes , Convex Analysis and Monotone Operator Theory in Hilbert Spaces (Springer, New York, 2010). Google Scholar
5. G. Blanchard and N. Krämer, Optimal learning rates for kernel conjugate gradient regression, preprint (2010), arXiv:1009.5839 [math.ST]. Google Scholar
6. G. Blanchard and N. Mücke , Optimal rates for regularized of statistical inverse learning problems, Found. Comput. Math. 18(4) (2018) 971–1013. Crossref, Web of Science, Google Scholar
7. J. Borwein, A. J. Guirao, P. Hájek and J. Vanderwerff , Uniformly convex functions on Banach spaces, Proc. Amer. Math. Soc. 137(3) (2009) 1081–1091. Crossref, Web of Science, Google Scholar
8. J. Cai , An operator approach to analysis of conditional kernel canonical correlation, Int. J. Wavelets Multiresolut. Inf. Process. 13(4) (2015) 1550024. Link, Web of Science, Google Scholar
9. A. Caponnetto and E. De Vito , Optimal rates for the regularized least-squares algorithm, Found. Comput. Math. 7 (2007) 331–368. Crossref, Web of Science, Google Scholar
10. H. Chen, W. Huang and D. R. Chen , Simultaneous estimations of optimal directions and optimal transformations for function data, Int. J. Wavelets Multiresolut. Inf. Process. 18(5) (2018) 1850043. Link, Web of Science, Google Scholar
11. H. Chen and J. T. Wu , Kernel methods for independence measurement with coefficient constraints, Int. J. Wavelets Multiresolut. Inf. Process. 12(1) (2014) 1450006. Link, Web of Science, Google Scholar
12. A. Christmann, A. V. Messem and I. Steinwart , On consistency and robustness properties of support vector machines for heavy-tailed distributions, Stat. Interface 2 (2009) 311–327. Crossref, Web of Science, Google Scholar
13. A. Christmann and I. Steinwart , On robustness properties of convex risk minimization methods for pattern recognition, J. Mach. Learn. Res. 5 (2004) 1007–1034. Web of Science, Google Scholar
14. A. Christmann and I. Steinwart , Consistency and robustness of kernel based regression methods, Bernoulli 13 (2007) 799–819. Crossref, Web of Science, Google Scholar
15. J. A. Clarkson , Uniformly convex spaces, Trans. Amer. Math. Soc. 40 (1936) 396–414. Crossref, Google Scholar
16. P. L. Combettes, S. Salzo and S. Villa , Regularized learning schemes in feature Banach spaces, Anal. Appl. 16(1) (2018) 1–54. Link, Web of Science, Google Scholar
17. F. Cucker and S. Smale , On the mathematical foundations of learning theory, Bull. Amer. Math. Soc. 39 (2001) 1–49. Crossref, Web of Science, Google Scholar
18. F. Cucker and S. Smale , Best choices for regularization parameters in learning theory: On the bias-variance problem, Found. Comput. Math. 2(4) (2002) 413–428. Crossref, Web of Science, Google Scholar
19. F. Cucker and D. X. Zhou , Learning Theory: An Approximation Theory Viewpoint (Cambridge University Press, New York, 2007). Crossref, Google Scholar
20. E. De Vito, S. Pereverzyev and L. Rosasco , Adaptive kernel methods using the balancing principle, Found. Comput. Math. 10 (2010) 455–479. Crossref, Web of Science, Google Scholar
21. E. De Vito, L. Rosasco, A. Caponnetto, M. Piana and A. Verri , Some properties of regularized kernel methods, J. Mach. Learn. Res. 5 (2004) 1363–1390. Web of Science, Google Scholar
22. R. A. Devore and G. G. Lorentz , Constructive Approximation (Springer-Verlag, New York, 1993). Crossref, Google Scholar
23. X. M. Dong, H. Weng, J. Shi and Y. H. Gu , Randomized multi-scale kernels learning with sparsity constant regularization for regression, Int. J. Wavelets Multiresolut. Inf. Process. 17(6) (2019) 1950048. Link, Web of Science, Google Scholar
24. M. Ebets and I. Steinwart , Optimal regression rates for SVMs using Gaussian kernels, Electron. J. Stat. 6 (2012) 2627–2668. Google Scholar
25. T. Evgeniou, M. Pontil and T. Poggio , Regularization networks and support vector machines, Adv. Comput. Math. 13 (2000) 1–50. Crossref, Web of Science, Google Scholar
26. G. E. Fasshauer, F. J. Hickernell and Q. Ye , Solving support vector machines in reproducing kernel Banach spaces with positive definite functions, Appl. Comput. Harmon. Anal. 38 (2015) 115–139. Crossref, Web of Science, Google Scholar
27. S. Fischer and I. Steinwart , Sobolev norm learning rates for regularized least-squares algorithm, J. Mach. Learn. Res. 21 (2020) 1–38. Web of Science, Google Scholar
28. R. Hable and A. Christmann , Qualitative robustness of support vector machines, J. Multivar. Anal. 102(6) (2011) 993–1007. Crossref, Web of Science, Google Scholar
29. O. Hanner , On the uniform convexity of $L^{p}$ $L^{p}$ and $l^{p}$ $l^{p}$ , Ark. Mat. 3 (1956) 239–244. Crossref, Google Scholar
30. Q. He and C. X. Wu , Separating theorem of samples in Banach space for support vector machine learning, Int. J. Mach. Learn. Cybern. 2 (2011) 49–54. Crossref, Web of Science, Google Scholar
31. J. C. Kuang , Applied Inequalities (Shandong Science and Technology Press, 2004) (in Chinese). Google Scholar
32. Y. W. Lei and D. X. Zhou, Convergence of online mirror descent, preprint (2018), arXiv:1802.06357 [cs.LG]. Google Scholar
33. Y. W. Lei and D. X. Zhou , Analysis on online composite mirror descent algorithm, Neural Comput. 29 (2017) 1–36. Crossref, Web of Science, Google Scholar
34. J. H. Lin, A. Rudi, L. Rosasco and V. Cevher, Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces, preprint (2018), arXiv:1801.06720 [stat.ML]. Google Scholar
35. R. R. Lin, H. Z. Zhang and J. Zhang, On reproducing kernel Banach spaces: Generic definitions and unified framework of constructions, https://www.researchgate.net/publication/330194973. Google Scholar
36. H. X. Liu, B. H. Sheng and P. X. Ye , The improved learning rate for regularized regression with RKBSs, Int. J. Mach. Learn. Cybern. 8 (2017) 1235–1245. Crossref, Web of Science, Google Scholar
37. Y. X. Ma and H. W. Sun , Application of integral operator for vector-valued regression learning, Int. J. Wavelets Multiresolut. Inf. Process. 13(6) (2015) 1550047. Link, Web of Science, Google Scholar
38. C. Markett , Product formulas and convolution structure for Fourier–Bessel series, Constr. Approx. 5 (1989) 383–404. Crossref, Web of Science, Google Scholar
39. C. A. Micchelli and M. Pontil , Learning the kernel function via regularization, J. Mach. Learn. Res. 6 (2005) 1099–1125. Web of Science, Google Scholar
40. C. A. Micchelli, Y. S. Xu and H. Z. Zhang , Universal kernels, J. Mach. Learn. Res. 7 (2006) 2651–2667. Web of Science, Google Scholar
41. G. Michael, R. Christian and Z. Barbara , Regularized kernel-based reconstruction in generalized Besov spaces, Found. Comput. Math. 18 (2018) 459–508. Crossref, Web of Science, Google Scholar
42. M. Z. Nashed and Q. Y. Sun , Sampling and reconstruction of signals in a reproducing kernel subspace of $L^{p} (ℜ^{d})$ , J. Funct. Anal. 258(7) (2010) 2422–2452. Crossref, Web of Science, Google Scholar
43. V. S. Pawelke , Ein satz vom Jacksonschen type für algebraische polynome, Acta Sci. Math. (Szeged) 33 (1972) 323–336. Google Scholar
44. J. Peypouquet , Convex Optimization in Normed Spaces: Theory, Methods and Examples (Springer-Verlag, New York, 2014). Google Scholar
45. M. Pontil, Y. M. Ying and D. X. Zhou, Error analysis for online gradient descent algorithms in reproducing kernel Hilbert spaces, https://www.researchgate.net/publication/228787486. Google Scholar
46. W. Rudin , Real and Complex Analysis, 3rd edn. (McGraw-Hill, New York, 1987). Google Scholar
47. B. H. Sheng , Approximation of periodic functions by spherical translation networks, Acta Math. Sin. Chinese Ser. 50(1) (2007) 55–62. Google Scholar
48. B. H. Sheng , A way of constructing spherical approximation operators, J. Sys. Sci. Math. Scis. 28(4) (2008) 456–467 (in Chinese). Google Scholar
49. B. H. Sheng , The weighted norm for some Mercer kernel matrices, Acta Math. Sci. 33A(1) (2013) 6–15 (in Chinese). Google Scholar
50. B. H. Sheng, H. X. Liu and H. M. Wang , The learning rate for the kernel regularized regression (KRR) with a differentiable strongly convex loss, Commun. Pure Appl. Anal. 19(8) (2020) 3973–4005. Crossref, Web of Science, Google Scholar
51. B. H. Sheng and J. L. Wang , On the $K$ -functional in learning theory, Anal. Appl. 18 (2020) 423–446. Link, Web of Science, Google Scholar
52. B. H. Sheng, J. L. Wang and Z. X. Chen , The covering number for some Mercer kernel Hilbert spaces on the unit sphere, Taiwanese J. Math. 15(3) (2011) 1325–1340. Crossref, Web of Science, Google Scholar
53. B. H. Sheng, J. L. Wang and P. Li , On the covering number of some Mercer kernel Hilbert spaces, J. Complexity 24(2) (2008) 241–258. Crossref, Web of Science, Google Scholar
54. B. H. Sheng and P. X. Ye , The learning rates of regularized regression based on reproducing kernel Banach spaces, Abstr. Appl. Anal. 2013, Article ID: 694181, 10 pp., https://doi.org/10.1155/2013/694181. Google Scholar
55. S. Smale and D. X. Zhou , Estimating the approximation error in learning theory, Anal. Appl. 1 (2003) 17–41. Link, Web of Science, Google Scholar
56. S. Smale and D. X. Zhou , Shannon sampling and function reconstruction from point values, Bull Amer. Math. Soc. (N. S.) 41(3) (2004) 279–305. Crossref, Web of Science, Google Scholar
57. S. Smale and D. X. Zhou , Learning theory estimates via integral operators and their applications, Constr. Approx. 26 (2007) 153–172. Crossref, Web of Science, Google Scholar
58. B. K. Sriperumbudur, K. Fukumizu and G. R. G. Lanckriet , Universality, characteristic kernels and RKHS embedding of measures, J. Mach. Learn. Res. 9 (2010) 773–780. Google Scholar
59. B. K. Sriperumbudur, K. Fukumizu and G. R. G. Lanckriet , Learning in Hilbert vs. Banach spaces: A measure embedding viewpoint, Adv. Neural Inf. Process. Syst. 24 (2011) 1773–1781. Google Scholar
60. I. Steinwart , Sparseness of support vector machines, J. Mach. Learn. Res. 4 (2003) 1071–1105. Web of Science, Google Scholar
61. I. Steinwart and A. Christmann , Support Vector Machines (Springer-Verlag, New York, 2008). Google Scholar
62. I. Steinwart, D. Hush and C. Scovel , Optimal rates for regularized least square regression, Proc. 22nd Annual Conf. Learning Theory (2009), pp. 79–93. Google Scholar
63. I. Steinwart, C. Scovel , Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs, Constr. Approx. 35 (2012) 363–417. Crossref, Web of Science, Google Scholar
64. H. W. Sun , Mercer theorem for RKHS on noncompact sets, J. Complexity 21 (2005) 337–349. Crossref, Web of Science, Google Scholar
65. H. Z. Tong, D. R. Chen and L. Z. Peng , Analysis of support vector machines regression, Found. Comput. Math. 9 (2009) 243–257. Crossref, Web of Science, Google Scholar
66. H. Z. Tong, D. R. Chen and F. H. Yang , Fast learning rates for regularized regression algorithms, Sci. Sin. Math. 42(12) (2012) 1251–1262 (in Chinese). Crossref, Google Scholar
67. K. Triméche , Generalized Harmonic Analysis and Wavelet Packets (Gordon and Breach Science Publishers, Singapore, 2001). Google Scholar
68. Y. J. Wang, H. Chen, B. Q. Song and H. Li , Regularized modal regression with data-dependent hypothesis spaces, Int. J. Wavelets Multiresolut. Inf. Process. 17(6) (2019) 1950047. Link, Web of Science, Google Scholar
69. R. Wang and Y. S. Xu, Operator reproducing kernel Hilbert spaces, preprint (2015), arXiv:1512.05923v1 [math. FA]. Google Scholar
70. Q. Wu, Y. M. Ying and D. X. Zhou , Learning rates of least-square regularized regression, Found. Comput. Math. 6(2) (2006) 171–192. Crossref, Web of Science, Google Scholar
71. H. K. Xu , Inequalities in Banach spaces with applications, Nonlinear Anal. Theory Methods Appl. 16(12) (1991) 1127–1138. Crossref, Web of Science, Google Scholar
72. Y. S. Xu and Q. Ye , Generalized Mercer kernels and reproducing kernel Banach spaces, Mem. Amer. Math. Soc. 258 (1243) (2019) 1–122. Crossref, Web of Science, Google Scholar
73. Y. M. Ying , Convergence analysis of online algorithms, Adv. Comput. Math. 27 (2007) 273–291. Crossref, Web of Science, Google Scholar
74. Y. M. Ying and D. X. Zhou , Unregularized online learning algorithms with general loss functions, Appl. Comput. Harmon. Anal. 42(2) (2017) 224–244. Crossref, Web of Science, Google Scholar
75. C. Zălinescu , On uniformly convex functions, J. Math. Anal. Appl. 95 (1983) 344–374. Crossref, Web of Science, Google Scholar
76. C. Zălinescu , Convex Analysis in General Vector Spaces (World Scientific, Singapore, 2002). Link, Google Scholar
77. H. Z. Zhang, Y. S. Xu and J. Zhang , Reproducing kernel Banach spaces for machine learning, J. Mach. Learn. Res. 10 (2009) 2741–2775. Web of Science, Google Scholar
78. H. Z. Zhang and J. Zhang , Generalized semi-inner products and applications to regularized learning, J. Math. Anal. Appl. 372 (2010) 181–196. Crossref, Web of Science, Google Scholar
79. H. Z. Zhang and J. Zhang , Regularized learning in Banach spaces as an optimization problem: Representer theorems, J. Glob. Optim. 48(1) (2012) 1–16. Google Scholar
80. D. X. Zhou , The covering number in learning theory, J. Complexity 18 (2002) 739–767. Crossref, Web of Science, Google Scholar
81. D. X. Zhou , Capacity of reproducing kernel spaces in learning theory, IEEE Trans. Inform. Theory 49(7) (2003) 1743–1752. Crossref, Web of Science, Google Scholar
82. G. Z. Zhou and B. H. Sheng , On approximation by a kind of spherical zonal translation network operator, Acta Math. Sci. 25A(2) (2005) 269–276 (in Chinese). Google Scholar