SPECIAL ISSUE: Fusion of Multiple Classifiers; Edited by F. Roli and J. KittlerNo Access

BIAS-VARIANCE CONTROL VIA HARD POINTS SHAVING

STEFANO MERLER

ITC-irst, Centro per la Ricerca Scientifica e Tecnologica, via Sommarive 18, I-38050, Povo, Trento, Italy

Search for more papers by this author

BRUNO CAPRILE

ITC-irst, Centro per la Ricerca Scientifica e Tecnologica, via Sommarive 18, I-38050, Povo, Trento, Italy

Search for more papers by this author

, and

CESARE FURLANELLO

ITC-irst, Centro per la Ricerca Scientifica e Tecnologica, via Sommarive 18, I-38050, Povo, Trento, Italy

Search for more papers by this author

https://doi.org/10.1142/S0218001404003460Cited by:13 (Source: Crossref)

Abstract

In this paper, we propose a regularization technique for AdaBoost. The method implements a bias-variance control strategy in order to avoid overfitting in classification tasks on noisy data. The method is based on a notion of easy and hard training patterns as emerging from analysis of the dynamical evolutions of AdaBoost weights. The procedure consists in sorting the training data points by a hardness measure, and in progressively eliminating the hardest, stopping at an automatically selected threshold. Effectiveness of the method is tested and discussed on synthetic as well as real data.

Keywords:

References

L. Breiman et al. , Classification and Regression Trees ( Wadsworth , Belmont, CA , 1984 ) . Google Scholar
B. Caprile, C. Furlanello and S. Merler, Multiple Classifier Systems, Lecture Notes in Computer Science 2364, eds. J. Kittler and F. Roli (Springer, 2002) pp. 72–80. Crossref, Google Scholar
T. G. Dietterich, Mach. Learn. 40(2), 139 (2000), DOI: 10.1023/A:1007607513941. Crossref, Web of Science, Google Scholar
P. Domingos, A unified bias-variance decomposition and its applications, Proc. 17th Int. Conf. Machine Learning (Morgan Kaufmann, San Francisco, CA, 2000) pp. 231–238. Google Scholar
B. Efron and R. Tibshirani , An Introduction to the Bootstrap , Monographs on Statistic and Applied Probability 57 ( Chapman & Hall , 1993 ) . Crossref, Google Scholar
Y. Freund and R. E. Schapire, J. Comput. Syst. Sci. 55(1), 119 (1997), DOI: 10.1006/jcss.1997.1504. Crossref, Web of Science, Google Scholar
J. Friedman, T. Hastie and R. Tibshirani, Ann. Statist. 38(2), 337 (2000). Crossref, Google Scholar
A. J. Grove and D. Schuurmans, Boosting in the limit: maximizing the margin of learned ensembles, AAAI/IAAI (1998) pp. 692–699. Google Scholar
W. Jiang, Process consistency for AdaBoost, Technical Report, Department of Statistics, Northwestern University, 2000 . Google Scholar
J. R. Quinlan, Bagging, boosting, and C4.5, Thirteenth Nat. Conf. Artificial Intelligence (AAAI Press/MIT Press, Cambridge, 1996) pp. 163–175. Google Scholar
G. Rätsch, T. Onoda and K. R. Müller, Mach. Learn. 42, 287 (2001). Crossref, Web of Science, Google Scholar
R. E. Schapire , The boosting approach to machine learning: an overview , MSRI Workshop on Nonlinear Estimation and Classification ( 2002 ) . Google Scholar
R. E. Schapireet al., Ann. Statist. 26(5), 1651 (1998). Web of Science, Google Scholar
M. Sebban, R. Nock and S. Lallich, J. Mach. Learn. Res. 3, 863 (2002), DOI: 10.1162/jmlr.2003.3.4-5.863. Web of Science, Google Scholar
V. Vapnik , The Nature of Statistical Learning Theory , Statistics for Engineering and Information Science ( Springer Verlag , 2000 ) . Crossref, Google Scholar

Vol. 18, No. 05

Metrics

Downloaded 27 times

History

Keywords

PDF download

System Upgrade on Tue, May 28th, 2024 at 2am (EDT)

BIAS-VARIANCE CONTROL VIA HARD POINTS SHAVING

Abstract

Recommended