No Access

WEIGHTED THRESHOLD-BASED CLUSTERING FOR INTRUSION DETECTION SYSTEMS

VLADIMIR NIKULIN

Computer Sciences Laboratory, College of Engineering and Computer Science, Australian National University, Canberra, ACT 0200, Australia

Search for more papers by this author

https://doi.org/10.1142/S1469026806001770Cited by:6 (Source: Crossref)

Abstract

Signature-based intrusion detection systems look for known, suspicious patterns in the input data. In this paper we explore compression of labeled empirical data using threshold-based clustering with regularization. The main target of clustering is to compress training dataset to the limited number of signatures, and to minimize the number of comparisons that are necessary to determine the status of the input event as a result. Essentially, the process of clustering includes merging of the clusters which are close enough. As a consequence, we will reduce original dataset to the limited number of labeled centroids. In a complex with k-nearest-neighbor (kNN) method, this set of centroids may be used as a multi-class classifier. Clearly, different attributes have different importance depending on the particular training database and given cost matrix. This importance may be regulated in the definition of the distance using linear weight coefficients. The paper introduces special procedure to estimate above weight coefficients.

The experiments on the KDD-99 intrusion detection dataset have confirmed the effectiveness of the proposed methods.

Keywords:

Remember to check out the Most Cited Articles!
Check out these titles in artificial intelligence!

References

C. Kruegel and T. Toth, Using decision trees to improve signature-based intrusion detection, RAID, eds. G. Vigna, E. Jonsson and C. Kruegel (Springer-Verlag, Berlin, Heidelberg, 2003) pp. 173–191. Google Scholar
F. Gonzales and D. Dasgupta, Genetic Programming and Evolvable Machines 4, 383 (2003), DOI: 10.1023/A:1026195112518. Crossref, Google Scholar
M. Markou and S. Singh, Signal Process 83, 2481 (2003), DOI: 10.1016/j.sigpro.2003.07.018. Crossref, Google Scholar
M. Markou and S. Singh, Signal Process 83, 2499 (2003), DOI: 10.1016/j.sigpro.2003.07.019. Crossref, Google Scholar
K. Wang, G. Cretu and S. Stolfo, Anomalous payload-based worm detection and signature generation, RAID 2005, Lecture Notes in Computer Science 3858, eds. A. Valdes and D. Zamboni (Springer-Verlag, Berlin, Heidelberg, 2006) pp. 227–246. Google Scholar
M. Schonlauet al., Stat. Sci. 16(1), 58 (2001), DOI: 10.1214/ss/998929476. Google Scholar
S. Coull et al. , Intrusion detection: A bioinformatics approach , Proc. 19th Ann. Comput. Security Appl. Conf. ( 2003 ) , DOI: 10.1109/CSAC.2003.1254307 . Google Scholar
R. Goldman, A statistic model for intrusions, RAID 2002, Lecture Notes in Computer Science 2516, eds. A. Wespi, G. Vigna and L. Deri (Springer-Verlag, Berlin, Heidelberg, 2002) pp. 199–218. Google Scholar
C. Kruegel et al. , Bayesian event classification for intrusion detection , Proc. 19th Ann. Comput. Security Appl. Conf. ( 2003 ) , DOI: 10.1109/CSAC.2003.1254306 . Google Scholar
I. Steiwart, D. Hush and C. Scovel, J. Mach. Learn. Res. 6, 211 (2005). Google Scholar
A. Lazarevic et al. , A comparative study of anomaly detection schemes in network intrusion detection , Proc. 4th Int. Conf. Data Mining ( SIAM , 2003 ) . Google Scholar
W. Henley and D. Hand, The Statistician 45(1), 77 (1996), DOI: 10.2307/2348414. Crossref, Google Scholar
L. Portnoy , E. Eskin and S. Stolfo , Intrusion detection with unlabeled data using clustering , ACM CSS Workshop on Data Mining Applied to Security ( 2001 ) . Google Scholar
V. Nikulin, Threshold-based clustering with hard regularization for intrusion detection systems, Workshop on Learning Algorithms Pattern Recognition, AI-2005, eds. R. Ghosh, B. Verma and X. Li (UTS, 2005) pp. 78–87. Google Scholar
X. Qin and W. Lee, Statistical causality analysis of INFOSEC alert data, RAID, eds. G. Vigna, E. Jonsson and C. Kruegel (Springer-Verlag, Berlin, Heidelberg, 2003) pp. 73–93. Google Scholar
J. Hartigan , Clustering Algorithms ( John Wiley & Sons , 1975 ) . Google Scholar
V. Nikulin and A. Smola, Parametric model-based clustering, Data Mining, Intrusion Detection, Information Assurance, and Data Network Security, ed. B. Dasarathy (2005) pp. 190–201, DOI: 10.1117/12.603199. Google Scholar
S. Bhatia, Adaptive k-means clustering, 17th Int. FLAIRS Conf. (2004) pp. 695–699. Google Scholar
KDD, KDD-99 classifier learning contest, Results of the 1999 KDD Classifier Learning Contest: http://www-cse.ucsd.edu/users/elkan/clresults.html (1999) . Google Scholar
R. Lippmannet al., Analysis and results of the 1999 DARPA off-line intrusion detection evaluation, RAID, eds. H. Debar, L. Me and F. Wu (Springer-Verlag, Berlin, Heidelberg, 2000) pp. 162–182. Google Scholar
M. Mahoney and P. Chan, An analysis of the 1999 DARPA/Lincoln laboratory evaluation data for network anomaly detection, RAID, eds. G. Vigna, E. Jonsson and C. Kruegel (Springer-Verlag, Berlin, Heidelberg, 2003) pp. 220–237. Google Scholar
C. Elkan, SIGKDD Explorations, ACM SIGKDD 1(2), 63 (2000). Crossref, Google Scholar
B. Pfahringer, SIGKDD Explorations, ACM SIGKDD 1(2), 65 (2000). Crossref, Google Scholar
I. Levin, SIGKDD Explorations, ACM SIGKDD 1(2), 67 (2000). Crossref, Google Scholar
R. Agrawal , T. Imielinski and A. Swami , Mining association rules between sets of items in large databases , Proc. ACM SIGMOD Conf. ( 1993 ) , DOI: 10.1145/170035.170072 . Google Scholar
R. Agarwal and M. Joshi, PNrule: A new framework for learning classified models in data mining (a case-study in network intrusion detection), Proc. 1st Int. Conf. Data Mining (SIAM, 2001) pp. 225–232. Google Scholar
W. Lee , S. Stolfo and K. Mok , Mining audit data to build intrusion detection models , Mining Audit Data to Build Intrusion Detection Models, KDD-Proceedings ( 1998 ) . Google Scholar
J. Friedman and J. Meulman, J. Roy. Statist. Soc. 66(4), 815 (2004), DOI: 10.1111/j.1467-9868.2004.02059.x. Crossref, Google Scholar
J. Gomez , D. Dasgupta and O. Nasraoui , A new gravitational clustering algorithm , Proc. 4th Int. Conf. Data Mining ( SIAM , 2003 ) . Google Scholar
M. Sabhnani and G. Serpen, Application of machine learning algorithms to KDD intrusion detection dataset within misuse detection context, Proc. Int. Conf. Maching Learning, Models, Technologies and Applications (MLMTA 2003) (2003) pp. 209–215. Google Scholar
D. Yeung and C. Chow, Parzen-window network intrusion detectors, 16th Int. Conf. Pattern Recognition (2002) pp. 385–388, DOI: 10.1109/ICPR.2002.1047476. Google Scholar
V. Nikulin, Universal clustering with family of power loss functions in probabilistic space, IDEAL 2005, Lecture Notes in Computer Science 3578, eds. M. Gallagher, J. Hogan and F. Maire (Springer-Verlag, 2005) pp. 311–318. Google Scholar