Special Issue on Recent Advances in Techniques for Intelligent Systems; Guest Editors: Susan Haller and Ingrid RussellNo Access

KEYWORD EXTRACTION FROM A SINGLE DOCUMENT USING WORD CO-OCCURRENCE STATISTICAL INFORMATION

Y. MATSUO

National Institute of Advanced Industrial Science and Technology, Japan

Search for more papers by this author

and

M. ISHIZUKA

University of Tokyo, Japan

Search for more papers by this author

https://doi.org/10.1142/S0218213004001466Cited by:471 (Source: Crossref)

Abstract

We present a new keyword extraction algorithm that applies to a single document without using a corpus. Frequent terms are extracted first, then a set of co-occurrences between each term and the frequent terms, i.e., occurrences in the same sentences, is generated. Co-occurrence distribution shows importance of a term in the document as follows. If the probability distribution of co-occurrence between term a and the frequent terms is biased to a particular subset of frequent terms, then term a is likely to be a keyword. The degree of bias of a distribution is measured by the χ²-measure. Our algorithm shows comparable performance to tfidf without using a corpus.

Keywords:

Remember to check out the Most Cited Articles!
Check out Notable Titles in Artificial Intelligence.

References

K. W. Church and P. Hanks, Computational Linguistics 16(1), 22 (1990). Google Scholar
I. Dagan, L. Lee and F. Pereira, Machine Learning 34(1), 43 (1999), DOI: 10.1023/A:1007537716579. Crossref, Web of Science, Google Scholar
T. Dunning, Computational Linguistics 19(1), 61 (1993). Google Scholar
R. Feldmanet al., Text mining at the term level, Proceedings of the Second European Symposium on Principles of Data Mining and Knowledge Discovery (1998) p. 65. Google Scholar
Johannes Fürnkranz. A study using n-grams features for text categorization. Technical Report OEFAI-TR-98-30, Austrian Research Institute for Artificial Intelligence, 1998 . Google Scholar
Thomas Hofmann and Jan Puzicha. Statistical models for co-occurrence data. Technical Report AIM-1625, Massachusetts Institute of Technology, 1998 . Google Scholar
K. Kageura and B. Umino, Terminology 3(2), 259 (1996). Crossref, Google Scholar
H. P. Luhn, IBM Journal of Research and Development 1(4), 390 (1957), DOI: 10.1147/rd.14.0309. Google Scholar
M. Nagao, M. Mizutani and H. Ikeda, Transactions of Information Processing Society of Japan 17(2), 110 (1976). Google Scholar
T. Noreault , M. McGill and M. B. Koll , A Performance Evaluation of Similarity Measure, Document Term Weighting Schemes and Representations in a Boolean Environment ( Butterworths , London , 1977 ) . Google Scholar
Y. Ohsawa , N. E. Benson and M. Yachida , KeyGraph: Automatic indexing by co-occurrence graph based on building construction metaphor , Proceedings of the Advanced Digital Library Conference . Google Scholar
Yukio Ohsawa. Chance discoveries for making decisions in complex real world. New Generation Computing, to appear . Google Scholar
F. Pereira, N. Tishby and L. Lee, Distributional clustering of English words, Proceedings of the 31th Meeting of the Association for Computational Linguistics pp. 183–190. Google Scholar
M. F. Porter, Program 14(3), 130 (1980). Crossref, Google Scholar
M. Rajman and R. Besancon , Text mining – knowledge extraction from unstructured textual data , Proceedings of the 6th Conference of International Federation of Classification Societies . Google Scholar
G. Salton , Automatic Text Processing ( Addison-Wesley , 1988 ) . Google Scholar
H. Schutze and J. O. Pederson, Information Processing and Management 33(3), 307 (1997), DOI: 10.1016/S0306-4573(96)00068-4. Crossref, Web of Science, Google Scholar
K. Sparck-Jones, Journal of Documentation 28(5), 111 (1972). Google Scholar
K. Tanaka and H. Iwasaki, Extraction of lexical translations from non-aligned corpora, Proceedings of the 16th International Conference on Computational Linguistics p. 580. Google Scholar
A. M. Turing, Mind 59, 433 (1950). Google Scholar