Regular PapersNo Access

EMPIRICAL ANALYSIS OF THE EFFECT OF DIMENSION REDUCTION AND WORD ORDER ON SEMANTIC VECTORS

LAURIANNE SITBON

Queensland University of Technology, Australia

Search for more papers by this author

PETER D. BRUZA

Queensland University of Technology, Australia

Search for more papers by this author

, and

CHRISTIAN PROKOPP

Queensland University of Technology, Australia

Search for more papers by this author

https://doi.org/10.1142/S1793351X12500055Cited by:2 (Source: Crossref)

Abstract

The aim of this paper is to provide a comparison of various algorithms and parameters to build reduced semantic spaces. The effect of dimension reduction, the stability of the representation and the effect of word order are examined in the context of the five algorithms bearing on semantic vectors: Random projection (RP), singular value decomposition (SVD), non-negative matrix factorization (NMF), permutations and holographic reduced representations (HRR). The quality of semantic representation was tested by means of synonym finding task using the TOEFL test on the TASA corpus. Dimension reduction was found to improve the quality of semantic representation but it is hard to find the optimal parameter settings. Even though dimension reduction by RP was found to be more generally applicable than SVD, the semantic vectors produced by RP are somewhat unstable. The effect of encoding word order into the semantic vector representation via HRR did not lead to any increase in scores over vectors constructed from word co-occurrence in context information. In this regard, very small context windows resulted in better semantic vectors for the TOEFL test.

Keywords:

References

D. Achlioptas, Database-friendly random projections, Proceedings of the Symposium on Principles of Database Systems. (2001) pp. 274–281. Google Scholar
E. Bingham and H. Mannila, Random projection in dimensionality reduction: Applications to image and text data, Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining (2001) pp. 245–250. Google Scholar
D. M. Blei, A. Y. Ng and M. I. Jordan, Journal of Machine Learning Research 3, 993 (2003). Web of Science, Google Scholar
J. A. Bullinaria and J. P. Levy, Behavior Research Methods 39(3), 510 (2007). Crossref, Web of Science, Google Scholar
J. A. Bullinaria and J. P. Levy, Behavior Research Methods 44, (2012). Google Scholar
C. Burgess, K. Livesay and K. Lund, Discourse Processes 25, 211 (1998). Crossref, Web of Science, Google Scholar
J. R. Firth, Studies in Linguistic Analysis 1 (1957). Google Scholar
G. Golub and C. V. Loan , Matrix Computations ( John Hopkins University Ptess , 1996 ) . Google Scholar
T. L. Griffiths and M. Steyvers, Psychological Review 114(2), 211 (2007). Crossref, Web of Science, Google Scholar
M. Humphreys, J. Bain and R. Pike, Psychological Review 96, 208 (1989). Crossref, Web of Science, Google Scholar
M. N. Jones, W. Kintsch and D. J. K. Mewhort, Journal of Memory and Language 55, 534 (2006). Crossref, Web of Science, Google Scholar
M. N. Jones and D. J. K. Mewhort, Psychological Review 114(1), 1 (2007). Crossref, Web of Science, Google Scholar
P. Kanerva , J. Kristoferson and A. Holst , Random indexing of text samples for latent semantic analysis , Proceedings of the 22nd Annual Conference of the Cognitive Science Society ( 2000 ) . Google Scholar
T. Landauer, The Psychology of Learning and Motivation 41, ed. B. Ross (Academic Press, 2002) pp. 43–84. Google Scholar
T. Landauer and S. T. Dumais, Psychological Review 104(2), 211 (1997). Crossref, Web of Science, Google Scholar
D. D. Lee and H. S. Seung, Nature 401, 788 (1999). Crossref, Web of Science, Google Scholar
D. D. Lee and H. S. Seung, Advances in Neural Information Processing Systems 13, 556 (2001). Google Scholar
W. Lowe, Towards a theory of semantic space, Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society (Lawrence Erlbaum Associates, 2001) pp. 576–581. Google Scholar
K. Lund and C. Burgess, Behaviour Research Methods, Instruments & Computers 28(2), 203 (1996). Crossref, Google Scholar
K. Lund and C. Burgess, Behavior Research Methods, Instruments & Computers 28(2), 203 (1996). Crossref, Google Scholar
C. D. Manning and H. Schütze , Foundations of Statistical Natural Language Processing ( MIT Press , Cambridge, MA , 1999 ) . Google Scholar
T. A. Plate , Holographic Reduced Representation: Distributed Representation for Cognitive Structures , CSLI Lecture Notes 150 ( 2003 ) . Google Scholar
R. Rapp, Word sense discovery based on sense descriptor dissimilarity, Proceedings of the Ninth Machine Translation Summit (2003) pp. 315–322. Google Scholar
M. Sahlgren , An introduction to random indexing , Proceedings of Methods and Applications of Semantic Indexing Workshop, 7th International Conference on Terminology and Knowledge Engineering ( 2005 ) . Google Scholar
M. Sahlgren , A. Holst and P. Kanerva , Permutations as a means to encode order in word space , Proceedings of the 30th Annual Meeting of the Cognitive Science Society (CogSci'08) ( 2008 ) . Google Scholar
H. Schütze, Computational linguistics 24(1), 97 (1998). Web of Science, Google Scholar
L. Sitbon , P. Bellot and P. Blache , Evaluation of lexical resources and semantic networks on a corpus of mental associations , Proceedings of the 6th Language Resources and Evaluation Conference ( 2008 ) . Google Scholar
L. Sitbon and P. Bruza, On the relevance of documents for semantic representation, Proceedings of the 13th Australiasian Document Computing Symposium (2008) pp. 19–22. Google Scholar
P. Turney, Mining the web for synonyms: Pmi-ir versus lsa on toefl, Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001), eds. L. De Raedt and P. Flach (2001) pp. 491–501. Google Scholar
P. D. Turneyet al., Combining independent modules to solve multiple-choice synonym and analogy problems, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03) (2003) pp. 482–489. Google Scholar
P. Turney and P. Pantel, Journal of Artificial Intelligence Research 37, 141 (2010). Crossref, Web of Science, Google Scholar
D. Widdows , Geometry and Meaning ( CSLI Publications , 2004 ) . Google Scholar
D. Widdows and K. Ferraro , Semantic vectors: A scalable open source package and online technology management application , Proceedings of the 6th Language Resources and Evaluation Conference ( 2008 ) . Google Scholar
W. Xu, X. Liu and Y. Gong, Document clustering based on non-negative matrix factorization, Proceedings of SIGIR'03 (2003) pp. 267–273. Google Scholar