Improving Vector Space Word Representations Using Multilingual Correlation

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462–471, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Improving Vector Space Word Representations Using Multilingual Correlation Manaal Faruqui and Chris Dyer Carnegie Mellon University Pittsburgh, PA, 15213, USA {mfaruqui, cdyer}@cs.cmu.edu Abstract The distributional hypothesis of Harris (1954), according to which the meaning of words is evidenced by the contexts they occur in, has motivated several effec- tive techniques for obtaining vector space semantic representations of words using unannotated text corpora. This paper ar- gues that lexico-semantic content should additionally be invariant across languages and proposes a simple technique based on canonical correlation analysis (CCA) for incorporating multilingual evidence into vectors generated monolingually. We evaluate the resulting word representations on standard lexical semantic evaluation tasks and show that our method produces substantially better semantic representa- tions than monolingual techniques. 1 Introduction Data-driven learning of vector-space word embed- dings that capture lexico-semantic properties is a technique of central importance in natural lan- guage processing. Using cooccurrence statistics from a large corpus of text (Deerwester et al., 1990; Turney and Pantel, 2010),1 it is possible to construct high-quality semantic vectors — as judged by both correlations with human judge- ments of semantic relatedness (Turney, 2006; Agirre et al., 2009) and as features for downstream applications (Turian et al., 2010). The observation that vectors representing cooc- currence tendencies would capture meaning is ex- pected according to the distributional hypothe- sis (Harris, 1954), famously articulated by Firth 1Related approaches use the internal representations from neural network models of word sequences (Collobert and We- ston, 2008) or continuous bags-of-context wordsels (Mikolov et al., 2013a) to arrive at vector representations that likewise capture cooccurence tendencies and meanings. (1957) as You shall know a word by the company it keeps. Although there is much evidence in fa- vor of the distributional hypothesis, in this paper we argue for incorporating translational context when constructing vector space semantic models (VSMs). Simply put: knowing how words trans- late is a valuable source of lexico-semantic infor- mation and should lead to better VSMs. Parallel corpora have long been recognized as valuable for lexical semantic applications, in- cluding identifying word senses (Diab, 2003; Resnik and Yarowsky, 1999) and paraphrase and synonymy relationships (Bannard and Callison- Burch, 2005). The latter work (which we build on) shows that if different words or phrases in one lan- guage often translate into a single word or phrase type in a second language, this is good evidence that they are synonymous. To illustrate: the En- glish word forms aeroplane, airplane, and plane are observed to translate into the same Hindi word: ���� ��� (vaayuyaan). Thus, even if we did not know the relationship between the English words, this translation fact is evidence that they all have the same meaning. How can we exploit information like this when constructing VSMs? We propose a technique that first constructs independent VSMs in two lan- guages and then projects them onto a common …

Content truncated to 500 words