Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders

Proceedings of NAACL-HLT 2016, pages 1346–1356, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders Simon ˇSuster University of Groningen Netherlands s.suster@rug.nl Ivan Titov University of Amsterdam Netherlands titov@uva.nl Gertjan van Noord University of Groningen Netherlands g.j.m.van.noord@rug.nl Abstract We present an approach to learning multi-sense word embeddings relying both on monolingual and bilingual information. Our model consists of an encoder, which uses monolingual and bilingual context (i.e. a parallel sentence) to choose a sense for a given word, and a decoder which predicts context words based on the cho- sen sense. The two components are estimated jointly. We observe that the word representa- tions induced from bilingual data outperform the monolingual counterparts across a range of evaluation tasks, even though crosslingual information is not available at test time. 1 Introduction Approaches to learning word embeddings (i.e. real- valued vectors) relying on word context have received much attention in recent years, and the induced rep- resentations have been shown to capture syntactic and semantic properties of words. They have been evaluated intrinsically (Mikolov et al., 2013a; Baroni et al., 2014; Levy and Goldberg, 2014) and have also been used in concrete NLP applications to deal with word sparsity and improve generalization (Turian et al., 2010; Collobert et al., 2011; Bansal et al., 2014; Passos et al., 2014). While most work to date has focused on developing embedding models which rep- resent a word with a single vector, some researchers have attempted to capture polysemy explicitly and have encoded properties of each word with multi- ple vectors (Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Chen et al., 2014; Li and Jurafsky, 2015). In parallel to this work on multi-sense word em- beddings, another line of research has investigated in- tegrating multilingual data, with largely two distinct goals in mind. The first goal has been to obtain repre- sentations for several languages in the same semantic space, which then enables the transfer of a model (e.g., a syntactic parser) trained on annotated training data in one language to another language lacking this annotation (Klementiev et al., 2012; Hermann and Blunsom, 2014; Gouws et al., 2014; Chandar A P et al., 2014). Secondly, information from another language can also be leveraged to yield better first- language embeddings (Guo et al., 2014). Our paper falls in the latter, much less explored category. We ad- here to the view of multilingual learning as a means of language grounding (Faruqui and Dyer, 2014b; Zou et al., 2013; Titov and Klementiev, 2012; Snyder and Barzilay, 2010; Naseem et al., 2009). Intuitively, polysemy in one language can be at least partially resolved by looking at the translation of the word and its context in another language (Kaji, 2003; Ng et al., 2003; Diab and Resnik, 2002; Ide, 2000; Dagan and Itai, 1994; Brown et al., 1991). Better sense as- signment can then lead to better sense-specific word embeddings. We propose a model that uses …

Content truncated to 500 words