Multilingual Models for Compositional Distributed Semantics

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 58–68, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Multilingual Models for Compositional Distributed Semantics Karl Moritz Hermann and Phil Blunsom Department of Computer Science University of Oxford Oxford, OX1 3QD, UK {karl.moritz.hermann,phil.blunsom}@cs.ox.ac.uk Abstract We present a novel technique for learn- ing semantic representations, which ex- tends the distributional hypothesis to mul- tilingual data and joint-space embeddings. Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences, while maintaining sufficient distance between those of dissimilar sentences. The mod- els do not rely on word alignments or any syntactic information and are success- fully applied to a number of diverse lan- guages. We extend our approach to learn semantic representations at the document level, too. We evaluate these models on two cross-lingual document classification tasks, outperforming the prior state of the art. Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic rela- tionships across languages without paral- lel data. 1 Introduction Distributed representations of words provide the basis for many state-of-the-art approaches to var- ious problems in natural language processing to- day. Such word embeddings are naturally richer representations than those of symbolic or discrete models, and have been shown to be able to capture both syntactic and semantic information. Success- ful applications of such models include language modelling (Bengio et al., 2003), paraphrase detec- tion (Erk and Pad´o, 2008), and dialogue analysis (Kalchbrenner and Blunsom, 2013). Within a monolingual context, the distributional hypothesis (Firth, 1957) forms the basis of most approaches for learning word representations. In Figure 1: Model with parallel input sentences a and b. The model minimises the distance between the sentence level en- coding of the bitext. Any composition functions (CVM) can be used to generate the compositional sentence level repre- sentations. this work, we extend this hypothesis to multilin- gual data and joint-space embeddings. We present a novel unsupervised technique for learning se- mantic representations that leverages parallel cor- pora and employs semantic transfer through com- positional representations. Unlike most methods for learning word representations, which are re- stricted to a single language, our approach learns to represent meaning across languages in a shared multilingual semantic space. We present experiments on two corpora. First, we show that for cross-lingual document clas- sification on the Reuters RCV1/RCV2 corpora (Lewis et al., 2004), we outperform the prior state of the art (Klementiev et al., 2012). Second, we also present classification results on a mas- sively multilingual corpus which we derive from the TED corpus (Cettolo et al., 2012). The re- sults on this task, in comparison with a number of strong baselines, further demonstrate the relevance of our approach and the success of our method in learning multilingual semantic representations over a wide range of languages. 58 2 Overview Distributed representation learning describes the task of learning continuous representations for dis- crete objects. Here, …

Content truncated to 500 words