Jointly Learning to Embed and Predict with Multiple Languages

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2019–2028, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Jointly Learning to Embed and Predict with Multiple Languages Daniel C. Ferreira∗ André F. T. Martins†∗♯ Mariana S. C. Almeida∗♯ ∗Priberam Labs, Alameda D. Afonso Henriques, 41, 2o, 1000-123 Lisboa, Portugal †Unbabel Lda, Rua Visconde de Santarém, 67-B, 1000-286 Lisboa, Portugal ♯Instituto de Telecomunicações, Instituto Superior Técnico, 1049-001 Lisboa, Portugal {dcf,mla}@priberam.pt, {andre.martins}@unbabel.com Abstract We propose a joint formulation for learn- ing task-specific cross-lingual word em- beddings, along with classifiers for that task. Unlike prior work, which first learns the embeddings from parallel data and then plugs them in a supervised learning problem, our approach is one- shot: a single optimization problem com- bines a co-regularizer for the multilin- gual embeddings with a task-specific loss. We present theoretical results showing the limitation of Euclidean co-regularizers to increase the embedding dimension, a limitation which does not exist for other co-regularizers (such as the ℓ1- distance). Despite its simplicity, our method achieves state-of-the-art accura- cies on the RCV1/RCV2 dataset when transferring from English to German, with training times below 1 minute. On the TED Corpus, we obtain the highest re- ported scores on 10 out of 11 languages. 1 Introduction Distributed representations of text (embeddings) have been the target of much research in natural language processing (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014; Levy et al., 2015). Word embeddings partially capture semantic and syntactic properties of text in the form of dense real vectors, making them apt for a wide variety of tasks, such as language model- ing (Bengio et al., 2003), sentence tagging (Turian et al., 2010; Collobert et al., 2011), sentiment anal- ysis (Socher et al., 2011), parsing (Chen and Man- ning, 2014), and machine translation (Zou et al., 2013). At the same time, there has been a consis- tent progress in devising “universal” multilin- gual models via cross-lingual transfer techniques of various kinds (Hwa et al., 2005; Zeman and Resnik, 2008; McDonald et al., 2011; Ganchev and Das, 2013; Martins, 2015). This line of re- search seeks ways of using data from resource- rich languages to solve tasks in resource-poor languages. Given the difficulty of handcrafting language-independent features, it is highly appeal- ing to obtain rich, delexicalized, multilingual rep- resentations embedded in a shared space. A string of work started with Klementiev et al. (2012) on learning bilingual embeddings for text classification. Hermann and Blunsom (2014) pro- posed a noise-contrastive objective to push the embeddings of parallel sentences to be close in space. A bilingual auto-encoder was proposed by Chandar et al. (2014), while Faruqui and Dyer (2014) applied canonical correlation analysis to parallel data to improve monolingual embeddings. Other works optimize a sum of monolingual and cross-lingual terms (Gouws et al., 2015; Soyer et al., 2015), or introduce bilingual variants of skip-gram (Luong et al., 2015; Coulmance et al., 2015). Recently, Pham et al. (2015) extended the non-compositional paragraph vectors …

Content truncated to 500 words