Lessons Learned in Multilingual Grounded Language Learning

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 402–412 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 402 Lessons learned in multilingual grounded language learning Ákos Kádár Tilburg University a.kadar@uvt.nl Desmond Elliott∗ University of Copenhagen de@di.ku.dk Marc-Alexandre Côté Microsoft Research Montreal macote@microsoft.com Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Afra Alishahi Tilburg University a.alishahi@uvt.nl Abstract Recent work has shown how to learn better visual-semantic embeddings by leveraging im- age descriptions in more than one language. Here, we investigate in detail which conditions affect the performance of this type of grounded language learning model. We show that multi- lingual training improves over bilingual train- ing, and that low-resource languages benefit from training with higher-resource languages. We demonstrate that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language en- ables further improvements via an additional caption-caption ranking objective. 1 Introduction Multimodal representation learning is largely mo- tivated by evidence of perceptual grounding in hu- man concept acquisition and representation (Barsa- lou et al., 2003). It has been shown that visually grounded word and sentence-representations (Kiela et al., 2014; Baroni, 2016; Elliott and Kádár, 2017; Kiela et al., 2017; Yoo et al., 2017) improve perfor- mance on the downstream tasks of paraphrase iden- tification, semantic entailment, and multimodal ma- chine translation (Dolan et al., 2004; Marelli et al., 2014; Specia et al., 2016). Multilingual sentence representations have also been successfully applied to many-languages-to-one character-level machine translation (Chung et al., 2016) and multilingual dependency parsing (Ammar et al., 2016). Recently, Gella et al. (2017) proposed to learn both bilingual and multimodal sentence represen- tations using images paired with captions indepen- dently collected in English and German. Their results show that bilingual training improves image- sentence ranking performance over a monolingual ∗Work carried out at the University of Edinburgh. baseline, and it improves performance on semantic textual similarity benchmarks (Agirre et al., 2014, 2015). These findings suggest that it may be benefi- cial to consider another language as another modal- ity in a monolingual grounded language learning model. In the grounded learning scenario, descrip- tions of an image in multiple languages can be considered as multiple views of the same or closely related data. These additional views can help over- come the problems of data sparsity, and have prac- tical implications for efficiently collecting image- text datasets in different languages. In real-life applications, many tasks and domains can involve code switching (Barman et al., 2014), which is easier to deal with using a multilingual model. Fur- thermore, it is more convenient to maintain a single multilingual system than one system for each con- sidered language. However, there is a need for a systematic exploration of the conditions under which it is useful to add additional views of the data. We investigate the impact of the following conditions on the performance of a multilingual grounded language learning model in sentence and …

Content truncated to 500 words