Does Syntactic Knowledge in Multilingual Language Models Transfer Across Languages?

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 374–377 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 374 Does Syntactic Knowledge in Multilingual Language Models Transfer Across Languages? Prajit Dhar Arianna Bisazza Leiden Institute of Advanced Computer Science Leiden University, The Netherlands {p.dhar,a.bisazza}@liacs.leidenuniv.nl Abstract Recent work has shown that neural models can be successfully trained on multiple languages simultaneously. We investigate whether such models learn to share and exploit common syntactic knowledge among the languages on which they are trained. This extended abstract presents our preliminary results. 1 Introduction Recent work has shown that state-of-the-art neu- ral models of language and translation can be suc- cessfully trained on multiple languages simulta- neously without changing the model architecture ( ¨Ostling and Tiedemann, 2017; Johnson et al., 2017). In some cases this leads to improved per- formance compared to models only trained on a specific language, suggesting that multilingual models learn to share useful knowledge cross- lingually through their learned representations. While a large body of research exists on the mul- tilingual mind, the mechanisms explaining knowl- edge sharing in computational multilingual mod- els remain largely unknown: What kind of knowl- edge is shared among languages? Do multilingual models mostly benefit from a better modeling of lexical entries or do they also learn to share more abstract linguistic categories? We focus on the case of language models (LM) trained on two languages, one of which (L1) is over-resourced with respect to the other (L2), and investigate whether the syntactic knowledge learned for L1 is transferred to L2. To this end we use the long-distance agreement benchmark re- cently introduced by Gulordava et al. (2018). 2 Background The recent advances in neural networks have opened the way to the design of architecturally simple multilingual models for various NLP tasks, such as language modeling or next word predic- tion (Tsvetkov et al., 2016; ¨Ostling and Tiede- mann, 2017; Malaviya et al., 2017; Tiedemann, 2018), translation (Dong et al., 2015; Zoph et al., 2016; Firat et al., 2016; Johnson et al., 2017), morphological reinflection (Kann et al., 2017) and more (Bjerva, 2017). A practical benefit of training models multilingually is to transfer knowledge from high-resource languages to low- resource ones and improve task performance in the latter. Here we aim at understanding how lin- guistic knowledge is transferred among languages, specifically at the syntactic level, which to our knowledge has not been studied so far. Assessing the syntactic abilities of monolin- gual neural LMs trained without explicit super- vision has been the focus of several recent stud- ies: Linzen et al. (2016) analyzed the performance of LSTM LMs at an English subject-verb agree- ment task, while Gulordava et al. (2018) extended the analysis to various long-range agreement pat- terns in different languages. The latter study found that state-of-the-art LMs trained on a standard log- likelihood objective capture non-trivial patterns of syntactic agreement and can approach the perfor- mance levels of humans, even when tested on syn- …

Content truncated to 500 words