Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2391–2406, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2391 Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations Arturo Oncevay Barry Haddow Alexandra Birch School of Informatics, University of Edinburgh, Scotland {a.oncevay,a.birch}@ed.ac.uk,bhaddow@inf.ed.ac.uk Abstract Sparse language vectors from linguistic typol- ogy databases and learned embeddings from tasks like multilingual machine translation have been investigated in isolation, without analysing how they could benefit from each other’s language characterisation. We propose to fuse both views using singular vector canon- ical correlation analysis and study what kind of information is induced from each source. By inferring typological features and language phylogenies, we observe that our representa- tions embed typology and strengthen correla- tions with language relationships. We then take advantage of our multi-view language vector space for multilingual machine trans- lation, where we achieve competitive overall translation accuracy in tasks that require in- formation about language similarities, such as language clustering and ranking candidates for multilingual transfer. With our method, which is also released as a tool, we can easily project and assess new languages without expensive retraining of massive multilingual or ranking models, which are major disadvantages of re- lated approaches. 1 Introduction Recent surveys consider linguistic typology as a po- tential source of knowledge to support multilingual natural language processing (NLP) tasks (O’Horan et al., 2016; Ponti et al., 2019). Linguistic typol- ogy studies language variation in terms of their functional processes (Comrie, 1989). Several typo- logical knowledge bases (KB) have been crafted, from where we can extract categorical language features (Littell et al., 2017). Nevertheless, their sparsity and reduced coverage present a challenge for an end-to-end integration into NLP algorithms. For example, the World Atlas of Language Struc- ture (WALS; Dryer and Haspelmath, 2013) encodes 143 features for 2,679 languages, but their mean coverage per language is barely around 14%. Dense and data-driven language representations have emerged in response. They are computed from multilingual settings of language modelling ( ¨Ostling and Tiedemann, 2017) and neural machine translation (NMT) (Malaviya et al., 2017). How- ever, the language diversity in the corpus-based representations is limited. The language coverage could be broadened with other knowledge, such as that encoded in WALS, to distinguish even more language properties. Therefore, to obtain the best of both views (KB and task-learned) with minimal information loss, we project a shared space of dis- crete and continuous features using a variant of canonical correlation analysis (Raghu et al., 2017). For our study, we fuse language-level embed- dings from multilingual machine translation with syntactic features of WALS. We inspect how much typological knowledge is present by predicting fea- tures for new languages. Then, we infer language phylogenies and inspect whether specific relation- ships are induced from the task-learned vectors. Furthermore, to demonstrate that our approach has practical benefits in NLP, we apply our lan- guage vectors in multilingual NMT with language clustering (Tan et al., 2019) and adapt the ranking of related languages for …

Content truncated to 500 words