75 Languages, 1 Model: Parsing Universal Dependencies Universally

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2779–2795, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2779 75 Languages, 1 Model: Parsing Universal Dependencies Universally Dan Kondratyuk1,2 and Milan Straka1 1Charles University, Institute of Formal and Applied Linguistics 2Saarland University, Department of Computational Linguistics dankondratyuk@gmail.com, straka@ufal.mff.cuni.cz Abstract We present UDify, a multilingual multi-task model capable of accurately predicting uni- versal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for all 124 Universal Dependencies treebanks across 75 languages. By leveraging a multi- lingual BERT self-attention model pretrained on 104 languages, we found that fine-tuning it on all datasets concatenated together with simple softmax classifiers for each UD task can meet or exceed state-of-the-art UPOS, UFeats, Lemmas, (and especially) UAS, and LAS scores, without requiring any recurrent or language-specific components. We evaluate UDify for multilingual learning, showing that low-resource languages benefit the most from cross-linguistic annotations. We also evalu- ate for zero-shot learning, with results suggest- ing that multilingual training provides strong UD predictions even for languages that nei- ther UDify nor BERT have ever been trained on. Code for UDify is available at https:// github.com/hyperparticle/udify. 1 Introduction In the absence of annotated data for a given lan- guage, it can be considerably difficult to create models that can parse the language’s text accu- rately. Multilingual modeling presents an attrac- tive way to circumvent this low-resource limita- tion. In a similar way learning a new language can enhance the proficiency of a speaker’s previ- ous languages (Abu-Rabia and Sanitsky, 2010), a model which has access to multilingual informa- tion can begin to learn generalizations across lan- guages that would not have been possible through monolingual data alone. Works such as McDonald et al. (2011); Naseem et al. (2012); Duong et al. (2015); Ammar et al. (2016); de Lhoneux et al. (2018); Kitaev and Klein (2018); Mulcaire et al. (2019) consistently demonstrate how pairing the “The best optimizer is grad student descent” The best op ##timi ##zer is grad student descent ... BERT x→x (optimizer) Layer Attention 4 (is) nsubj Number=Sing NOUN Dependency Head Dependency Tag Lemma UFeats UPOS Figure 1: An illustration of the UDify network ar- chitecture with task-specific layer attention, inputting word tokens and outputting UD annotations for each token. training data of similar languages can boost eval- uation scores of models predicting syntactic infor- mation like part-of-speech and dependency trees. Multilinguality not only can improve a model’s evaluation performance, but can also reduce the cost of training multiple models for a collection of languages (Johnson et al., 2017; Smith et al., 2018). However, scaling to a higher number of lan- guages can often be problematic. Without an ample supply of training data for the considered languages, it can be difficult to form appropri- ate generalizations and especially difficult if those 2780 languages are distant from each other. But re- cent techniques in language model pretraining can profit from a drastically …

Content truncated to 500 words