Genre as Weak Supervision for Cross-lingual Dependency Parsing

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4786–4802 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4786 Genre as Weak Supervision for Cross-lingual Dependency Parsing Max Müller-Eberstein and Rob van der Goot and Barbara Plank Department of Computer Science IT University of Copenhagen, Denmark mamy@itu.dk, robv@itu.dk, bapl@itu.dk Abstract Recent work has shown that monolingual masked language models learn to represent data-driven notions of language variation which can be used for domain-targeted train- ing data selection. Dataset genre labels are al- ready frequently available, yet remain largely unexplored in cross-lingual setups. We har- ness this genre metadata as a weak supervi- sion signal for targeted data selection in zero- shot dependency parsing. Specifically, we project treebank-level genre information to the finer-grained sentence level, with the goal to amplify information implicitly stored in unsu- pervised contextualized representations. We demonstrate that genre is recoverable from multilingual contextual embeddings and that it provides an effective signal for training data selection in cross-lingual, zero-shot scenarios. For 12 low-resource language treebanks, six of which are test-only, our genre-specific meth- ods significantly outperform competitive base- lines as well as recent embedding-based meth- ods for data selection. Moreover, genre-based data selection provides new state-of-the-art re- sults for three of these target languages. 1 Introduction Multilingual masked language models (MLMs) trained on immense quantities of heterogeneous texts (Devlin et al., 2019; Brown et al., 2020; Con- neau et al., 2020) have recently made applications such as highly cross-lingual dependency parsing a reality (Kondratyuk and Straka, 2019). Adja- cently, it has also been recognized that they cap- ture characteristics relevant for training data se- lection (Aharoni and Goldberg, 2020) and can be efficiently fine-tuned for higher task-specific per- formance (Gururangan et al., 2020; Dai et al., 2020; Lauscher et al., 2020; Üstün et al., 2020). These considerations are especially important in compu- tationally restricted environments and when data from the target distribution are unavailable. Universal Dependencies Target Parser Genre Selection Targeted Training Data . . . Figure 1: Genre-driven Training Data Selection for a zero-shot target treebank. In absence of annotated in- language data, we propose genre as a weak supervision signal for targeted instance selection from a large pool of out-of-language treebanks. Universal Dependencies (Nivre et al., 2020; UD) provides an extensive testing ground for such sce- narios: Its language diversity is constantly increas- ing (from 10 in v1.0 to 104 in v2.7) and low- resource languages are often limited to a single test-set-only treebank. As most of the 7,000+ lan- guages in the world similarly lack any annotated training data, effective zero-shot transfer learning is crucial for achieving wider linguistic coverage. Criteria for selecting training data within such settings vary, and a practitioner may determine rele- vance by proxy of language relatedness or treebank content. This leads us to the question: If our goal is to develop a parser for a known domain in an unseen language, can a signal such as genre guide our selection of cross-lingual training data from a …

Content truncated to 500 words