Climbing the Tower of Treebanks: Improving Low-Resource Dependency Parsing via Hierarchical Source Selection

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4878–4888 August 1–6, 2021. ©2021 Association for Computational Linguistics 4878 Climbing the Tower of Treebanks: Improving Low-Resource Dependency Parsing via Hierarchical Source Selection Goran Glavaˇs University of Mannheim Data and Web Science Group goran@informatik.uni-mannheim.de Ivan Vuli´c University of Cambridge Language Technology Lab iv250@cam.ac.uk Abstract Recent work on multilingual dependency pars- ing focused on developing highly multilingual parsers that can be applied to a wide range of low-resource languages. In this work, we sub- stantially outperform such “one model to rule them all” approach with a heuristic selection of languages and treebanks on which to train the parser for a specific target language. Our approach, dubbed TOWER, first hierarchically clusters all Universal Dependencies languages based on their mutual syntactic similarity com- puted from human-coded URIEL vectors. For each low-resource target language, we then climb this language hierarchy starting from the leaf node of that language and heuristi- cally choose the hierarchy level at which to col- lect training treebanks. This treebank selection heuristic is based on: (i) the aggregate size of all treebanks subsumed by the hierarchy level and (ii) the similarity of the languages in the training sample with the target language. For languages without development treebanks, we additionally use (ii) for model selection (i.e., early stopping) in order to prevent overfitting to development treebanks of closest languages. Our TOWER approach shows substantial gains for low-resource languages over two state-of- the-art multilingual parsers, with more than 20 LAS point gains for some of those languages. Parsing models and code available at: https: //github.com/codogogo/towerparse. 1 Introduction Syntactic parsing – grounded in a wide variety of formalisms (Taylor et al., 2003; De Marneffe et al., 2006; Hockenmaier and Steedman, 2007; Nivre et al., 2016, inter alia) – has been the backbone of natural language processing (NLP) for decades, and an indispensable preprocessing step for tack- ling higher-level language understanding tasks. A recent major paradigm shift in NLP towards large- scale pretrained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) and their end-to-end fine-tuning for downstream tasks has reduced the downstream relevance of su- pervised syntactic parsing. What is more, there is more and more evidence that PLMs implicitly ac- quire rich syntactic knowledge through large-scale pretraining (Hewitt and Manning, 2019; Chi et al., 2020) and that exposing them to explicit syntax from human-coded treebanks does not offer sig- nificant language understanding benefits (Kuncoro et al., 2020; Glavaˇs and Vuli´c, 2021). In order to implicitly acquire syntactic competencies, however, PLMs need language-specific corpora at the scale at which it can only be obtained for a tiny por- tion of world’s 7,000+ languages. For the remain- ing vast majority of languages – with limited-size monolingual corpora – explicit syntax still provides valuable linguistic bias for more sample-efficient learning in downstream NLP tasks. Reliable syntactic parsing requires annotated treebanks of reasonable size: this prerequisite is, unfortunately, satisfied for even fewer languages. Despite the multi-year, well-coordinated annota- tion efforts …

Content truncated to 500 words