The Fragility of Multi-Treebank Parsing Evaluation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5345–5359 October 12–17, 2022. 5345 The Fragility of Multi-Treebank Parsing Evaluation Iago Alonso-Alonso, David Vilares and Carlos Gómez-Rodríguez Universidade da Coruña, CITIC Departamento de Ciencias de la Computación y Tecnologías de la Información Campus de Elviña s/n, 15071 A Coruña, Spain {iago.alonso,david.vilares,carlos.gomez}@udc.es Abstract Treebank selection for parsing evaluation and the spurious effects that might arise from a bi- ased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclu- sions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this ex- periment, create vast amounts of random sub- sets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to de- tect potentially harmful strategies. 1 Introduction A limitation in NLP evaluation lies in the asso- ciation between solving a dataset versus solving a task. Datasets are domain-specific, their sizes differ and they are only available for a handful of languages and cultures (Hershcovich et al., 2022). Yet, we often ignore that the chances that these re- sults generalize in the real world are scarce. In this context, the conclusions extracted from a single dataset should be taken with caution. For dependency parsing, the Universal Depen- dencies framework (UD; Zeman et al., 2020) mit- igates some of these issues. For instance, version 2.8 of UD includes 202 treebanks and 114 lan- guages covering diverse linguistic typologies, tree- banks with different amounts of data, and domains. Paradoxically, this also complicates decisions when it comes to comparing dependency parsers in mul- tilingual environments, which can be summarized as: how to choose a small but representative set of treebanks? Although there are shared tasks (Ze- man et al., 2017, 2018) that do consider experi- ments over a wide set of treebanks and help under- stand parsing models, such setups do not usually stick when the shared tasks end, and authors often run their models only in a handful of treebanks (de Lhoneux and Nivre, 2016; Ma et al., 2018; Kulmizev et al., 2019, inter alia). This mostly hap- pens for justified reasons: lack of computational re- sources to train the models in a reasonable amount of time, energy usage concerns, difficulties to sum- marize large experiments, or interest in specific phenomena (e.g. non-projectivity). Thus, a good treebank selection strategy is crucial to reduce the chances of selecting an unrepresentative subset of treebanks, which could lead to weak conclusions. Furthermore, even when using the whole UD col- lection is viable, treebank selection can still be relevant as UD is not a representative sample of languages (e.g., 62 out of the 114 languages in v2.8 are Indo-European), so coarse-grained …

Content truncated to 500 words