Investigating UD Treebanks via Dataset Difficulty Measures

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1076–1089 May 2-6, 2023 ©2023 Association for Computational Linguistics Investigating UD Treebanks via Dataset Difficulty Measures Artur Kulmizev Joakim Nivre Department of Linguistics and Philology Uppsala University {artur.kulmizev,joakim.nivre}@lingfil.uu.se Abstract Treebanks annotated with Universal Depen- dencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent quality characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD tree- banks using three recently proposed accuracy- free dataset analysis methods: dataset cartogra- phy, V-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little informa- tion that is usable by a parser to surpass what can be achieved by simple heuristics. Further- more, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein. 1 Introduction Datasets have long played a crucial role in dictat- ing the pace of progress in NLP. Their function, for most tasks, is largely two-fold: 1) to collect data points (and their corresponding gold-standard labels) on which statistical models can be trained, and 2) to serve as benchmarks though which var- ious models can be evaluated and compared. In recent years, much research has been devoted to- wards developing new datasets, tasks, and bench- marks for NLP — so as to articulate the distin- guishing aspects of a bevy of new neural models. Syntactic parsing has remained an active area of re- search in this regard, and Universal Dependencies (UD) (Nivre et al., 2016, 2020; de Marneffe et al., 2021) has emerged as a crucial initiative within NLP, offering a set of cross-lingually consistent annotation principles that have since been adapted to over 100 languages. Notably, the CoNLL shared tasks of 2017 and 2018 (Zeman et al., 2017, 2018) featured UD at the forefront, inviting researchers to submit systems that could not only parse, but process the entirety of UD across its numerous annotation layers. Beyond parsing, UD has also been utilized for a variety of other ends, including cross-lingual transfer (Am- mar et al., 2016; Pires et al., 2019; Wu and Dredze, 2020; Lauscher et al., 2020), domain adaptation (Li et al., 2019, 2020; Stymne, 2020), and linguistic typology (Futrell et al., 2015; Hahn et al., 2020; Levshina, 2019). Though UD and other initiatives have aided in driving recent advances in NLP, overall progress has typically been measured via aggregate accuracy metrics, which provide little more than a bird’s eye view into the data. In the era of deep learn- ing, where popular models are notoriously opaque, it has thus proven vital to study the contents of datasets and identify aspects that may misrepre- sent model …

Content truncated to 500 words