Out-of-Domain Evaluation of Finnish Dependency Parsing

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1114–1124 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1114 Out-of-Domain Evaluation of Finnish Dependency Parsing Jenna Kanerva and Filip Ginter TurkuNLP, Department of Computing University of Turku, Finland {jmnybl, figint}@utu.fi Abstract The prevailing practice in the academia is to evaluate the model performance on in-domain evaluation data typically set aside from the training corpus. However, in many real world applications the data on which the model is applied may very substantially differ from the characteristics of the training data. In this paper, we focus on Finnish out-of-domain parsing by introducing a novel UD Finnish-OOD out-of-domain treebank including five very distinct data sources (web documents, clinical, online discussions, tweets, and poetry), and a total of 19,382 syntactic words in 2,122 sentences released under the Universal Dependencies framework. Together with the new treebank, we present extensive out-of-domain parsing evaluation utilizing the available section-level information from three different Finnish UD treebanks (TDT, PUD, OOD). Compared to the previously existing treebanks, the new Finnish-OOD is shown include sections more challenging for the general parser, creating an interesting evaluation setting and yielding valuable information for those applying the parser outside of its training domain. Keywords: Finnish, Universal Dependencies, Treebank, Parsing, Out-of-domain 1. Introduction During software development and model performance evaluation, the prevailing practice in the academia is to evaluate the model performance on an in-domain dataset. This typically means that the model is evalu- ated on a test section set aside from the training corpus, therefore the test dataset sharing the same properties as the data used to train the model. However, in many real world applications this may not be the case. The data on which the model is applied in its actual use in down- stream applications may in practice very substantially differ from the characteristics of the training data. In this paper, we focus on Finnish dependency parsing in the Universal Dependencies (UD) scheme. When both trained and tested on the UD dataset, the state of the art is approaching human performance (Virtanen et al., 2019). Consequently, the Finnish parser is in active use in the academia as well as in the commercial indus- try, and applied in numerous downstream tasks and do- mains as a text normalization and pre-processing com- ponent. In some cases, however, these application do- mains substantially differ in their characteristics from the training corpus and there is no hard evidence as to the effect on the parser performance. To address this question, we selected and annotated a new treebank meant solely for out-of-domain evalu- ation of the models trained on the UD Finnish-TDT dataset. This new UD Finnish-OOD treebank allows us to quantify the parsing performance in various down- stream applications, and to better understand the limits of generalization exhibited by the most recent depen- dency parsing methodology. In addition to introduc- ing the new dataset, we carry out several Finnish out- of-domain parsing experiments, where in addition …

Content truncated to 500 words