Error Analysis and the Role of Morphology

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1887–1900 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1887 Error Analysis and the Role of Morphology Marcel Bollmann Department of Computer Science University of Copenhagen Denmark marcel@di.ku.dk Anders Søgaard Department of Computer Science University of Copenhagen Denmark soegaard@di.ku.dk Abstract We evaluate two common conjectures in er- ror analysis of NLP models: (i) Morphology is predictive of errors; and (ii) the importance of morphology increases with the morphologi- cal complexity of a language. We show across four different tasks and up to 57 languages that of these conjectures, somewhat surpris- ingly, only (i) is true. Using morphological features does improve error prediction across tasks; however, this effect is less pronounced with morphologically complex languages. We speculate this is because morphology is more discriminative in morphologically simple lan- guages. Across all four tasks, case and gender are the morphological features most predictive of error. 1 Introduction In error analysis, we often blame morphology (Nivre, 2007; Bender, 2009), i.e., the productive inflection and derivation of new word forms. Mor- phology has been argued to be a major source of error in syntactic parsing (Tsarfaty et al., 2020), semantic parsing (¸Sahin and Steedman, 2018), ma- chine translation (Irvine et al., 2013; Burlot and Yvon, 2017) and a range of other tasks, in particu- lar in morphologically complex languages (Bender, 2009; Søgaard et al., 2018; Tsarfaty et al., 2020). This paper presents a large-scale study showing that morphology is, as commonly conjectured, an im- portant source of error across tasks, but somewhat surprisingly, that morphology is less predictive of errors in morphologically complex languages. English is a morphologically simple language, showing very limited inflection and expressing most concepts through syntactic structure instead; it is also the most-represented language at ma- jor natural language processing (NLP) venues and that with the largest amount of language resources available (Bender, 2011; Joshi et al., 2020). This w jego imieniu   POS=NOUN CASE=LOC GENDER=NEUT NUMBER=SING ... Random forest classifier Predict Figure 1: Overview of our methodology: We map each token to a set of morphological features and, based on this representation, predict whether some NLP system (e.g., a dependency parser) was correct () or made an error () on that token. makes it easy to ignore morphology when design- ing model architectures. As a consequence, we frequently observe that performance of NLP sys- tems on morphologically more complex languages lags behind that for English (e.g. Czarnowska et al., 2019; Tsarfaty et al., 2020). Complex morphology leads to the occurrence of rare inflected word forms. Polish nouns, for example, can inflect for number and seven dif- ferent cases; this makes it less likely that all of these inflected word forms appear in the train- ing data for our NLP models. Consequently, a model that correctly handles imi˛e ‘name’ (NOM.SG) might not have seen the less frequent form imion- ami (INST.PL), potentially resulting in errors. If the model has generally seen …

Content truncated to 500 words