Weird Inflects but OK: Making Sense of Morphological Generation Errors

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 140–151 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 140 Weird Inflects but OK: Making Sense of Morphological Generation Errors Kyle Gorman∗, Arya D. McCarthy†, Ryan Cotterell†, Ekaterina Vylomova‡, Miikka Silfverberg§, and Magdalena Markowska∗ ∗The Graduate Center, City University of New York, †Johns Hopkins University, ‡University of Melbourne, §University of Helsinki kgorman@gc.cuny.edu, arya@jhu.edu, ryan.cotterell@jhu.edu, evylomova@gmail.com, miikka.silfverberg@helsinki.fi, mmarkowska@gradcenter.cuny.edu Abstract We conduct a manual error analysis of the CoNLL–SIGMORPHON 2017 Shared Task on Morphological Reinflection. In this task, systems are given a word in citation form (e.g., hug) and asked to produce the corresponding inflected form (e.g., the simple past hugged). This design lets us analyze errors much like we might analyze children’s production errors. We propose an error taxonomy and use it to annotate errors made by the top two systems across twelve languages. Many of the ob- served errors are related to inflectional patterns sensitive to inherent linguistic properties such as animacy or affect; many others are failures to predict truly unpredictable inflectional be- haviors. We also find nearly one quarter of the residual “errors” reflect errors in the gold data. 1 Introduction A huge amount of work in natural language pro- cessing treats words as indivisible units, but the vast majority of the world’s languages have rich word-internal structure. For instance, 80% of the languages analyzed in the World Atlas of Linguis- tic Structure (Dryer and Haspelmath 2013) inflect verbs for tense and 65% inflect nouns for case. Generating and processing complex words is thus crucial for multilingual speech and language tech- nologies. Recent work on large-scale, multilingual com- putational modeling of morphology (e.g., Durrett and DeNero 2013, Cotterell et al. 2016) targets supervised inflection generation. Such tasks re- quire variable-length outputs, so they are less constrained than earlier segmentation-based tasks (e.g., Kurimo et al. 2010), but appear to be tractable with existing neural network–based models. For example, in the CoNLL–SIGMORPHON 2017 Shared Task (sub-task 1 and the “high-data condi- tion”), the focus of this study, the best-ranked sys- tem generates novel inflectional forms with 90% accuracy or better for 46 out of the 52 target lan- guages, It achieves perfect accuracy for four lan- guages (Cotterell et al. 2017). In light of these apparent success, we examine the failure modes of existing models for morpho- logical generation. We first propose and motivate an error taxonomy for this task, inspired by sim- ilar proposals for other natural language genera- tion and processing technologies such as grammat- ical error correction (e.g., Rozovskaya and Roth 2016) and machine translation (e.g., Popović and Ney 2011, Fishel et al. 2012, Irvine et al. 2013). We then use this taxonomy to perform a manual er- ror analysis of the CoNLL–SIGMORPHON 2017 Shared Task. Such analyses can help to identify strengths and weaknesses of existing systems, sug- gest future improvements, and guide development of strong ensemble models, but are often neglected or treated as an afterthought. This annotation also allows us …

Content truncated to 500 words