Searching for Search Errors in Neural Morphological Inflection

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1388–1394 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1388 Searching for Search Errors in Neural Morphological Inflection Martina Forster1 Clara Meister1 Ryan Cotterell1,2 1ETH Z¨urich 2University of Cambridge martfors@ethz.ch {clara.meister, ryan.cotterell}@inf.ethz.ch Abstract Neural sequence-to-sequence models are currently the predominant choice for language generation tasks. Yet, on word-level tasks, exact inference of these models reveals the empty string is often the global optimum. Prior works have speculated this phenomenon is a result of the inadequacy of neural models for language generation. However, in the case of morphological inflection, we find that the empty string is almost never the most probable solution under the model. Further, greedy search often finds the global optimum. These observations suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation. 1 Introduction Neural sequence-to-sequence models are om- nipresent in the field of natural language processing due to their impressive performance. They hold state of the art on a myriad of tasks, e.g., neural machine translation (NMT; Ott et al., 2018b) and abstractive summarization (AS; Lewis et al., 2019). Yet, an undesirable property of these models has been repeatedly observed in word-level tasks: When using beam search as the decoding strategy, increasing the beam width beyond a size of k = 5 often leads to a drop in the quality of solutions (Murray and Chiang, 2018; Yang et al., 2018; Cohen and Beck, 2019). Further, in the context of NMT, it has been shown that the empty string is fre- quently the most-probable solution under the model (Stahlberg and Byrne, 2019). Some suggest this is a manifestation of the general inadequacy of neural models for language generation tasks (Koehn and Knowles, 2017; Kumar and Sarawagi, 2019; Holtzman et al., 2020; Stahlberg, 2020); in this work, we find evidence demonstrating otherwise. k = 1 k = 10 k = 100 k = 500 NMT 63.1% 46.1% 44.3% 6.4% MI 0.8% 0.0% 0.0% 0.0% Table 1: Percentage of search errors—which we define as instances where the search strategy does not find the global optimum under the model—for Transform- ers trained on IWSLT’14 De-En (NMT) and SIGMOR- PHON 2020 (Morphological Inflection; MI) when de- coding with beam search for varying beam widths (k). MI results are averaged across languages. Sequence-to-sequence transducers for character- level tasks often follow the architectures of their word-level counterparts (Faruqui et al., 2016; Lee et al., 2017), and have likewise achieved state-of-the-art performance on e.g., morpholog- ical inflection generation (Wu et al., 2020) and grapheme-to-phoneme conversion (Yolchuyeva et al., 2019). Given prior findings, we might expect to see the same degenerate behavior in these models—however, we do not. We run a series of experiments on morphological inflection (MI) generators to explore whether neural transducers for this task are similarly poorly calibrated, i.e. are far from the true distribution p(y …

Content truncated to 500 words