On the Robustness of Cognate Generation Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4299–4305 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4299 On the Robustness of Cognate Generation Models Winston Wu David Yarowsky Center for Language and Speech Processing Department of Computer Science Johns Hopkins University {wswu,yarowsky}@jhu.edu Abstract We evaluate two popular neural cognate generation models’ robustness to several types of human-plausible noise: deletion, duplication, swapping, and keyboard errors, as well as a new type of error, phonological errors. We find that duplication and phonological substitutions are the least harmful due to cognate training pairs acting as naturally occurring noisy training data, while other types of noise are more harmful. We present an in-depth analysis of the models’ performance with respect to each error type to explain the models’ predictions. Keywords: cognates, robustness, seq2seq 1. Introduction Consider the following scenario: a speaker of a low- resource language, say Ilocano, is searching the web for some information. Because there are relatively few Ilocano web pages, the search engine may perform query expansion by translating the query into a higher- resource language like Tagalog. In many cases, this higher-resource language is a language of wider com- munication that the user may be familiar with (though may not be fluent in), or is related enough that the user can get the gist of results in that language. Such a search engine should be robust to the user’s query; humans are fallible and often make mistakes. In this work, we examine different types of noise generated by human errors and how these noisy inputs affect the performance of cognate generation models, which can be used to translate the user’s query. We investigate typical errors such as typing errors as well as cognitive errors, in which a user not fluent in the language may make spelling errors such as replacing t with d. These errors include character deletions, duplications, swaps, keyboard errors, and phonologically similar character substitutions (Table 1). Cognate generation models (Beinborn et al., 2013; Wu and Yarowsky, 2018b; Lewis et al., 2020) trans- late a single word (e.g. English tomato) into a cog- nate (e.g. Finnish tomaatti), a related word that has a common etymological parent (Nahuatl tomatl). These models are typically based on neural character- based sequence-to-sequence architectures and have also been increasingly used in a variety of related appli- cations, including language reconstruction (Meloni et al., 2021), modeling etymological and morphological forms (Wu and Yarowsky, 2020a; Vylomova et al., 2020; Wu et al., 2021b), phonology (Wu and Yarowsky, 2021), grapheme-to-phoneme (Gorman et al., 2020), and named entity transliteration (Merhav and Ash, 2018; Wu et al., 2018; Wu and Yarowsky, 2018a). As these neural models become more prevalent in NLP, it is im- Error Input Original hispana Deletion hispan�a Duplication hiispana Swap hisapna Keyboard hispanz Phonological hispanа Table 1: Different types of character perturbations, with the misspelled character in red. Notice in the phonolog- ical error, the last а is actually a Cyrillic character. perative that …

Content truncated to 500 words