Deciphering and Characterizing Out-of-Vocabulary Words for Morphologically Rich Languages

Proceedings of the 29th International Conference on Computational Linguistics, pages 5309–5326 October 12–17, 2022. 5309 Deciphering and Characterizing Out-of-Vocabulary Words for Morphologically Rich Languages Georgie Botev, Arya D. McCarthy, Winston Wu, and David Yarowsky Johns Hopkins University Abstract This paper presents a detailed foundational empirical case study of the nature of out-of- vocabulary words encountered in modern text in a moderate-resource language such as Bulgarian, and a multi-faceted distributional analysis of the underlying word-formation processes that can aid in their compositional translation, tag- ging, parsing, language modeling, and other NLP tasks. Given that out-of-vocabulary (OOV) words generally present a key open challenge to NLP and machine translation systems, especially toward the lower limit of resource availability, there are useful practical insights, as well as corpus-linguistic insights, from both a detailed manual and automatic taxonomic analysis of the types, multidimensional properties, and process- ing potential for multiple representative OOV data samples. 1 Introduction Even in a familiar language, unfamiliar words cause trouble for machine processing or comprehension of text. Any dictionary is innately incomplete in its coverage, unable to provide novel coinages and exhaustive forms. Without finding the word in a dictionary, the surface form and context afford only weak evidence for its meaning. The situation is even worse for languages other than English, especially morphologically rich languages, for two reasons: first, there is usually less annotated data available; and second, the coverage of such data is much lower due to the high number of different forms. Moreover, many words not found in even a small training corpus are in fact related to quite common words by processes such as inflection, derivation, compounding, or misspelling. In the work described herein, we therefore con- centrate on the problem of characterizing unknown words in terms of the processes by which they arise, and especially the relative frequencies at which such processes occur. This informs us of the distribution 0 5 10 15 20 25 Number ProductName WebAddress Transliteration Compound Conjugation Typo Target Foreign Name Geography Declension Figure 1: Taxonomized distribution of out-of-vocabulary types in Bulgarian Wikipedia, random sample of 100 types of out-of-vocabulary (OOV) words with respect to different dictionary sources. To do so, we conduct a study on a sample of two Bulgarian language corpora annotated by a na- tive speaker. Rather than treat OOV tokens as a monolithic and undifferentiated problem, we pro- gressively apply multi-faceted linguistic analyses to these corpora, characterizing both the words that these analyses explain and words yet to be explained, which we shall call the residual vocabulary. Our methods are a mixture of the vintage and the vogue: specialized edit distances, composition of finite-state transducers, a noisy channel model for language identification fitted with empirical Bayes, and neural network–based part of speech taggers. Collectively, our processes accurately explain more than two in three (69%) unknown Bulgarian words in a held-out set according to whether they are proper names, in- flections, derivations, compounds, foreign words, or misspellings (as illustrated in both Figures 1 and 3, discussed in …

Content truncated to 500 words