Lemma Hunting: Automatic Spelling Normalization for German CMC Corpora

Lemma Hunting: Automatic Spelling Normalization for CMC Corpora Eckhard Bick University of Southern Denmark eckhard.bick@mail.dk Abstract This paper presents and evaluates a method for automatic orthographic normalization and the treatment of out-of-vocabulary words (OOV) in German social media data. The system uses a cascade of spellchecking operations including casing-, sound- and keyboard-based letter permutations, as well as letter context likelihoods, and combines partial and root spellchecking with compound analysis and heuristic inflection analysis in novel ways. The system also handles contractions, elisions and some tokenization errors. In addition, pattern-based recognition of foreign words and abbreviations is attempted, supported by jargon-informed lexicon expansion. Contextual Constraint Grammar (CG) disambiguation is used to resolve possible ambiguity. For Twitter data, F-scores of 87.3 and 77.1 were achieved for the identification and correct lemmatization, respectively, of German spelling errors and non-standard abbreviations. 77.6% of foreign words were recognized with 86.5% precision and 1/3 POS errors. 1 Introduction Computer-mediated communication (CMC) is a notoriously difficult genre to annotate, an important issue being non-standard orthography and unusual word formation. For Social Media, in particular, Proisl (2018) and Beißwenger (2016) mention a host of problems such as out- of-vocabulary words (OOV), emoticons/emojis, interaction words (lach [laugh], heul [cry]), URL's and discourse links (hashtags and user id's), onomatopoeia, spelling variation and contractions, emphasis by upper-casing or letter repetition, as well as syntactic idiosyncrasies. In a corpus annotation scenario, all of these may lead to reduced lexicon coverage, affecting tagging performance. Thus, Neunerdt (2013) reports a drop in accuracy from 95.8% to 68% for OOV words, a problem he successfully tackled by adding a specialized web lexicon. But even with word additions and a correct (heuristic) POS assignment, a failure to group spelling variations, abbreviations and spelling errors under the same lemma negatively affects the possibility of corpus searches and statistics. In this paper, following Sidarenka et al. (2013), we suggest an automatic, spellchecking-like normalization process to address the problem, providing a common lemma for spelling variants and outright errors at the same time. For a language like German, compound analysis may also increase the search-accessibility of a corpus, and prevent false positive spelling corrections. The work presented here was performed on a large German Twitter and Facebook corpus compiled for the XPEROHS hate speech project (Baumgarten et al. 2019) and annotated with a multi-level Constraint Grammar (CG) parser (GerGram1). All examples in the paper are authentic exerpts from this corpus. 2 Systematic normalization A relatively straightforward first step of 1 https://visl.sdu.dk/de/parsing/automatic/ 16 Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022) Distributed under Creative Commons License Attribution 4.0 International (CC BY 4.0) normalization concerns systematic variation, especially re-casing of lower-cased German nouns and of words written in all-uppercase for emphasis, both very common in our corpus. However, ignoring upper/lower case may lead to ambiguity between two German words or a foreign and a German word. This needs to be resolved contextually and is a possible source of errors. Another case of systematic variation is gendering, which in …

Content truncated to 500 words