Tokenization Repair in the Presence of Spelling Errors

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 279–289 November 10–11, 2021. ©2021 Association for Computational Linguistics 279 Tokenization Repair in the Presence of Spelling Errors Hannah Bast, Matthias Hertel, Mostafa M. Mohamed University of Freiburg Freiburg, Germany {bast,hertelm,amin}@cs.uni-freiburg.de Abstract We consider the following tokenization repair problem: Given a natural language text with any combination of missing or spurious spaces, correct these. Spelling errors can be present, but it’s not part of the problem to correct them. For example, given: “Tispa per isabout token izaionrep air”, compute “Tis paper is about to- kenizaion repair”. We identify three key ingredients of high- quality tokenization repair, all missing from previous work: deep language models with a bidirectional component, training the models on text with spelling errors, and making use of the space information already present. Our methods also improve existing spell checkers by fixing not only more tokenization errors but also more spelling errors: once it is clear which characters form a word, it is much eas- ier for them to figure out the correct word. We provide six benchmarks that cover three use cases (OCR errors, text extraction from PDF, human errors) and the cases of par- tially correct space information and all spaces missing. We evaluate our methods against the best existing methods and a non-trivial baseline. We provide full reproducibil- ity under https://ad.cs.uni-freiburg. de/publications . 1 Introduction Tokenizing a given text into words is the first step in many natural language processing applications, in- cluding: search engines, translation services, spell checkers and all kinds of learning tasks performed on text. This tokenization is typically performed by the following simple method or a variant of it: define a set of word characters and take each maxi- mal sequence of word characters as one token. For example, for This algoritm runs in linear time (1) a simple such tokenization yields the six words This, algoritm, runs, in, linear, time. Note the spelling error in the second word. Spelling correction is not part of tokenization. We come back to this important aspect in Section 1.2. Missing and spurious spaces are common errors in digital text documents. We refer to the union of both types of errors as tokenization errors. Here is a variant of the sentence above with one missing space and one spurious space: This algor itm runsin linear time (2) In this paper, we consider the following tokeniza- tion repair problem: Given a sequence of charac- ters representing a natural language text, with an arbitrary amount of missing and spurious spaces and possibly also with spelling errors, compute the variant of the text with correct spacing. For example, given (2) above, compute (1). Tokenization repair can be considered as a gener- alization of the word segmentation problem, where the text is given without any space information. In- deed, we also evaluate our methods on this special case in Section 5. 1.1 Sources of tokenization errors Tokenization errors are typical in texts that are digi- tized by …

Content truncated to 500 words