Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7299–7316 July 9-14, 2023 ©2023 Association for Computational Linguistics Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora Svanhvít Lilja Ingólfsdóttir1 Pétur Orri Ragnarsson1 Haukur Páll Jónsson1 Haukur Barri Símonarson1 Vilhjálmur Þorsteinsson1 Vésteinn Snæbjarnarson1,2 1Miðeind ehf. 2University of Copenhagen {svanhvit, petur, haukurpj, haukur, vt, vesteinn}@mideind.is Abstract Grammatical error correction (GEC) is the task of correcting typos, spelling, punctuation and grammatical issues in text. Approaching the problem as a sequence-to-sequence task, we compare the use of a common subword unit vocabulary and byte-level encoding. Ini- tial synthetic training data is created using an error-generating pipeline, and used for finetun- ing two subword-level models and one byte- level model. Models are then finetuned fur- ther on hand-corrected error corpora, including texts written by children, university students, dyslexic and second-language writers, and eval- uated over different error types and origins. We show that a byte-level model enables higher correction quality than a subword approach, not only for simple spelling errors, but also for more complex semantic, stylistic and gram- matical issues. In particular, initial training on synthetic corpora followed by finetuning on a relatively small parallel corpus of real-world errors helps the byte-level model correct a wide range of commonly occurring errors. Our ex- periments are run for the Icelandic language but should hold for other similar languages, particularly morphologically rich ones. https://github.com/mideind/ byte-gec 1 Introduction Spelling mistakes due to typos and rushed writing, nonstandard punctuation and spelling, and gram- matical and stylistic issues are common to almost everyone who writes any kind of text. This ap- plies in any language and can distract the reader or make the communication miss its mark. This can hinder people who have difficulties writing text conforming to a particular language standard, be it due to disability, dyslexia, linguistic background, limited access to education or any other reason. Prejudice against people whose writing deviates from the standard can make some shy away from communicating with others, leaving their voices out of important discussions and restricting their opportunities (Alexander-Passe, 2015). Grammatical error correction (GEC) is the task of adjusting a text’s spelling, grammar, and lin- guistic style to conform to an approved language standard or convention (Rauf et al., 2017). While the latest work on GEC is based on Transformer models (Vaswani et al., 2017), the subword tok- enization methods commonly used in these models are a source of problems when it comes to typos and other variants (Schmaltz et al., 2017). Subword tokenization (Sennrich et al., 2016; Kudo, 2018) was presented as a solution to the open vocabulary problem, as it is a compromise between charac- ter encoding and whole word encoding, enabling unknown words to be represented using known subwords. However, a significant downside of subword to- kenization is how it is affected by noisy input; if a word contains a typo or other spelling variants, this can completely shift its representation. In ad- dition, in languages with rich …

Content truncated to 500 words