Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10367 - 10380 December 7-11, 2022 ©2022 Association for Computational Linguistics Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction Kai Shen1∗, Yichong Leng2∗, Xu Tan3† Siliang Tang1†, Yuan Zhang4, Wenjie Liu4, Edward Lin4 1Zhejiang University, 2University of Science and Technology of China 3Microsoft Research Asia, 4Microsoft Azure Speech 1{shenkai, siliang}@zju.edu.cn, 2lyc123go@mail.ustc.edu.cn 3,4{xuta, yuanz, liwenjie, edlin}@microsoft.com Abstract Text error correction aims to correct the errors in text sequences such as those typed by hu- mans or generated by speech recognition mod- els. Previous error correction methods usually take the source (incorrect) sentence as encoder input and generate the target (correct) sentence through the decoder. Since the error rate of the incorrect sentence is usually low (e.g., 10%), the correction model can only learn to correct on limited error tokens but trivially copy on most tokens (correct tokens), which harms the effective training of error correction. In this paper, we argue that the correct tokens should be better utilized to facilitate effective training and then propose a simple yet effective mask- ing strategy to achieve this goal. Specifically, we randomly mask out a part of the correct to- kens in the source sentence and let the model learn to not only correct the original error to- kens but also predict the masked tokens based on their context information. Our method en- joys several advantages: 1) it alleviates trivial copy; 2) it leverages effective training signals from correct tokens; 3) it is a plug-and-play module and can be applied to different models and tasks. Experiments on spelling error correc- tion and speech recognition error correction on Mandarin datasets and grammar error correc- tion on English datasets with both autoregres- sive and non-autoregressive generation models show that our method improves the correction accuracy consistently1. 1 Introduction Text error correction techniques have been widely used to correct the errors in text, such as gram- mar errors (Rothe et al., 2021; Grundkiewicz et al., ∗ Both authors contributed equally. This work was con- ducted when the first and second authors were interns in Mi- crosoft. †∗ Corresponding author 1The code for our model is provided for research purposes: https://github.com/microsoft/NeuralSpeech. 2019; Omelianchuk et al., 2020) or spelling er- rors (Cheng et al., 2020; Zhang et al., 2020) and automatic speech recognition (ASR) errors (Cucu et al., 2013; D’Haro and Banchs, 2016; Anantaram et al., 2018; Zhu et al., 2021), etc. Text error cor- rection is a challenging task because in realistic ASR transcripts or human-typed documents, it is common that the errors are sparse (for example, the error rate of an ASR system is usually below 10%) (Leng et al., 2021b). To handle the problem, previous works aim to improve the accuracy of cor- rection models by using error detector-corrector architecture and focusing on two directions: 1) promoting the accuracy of error detection (Zhang et al., 2020), 2) enhancing the capability of cor- rection (Cheng et al., 2020; Leng et al., …

Content truncated to 500 words