Spelling Error Correction with Soft-Masked BERT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 882–890 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 882 Spelling Error Correction with Soft-Masked BERT Shaohua Zhang1, Haoran Huang1, Jicong Liu2 and Hang Li1 1ByteDance AI Lab 2School of Computer Science and Technology, Fudan University {zhangshaohua.cs,huanghaoran,lihang.lh}@bytedance.com jcliu15@fudan.edu.cn Abstract Spelling error correction is an important yet challenging task because a satisfactory solu- tion of it essentially needs human-level lan- guage understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of- the-art method for the task selects a character from a list of candidates for correction (includ- ing non-correction) at each position of the sen- tence on the basis of BERT, the language repre- sentation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, ap- parently due to the way of pre-training it us- ing mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the for- mer being connected to the latter with what we call soft-masking technique. Our method of using ‘Soft-Masked BERT’ is general, and it may be employed in other language detection- correction problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly bet- ter than the baselines including the one solely based on BERT. 1 Introduction Spelling error correction is an important task which aims to correct spelling errors in a text either at word-level or at character-level (Yu and Li, 2014; Yu et al., 2014; Zhang et al., 2015; Wang et al., 2018b; Hong et al., 2019; Wang et al., 2019). It is crucial for many natural language applications such as search (Martins and Silva, 2004; Gao et al., 2010), optical character recognition (OCR) (Afli et al., 2016; Wang et al., 2018b), and essay scor- ing (Burstein and Chodorow, 1999). In this pa- Table 1: Examples of Chinese spelling errors Wrong: 埃及有金子塔。Egypt has golden towers. Correct: 埃及有金字塔。Egypt has pyramids. Wrong: 他的求胜欲很强，为了越狱在挖洞。 He has a strong desire to win and is digging for prison breaks Correct: 他的求生欲很强，为了越狱在挖洞。 He has a strong desire to survive and is digging for prison breaks. per, we consider Chinese spelling error correction (CSC) at character-level. Spelling error correction is also a very challeng- ing task, because to completely solve the problem the system needs to have human-level language understanding ability. There are at least two chal- lenges here, as shown in Table 1. First, world knowledge is needed for spelling error correction. Character 字 in the first sentence is mistakenly writ- ten as 子, where 金子塔 means golden tower and 金字塔 means pyramid. Humans can correct the typo by referring to world knowledge. Second, sometimes inference is also required. In the sec- ond sentence, the 4-th character 生 is …

Content truncated to 500 words