Rethinking Masked Language Modeling for Chinese Spelling Correction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10743–10756 July 9-14, 2023 ©2023 Association for Computational Linguistics Rethinking Masked Language Modeling for Chinese Spelling Correction Hongqiu Wu1,2,∗ and Shaohua Zhang3 and Yuchen Zhang3 and Hai Zhao1,2,† 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University 3ByteDance wuhongqiu@sjtu.edu.cn,zhang.shaohua.cs@gmail.com zhangyuc@gmail.com,zhaohai@cs.sjtu.edu.cn Abstract In this paper, we study Chinese Spelling Cor- rection (CSC) as a joint decision made by two separate models: a language model and an er- ror model. Through empirical analysis, we find that fine-tuning BERT tends to over-fit the error model while under-fit the language model, resulting in poor generalization to out- of-distribution error patterns. Given that BERT is the backbone of most CSC models, this phe- nomenon has a significant negative impact. To address this issue, we are releasing a multi- domain benchmark LEMON, with higher qual- ity and diversity than existing benchmarks, to allow a comprehensive assessment of the open domain generalization of CSC models. Then, we demonstrate that a very simple strategy – randomly masking 20% non-error tokens from the input sequence during fine-tuning – is suffi- cient for learning a much better language model without sacrificing the error model. This tech- nique can be applied to any model architec- ture and achieves new state-of-the-art results on SIGHAN, ECSpell, and LEMON1. 1 Introduction Chinese Spelling Correction (CSC) is a crucial task in natural language processing (NLP) behind many downstream applications, e.g, web search (Mar- tins and Silva, 2004; Gao et al., 2010), named en- tity recognition, optical character recognition (Afli et al., 2016; Gupta et al., 2021). It aims to detect and correct the potential spelling errors in a sen- tence. BERT (Devlin et al., 2019) and its enhanced variants have achieved state-of-the-art results in the current CSC community (name a few) (Zhang et al., 2020; Liu et al., 2021; Zhu et al., 2022). From a high-level perspective, CSC requires a language model and an error model working ∗Work was done during a cooperation with ByteDance. †Corresponding author; This paper was partially sup- ported by Key Projects of National Natural Science Foun- dation of China (U1836222 and 61733011). 1https://github.com/gingasan/lemon 我买的鸟会发出公鸽子的(生硬 -> 声音)。 第一个是跟(生音 -> 声音)有问题。 新的机器声影少一点。 The bird I bought would make the (stiff -> sound)... Training Stage Predict The first has a problem with the (raw -> sound). The new machine has less shadow. 新的机器声影少一点。 The new machine has less shadow. (声影 -> 声音) (shadow -> sound) 我买的鸟声音很生硬。 The bird I bought sounds stiff. 我买的鸟声音很(生硬 -> 声音)。 The bird I bought sounds (stiff -> sound). Over correction No detection model output ground truth test input train pair Testing Stage No correction Figure 1: Mistakes made by regularly fine-tuned BERT. collaboratively to make a decision (Kernighan et al., 1990). Suppose that the input sentence con- tains n characters X = (x1, ..., xn). The model predicts the corrected character at each position …

Content truncated to 500 words