Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4973–4984 August 1–6, 2021. ©2021 Association for Computational Linguistics 4973 Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction Piji Li Shuming Shi Tencent AI Lab, Shenzhen, China {pijili,shumingshi}@tencent.com Abstract We investigate the problem of Chinese Gram- matical Error Correction (CGEC) and present a new framework named Tail-to-Tail (TtT) non-autoregressive sequence prediction to ad- dress the deep issues hidden in CGEC. Con- sidering that most tokens are correct and can be conveyed directly from source to tar- get, and the error positions can be estimated and corrected based on the bidirectional con- text information, thus we employ a BERT- initialized Transformer Encoder as the back- bone model to conduct information modeling and conveying. Considering that only relying on the same position substitution cannot han- dle the variable-length correction cases, vari- ous operations such substitution, deletion, in- sertion, and local paraphrasing are required jointly. Therefore, a Conditional Random Fields (CRF) layer is stacked on the up tail to conduct non-autoregressive sequence pre- diction by modeling the token dependencies. Since most tokens are correct and easily to be predicted/conveyed to the target, then the models may suffer from a severe class imbal- ance issue. To alleviate this problem, focal loss penalty strategies are integrated into the loss functions. Moreover, besides the typical fix-length error correction datasets, we also construct a variable-length corpus to conduct experiments. Experimental results on stan- dard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of error De- tection and Correction1. 1 Introduction Grammatical Error Correction (GEC) aims to au- tomatically detect and correct the grammatical er- rors that can be found in a sentence (Wang et al., 2020c). It is a crucial and essential application task 1Code: https://github.com/lipiji/TtT ����������I feel fly long happy today! Correct ���������� ����������I feel very happy today! I am always happy when I come to Fei today! Type I Type II Type III ����������I very feel happy today! � � � Figure 1: Illustration for the three types of operations to correct the grammatical errors: Type I-substitution; Type II-deletion and insertion; Type III-local paraphras- ing. in many natural language processing scenarios such as writing assistant (Ghufron and Rosyida, 2018; Napoles et al., 2017; Omelianchuk et al., 2020), search engine (Martins and Silva, 2004; Gao et al., 2010; Duan and Hsu, 2011), speech recognition systems (Karat et al., 1999; Wang et al., 2020a; Kubis et al., 2020), etc. Grammatical errors may appear in all languages (Dale et al., 2012; Xing et al., 2013; Ng et al., 2014; Rozovskaya et al., 2015; Bryant et al., 2019), in this paper, we only fo- cus to tackle the problem of Chinese Grammatical Error Correction (CGEC) (Chang, 1995). We investigate the problem of CGEC and the related corpora from SIGHAN (Tseng et al., 2015) and NLPCC (Zhao et al., 2018) carefully, and …

Content truncated to 500 words