MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3118 - 3130 July 10-15, 2022 ©2022 Association for Computational Linguistics MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction Yue Zhang1†, Zhenghua Li1∗, Zuyi Bao2, Jiacheng Li1, Bo Zhang2, Chen Li2, Fei Huang2, Min Zhang1 1Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China; 2DAMO Academy, Alibaba Group, China 1{yzhang21,jcli20}@stu.suda.edu.cn, 1{zhli13,minzhang}@suda.edu.cn 2{zuyi.bzy,klayzhang.zb,puji.lc,f.huang}@alibaba-inc.com Abstract This paper presents MuCGEC, a multi- reference multi-source evaluation dataset for Chinese Grammatical Error Correc- tion (CGEC), consisting of 7,063 sentences collected from three Chinese-as-a-Second- Language (CSL) learner sources. Each sen- tence is corrected by three annotators, and their corrections are carefully reviewed by a senior annotator, resulting in 2.3 references per sentence. We conduct experiments with two mainstream CGEC models, i.e., the sequence- to-sequence model and the sequence-to-edit model, both enhanced with large pretrained language models, achieving competitive bench- mark performance on previous and our datasets. We also discuss CGEC evaluation methodolo- gies, including the effect of multiple refer- ences and using a char-based metric. Our an- notation guidelines, data, and code are avail- able at https://github.com/HillZha ng1999/MuCGEC. 1 Introduction Given a potentially noisy input sentence, gram- matical error correction (GEC) aims to detect and correct all errors and produce a clean sentence. Recently, GEC has increasingly gained attention for its vital value in various downstream scenarios (Grundkiewicz et al., 2020; Wang et al., 2021). To support GEC research, high-quality manu- ally labeled evaluation data is indispensable. For English GEC (EGEC), such datasets are abun- dant (Yannakoudakis et al., 2011; Dahlmeier et al., 2013; Ng et al., 2014; Napoles et al., 2017; Bryant et al., 2019; Napoles et al., 2019; Flachs et al., 2020). However, Chinese GEC (CGEC) evaluation datasets are relatively scarce. The two publicly available CGEC evaluation datasets are NLPCC18 and CGED, contributed by the NLPCC-2018 (Zhao † This work was partially done during the first author’s internship at Alibaba DAMO Academy. ∗ Corresponding author. Source 我不知道他何时返回回来。 I don’t know when he will return back. Ref. 1 我不知道他何时返回回来。 I don’t know when he will return. Ref. 2 我不知道他何时返回回来。 I don’t know when he will be back. Table 1: A CGEC example with two references. et al., 2018) and the series of CGED shared tasks (Rao et al., 2018, 2020), respectively. Most EGEC evaluation datasets provide mul- tiple references for each input sentence, such as CoNLL14-test (Ng et al., 2014) and BEA19-test (Bryant et al., 2019). In contrast, sentences in existing CGEC evaluation datasets usually have only one reference (i.e., 87% of the sentences in NLPCC18 and all in CGED). This is probably due to the different annotation workflows adopted. As suggested by Bryant and Ng (2015), enforc- ing multi-reference annotation is crucial for both GEC model evaluation and GEC data annotation, because there usually exist more than one accept- able reference with similar meanings for an incor- rect sentence, as illustrated by the example in Table 1. …

Content truncated to 500 words