Frustratingly Easy System Combination for Grammatical Error Correction

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1964 - 1974 July 10-15, 2022 ©2022 Association for Computational Linguistics Frustratingly Easy System Combination for Grammatical Error Correction Muhammad Reza Qorib∗, Seung-Hoon Na† and Hwee Tou Ng∗ ∗Department of Computer Science, National University of Singapore † Division of Computer Science and Engineering, Jeonbuk National University {mrqorib, nght}@comp.nus.edu.sg, nash@jbnu.ac.kr Abstract In this paper, we formulate system combina- tion for grammatical error correction (GEC) as a simple machine learning task: binary clas- sification. We demonstrate that with the right problem formulation, a simple logistic regres- sion algorithm can be highly effective for com- bining GEC models. Our method successfully increases the F0.5 score from the highest base GEC system by 4.2 points on the CoNLL-2014 test set and 7.2 points on the BEA-2019 test set. Furthermore, our method outperforms the state of the art by 4.0 points on the BEA-2019 test set, 1.2 points on the CoNLL-2014 test set with original annotation, and 3.4 points on the CoNLL-2014 test set with alternative annota- tion. We also show that our system combina- tion generates better corrections with higher F0.5 scores than the conventional ensemble.1 1 Introduction Grammatical error correction (GEC) is the task of detecting and correcting grammatical errors present in a text (Ng et al., 2013). Grammatical error cor- rection has achieved remarkable progress since the late 2000s, and has flourished more along with the development of sequence-to-sequence architecture. Grammatical error correction shared tasks such as CoNLL-2013 (Ng et al., 2013), CoNLL-2014 (Ng et al., 2014), and BEA-2019 (Bryant et al., 2019) also contribute to popularizing the task. With the success of Transformer (Vaswani et al., 2017) architecture in sequence-to-sequence tasks, most recent state-of-the-art grammatical error cor- rection systems use Transformer-based architecture. Even though they all use a Transformer-based archi- tecture, there are still some variations to the models, especially in the task formulation and pre-training data. Generally, we can divide the recent state-of- the-art systems into a sequence tagging approach 1Source code of this paper is publicly available at https: //github.com/nusnlp/esc. (Awasthi et al., 2019; Omelianchuk et al., 2020) that usually uses a large pre-trained masked lan- guage model, and a sequence-to-sequence ap- proach (Rothe et al., 2021; Stahlberg and Kumar, 2021; Kaneko et al., 2020) that usually pre-trains a Transformer architecture with synthetic data. The differences in the synthetic data generation meth- ods and seed corpora used also contribute to more diverse GEC systems. Figure 1: The F0.5 scores of the base GEC systems that we use in our experiments on selected error types in the BEA-2019 development set. With these differences, each model has its own strengths and weaknesses (Figure 1). Susanto et al. (2014) has demonstrated that the differences in the strengths of the GEC models can be utilized to gen- erate better grammatical error corrections by com- bining them through a system combination method. In this paper, we present our simple yet effective system combination method …

Content truncated to 500 words