Do Grammatical Error Correction Models Realize Grammatical Generalization?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4554–4561 August 1–6, 2021. ©2021 Association for Computational Linguistics 4554 Do Grammatical Error Correction Models Realize Grammatical Generalization? Masato Mita1, 2 and Hitomi Yanaka3, 1 1RIKEN AIP, 2Tohoku University, 3The University of Tokyo masato.mita@riken.jp | hyanaka@is.s.u-tokyo.ac.jp Abstract There has been an increased interest in data generation approaches to grammatical error correction (GEC) using pseudo data. How- ever, these approaches suffer from several is- sues that make them inconvenient for real- world deployment including a demand for large amounts of training data. On the other hand, some errors based on grammatical rules may not necessarily require a large amount of data if GEC models can realize grammatical generalization. This study explores to what extent GEC models generalize grammatical knowledge required for correcting errors. We introduce an analysis method using synthetic and real GEC datasets with controlled vocabu- laries to evaluate whether models can general- ize to unseen errors. We found that a current standard Transformer-based GEC model fails to realize grammatical generalization even in simple settings with limited vocabulary and syntax, suggesting that it lacks the generaliza- tion ability required to correct errors from pro- vided training examples. 1 Introduction Grammatical Error Correction (GEC) is the task of automatically correcting grammatical errors in a text. GEC’s mainstream approach is to con- sider the task as machine translation (MT) from an ungrammatical text to a grammatical text due to their structural similarity (Brockett et al., 2006; Junczys-Dowmunt et al., 2018). Therefore, many neural encoder-decoder models (EncDec), which are common in MT, have been proposed for GEC, and Transformer-based models have become stan- dard (Grundkiewicz and Junczys-Dowmunt, 2018; Zhao et al., 2019; Kaneko et al., 2020). More recently, there has been an increased interest in data generation approaches to GEC using pseudo data, i.e., improving performance by increasing the Train: Test 1: (Known Setting) Every polite cow *smile / smiles awkwardly Test 2: (Unknown Setting) Every white fox *run / runs quickly Every dog *run / runs quickly That slimy duck smiles / smiles awkwardly Some slimy cows smile / smile dramatically Figure 1: Overview of our proposed method for evalu- ating the generalization capability of GEC models. In the Known setting, the model must correct previously seen patterns. In the Unknown setting, the model is presented with an unseen pattern but with familiar vo- cabulary. We found significantly lower performance in the unknown setting, indicating that the model failed to generalize its grammatical knowledge. amount of training data using pseudo data with- out making any modifications to the model archi- tecture (Grundkiewicz et al., 2019; Kiyono et al., 2019). However, these approaches suffer from several issues that make them inconvenient for real-world deployment, including a demand for large amounts of training data. For example, Kiyono et al. (2019) reported that it was necessary to add about 60 mil- lion samples of pseudo-data to improve a standard measure of GEC, F0.5, by only two points. If GEC models can realize grammatical generalization, as …

Content truncated to 500 words