Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors

Findings of the Association for Computational Linguistics: ACL 2022, pages 4107 - 4118 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors Ryo Nagata Konan University / Japan nagata-acl2022@ ml.hyogo-u.ac.jp. Manabu Kimura GRAS Group, Inc. / Japan manabu.kimura@gras-group.co.jp Kazuaki Hanawa RIKEN / Japan Tohoku University / Japan kazuaki.hanawa@riken.jp Abstract In this paper, we explore the capacity of a language model-based method for grammat- ical error detection in detail. We first show that 5 to 10% of training data are enough for a BERT-based error detection method to achieve performance equivalent to what a non- language model-based method can achieve with the full training data; recall improves much faster with respect to training data size in the BERT-based method than in the non-language model method. This suggests that (i) the BERT- based method should have a good knowledge of the grammar required to recognize certain types of error and that (ii) it can transform the knowl- edge into error detection rules by fine-tuning with few training samples, which explains its high generalization ability in grammatical error detection. We further show with pseudo error data that it actually exhibits such nice proper- ties in learning rules for recognizing various types of error. Finally, based on these findings, we discuss a cost-effective method for detecting grammatical errors with feedback comments explaining relevant grammatical rules to learn- ers. 1 Introduction Recent studies have shown that masked language models pre-trained on a large corpus (hereafter, simply language models) achieve tremendous im- provements over a wide variety of natural language processing tasks with fine-tuning. These results suggest that they are also effective in recognizing erroneous words and phrases, the task known as grammatical error detection. There has been, how- ever, much less work on this aspect of grammatical error detection than in other tasks. One can argue that since language models are trained on language data produced by native speakers of a language (specifically, English in this paper), they might not work well on the target language data produced by non-native speakers of that language. In other words, English language models do not know at all about grammatical errors made by non-native speakers. Even apart from grammatical errors, the target language is different from the canon- ical English, meaning that it contains unnatural phrases and characteristic language usages that na- tive speakers do not normally use, as Nagata and Whittaker (2013) demonstrate. These differences might affect performance of language model-based methods in grammatical error detection. Actually, researchers have reported on perfor- mance of language models on grammatical error detection and correction. Cheng and Duan (2020) and Kaneko and Komachi (2019) have shown that BERT (Devlin et al., 2019)-based methods improve grammatical error detection performance in Chi- nese and English, respectively. Kaneko et al. (2020) and Didenko and Shaptala (2019) have shown a similar tendency in grammatical error correction. While these studies empirically prove the effective- ness of language models in grammatical error detec- tion …

Content truncated to 500 words