LM-Critic: Language Models for Unsupervised Grammatical Error Correction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7752–7763 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7752 LM-Critic: Language Models for Unsupervised Grammatical Error Correction Michihiro Yasunaga Jure Leskovec Percy Liang Stanford University {myasu,jure,pliang}@cs.stanford.edu Abstract Training a model for grammatical error correc- tion (GEC) requires a set of labeled ungram- matical/grammatical sentence pairs, but man- ually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled ex- amples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task. In this work, we show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be gram- matical if the LM assigns it a higher probability than its local perturbations. We apply this LM- Critic and BIFI along with a large set of unla- beled sentences to bootstrap realistic ungram- matical/grammatical pairs for training a correc- tor. We evaluate our approach on GEC datasets across multiple domains (CoNLL-2014, BEA- 2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting (+0.5 F0.5). 1 Introduction Grammatical error correction (GEC) is the task of fixing grammatical errors in text, such as typos, tense and article mistakes. Recent works cast GEC as a translation problem, using encoder-decoder models to map bad (ungrammatical) sentences into good (grammatical) sentences (Yuan and Briscoe, 2016; Xie et al., 2016; Ji et al., 2017; Chollampatt and Ng, 2018; Junczys-Dowmunt et al., 2018). These methods rely on a combination of human-labeled data (i.e., ⟨bad, good⟩ pairs) (Nicholls, 2003; Yannakoudakis et al., 2011; Bryant et al., 2019) and synthetic data, which are generated by corrupting good sentences into ⟨synthetic bad, good⟩ pairs (Awasthi et al., 2019; Kiyono et al., 2019). Human-labeled pairs are representative of real human errors but are expensive to obtain, while synthetic pairs are cheap but are unrealistic, deviating from the distribution of grammatical (a) Grammatical error correction (GEC) via LM-Critic She like cats. ✘ Bad (ungrammatical) ✓ Good (grammatical) Critic Sentence Bad Fixer She like cats. She likes cats. Good LM-Critic GEC system (b) Idea behind LM-Critic: Local optimum criterion Figure 1: Illustration of LM-Critic. (a) In this work, we train a fixer for grammatical error correction (GEC) by leveraging LM-Critic that assesses the grammaticality. (b) LM-Critic deems a sentence to be grammatical if a pretrained language model (e.g., GPT2) assigns it a higher probability than candidates in its local neighborhood (e.g., edit distance 1). errors humans make (Grundkiewicz et al., 2019). How to obtain inexpensive yet realistic paired data to improve GEC remains a key challenge, especially in domains or languages with no labeled GEC data (Napoles et al., 2019; Náplava and Straka, 2019). Break-It-Fix-It (BIFI; Yasunaga and Liang (2021)) is a recent method to obtain realistic paired data from unlabeled …

Content truncated to 500 words