Various Errors Improve Neural Grammatical Error Correction

Various Errors Improve Neural Grammatical Error Correction Shota Koyama1,2, Hiroya Takamura1,2, Naoaki Okazaki1,2 1Tokyo Institute of Technology 2National Institute of Advanced Industrial Science and Technology shota.koyama@nlp.c.titech.ac.jp takamura.hiroya@aist.go.jp okazaki@c.titech.ac.jp Abstract The lack of parallel data is an obstacle in data- driven grammatical error correction. Recently, researchers have addressed this problem by exploring data augmentation methods. number of studies have focused on improvingA the diversity of errors in generated data. To in- vestigate the importance of error diversity and its impact on performance improvements, we designed and integrated 188 modules to gen- erate token-level or span-level errors. Exper- imental results demonstrate that the diversity of errors is crucial to performance improve- ments. The presented approach performs bet- ter than the baseline of round-trip translation, a purely data-driven approach. Furthermore, we report that a larger monolingual corpus does not always result in better performance. 1 Introduction Grammatical error correction (GEC) is an impor- tant NLP application that automatically corrects mistakes in text, and can assist language learn- ers in writing text. Typically, GEC is treated as data-driven machine translation (MT), where an erroneous sentence is ‘translated’ into a correct one (Brockett et al., 2006). Sequence-to-sequence models (Sutskever et al., 2014), which were orig- inally proposed for MT, have also achieved excel- lent performance in GEC (Junczys-Dowmunt et al., 2018). There are a number of publicly available datasets for GEC. The Lang-8 corpus (Mizumoto et al., 2012) is the largest among them, containing about one million sentence pairs. However, the amount of data is still insufficient for training high-quality neu- ral machine translation (NMT) models. One promis- ing way to address this issue is by generating ad- ditional erroneous data by adding artificial errors to grammatical sentences. Therefore, various state- of-the-art studies have developed the GEC-specific data augmentation methods in their ‘pre-training and fine-tuning’ paradigm (Lichtarge et al., 2019; Grundkiewicz et al., 2019; Kiyono et al., 2019). Many studies aim at increasing the diversity in er- ror types in augmented data, such that the model can handle a wide range of errors. Wan et al. (2020) presented a data augmentation method that injects noises to hidden-state representations based on ER- RANT error types (Bryant et al., 2017). Stahlberg and Kumar (2021) used error generation models, controlling types of generated errors. However, there are two issues that need to be discussed on the error diversity in augmented data. First, we are not certain of performance changes when error types vary. Previous studies did not ana- lyze the contributions of diversity in error types due to its lack of explicit control in augmented data. Sec- ond, we are not certain of the performance of other important factors for improving the usefulness of artificial error data other than the variety of error types; for example, the training epochs and the size of the monolingual corpus for data augmentation. In order to address these issues, we explore a rule- based approach for data augmentation in the GEC task. Although we are uncertain whether a purely data-driven method for …

Content truncated to 500 words