Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3842 - 3852 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction Maksym Tarnavskyi†∗1, Artem Chernodub‡, and Kostiantyn Omelianchuk‡ †Ukrainian Catholic University, Faculty of Applied Sciences, tarnavskyi@ucu.edu.ua ‡Grammarly, firstname.lastname@grammarly.com Abstract In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large config- urations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architec- ture and vocabulary size. Our best ensemble achieves a new SOTA result with an F0.5 score of 76.05 on BEA-2019 (test), even without pre- training on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pre- trained on the generated Troy- datasets in com- bination with the publicly available synthetic PIE dataset achieves a near-SOTA1 result with an F0.5 score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available.2 1 Introduction The purpose of the Grammatical Error Correction (GEC) task is to correct grammatical errors in natu- ral texts. This includes correcting errors in spelling, punctuation, grammar, morphology, word choice, and others. An intelligent GEC system receives text containing mistakes and produces its corrected version. The GEC task is complicated and chal- lenging: the accuracy of edits, inference speed, and memory limitations are topics of intensive research. Currently, Machine Translation (MT) is the mainstream approach for GEC. In this setting, errorful sentences correspond to the source lan- guage, and error-free sentences correspond to the ∗This research was performed during Maksym Tar- navskyi’s work on Ms.Sc. thesis at Ukrainian Catholic Uni- versity (Tarnavskyi, 2021). 1To the best of our knowledge, our best single model gives way only to much heavier T5 model (Rothe et al., 2021). 2https://github.com/MaksTarnavskyi/ gector-large target language. Early GEC-MT methods lever- aged phrase-based statistical machine translation (PBSMT) (Yuan and Felice, 2013). Then this ap- proach rapidly evolved to seq2seq Neural Machine Translation (NMT) based on gated recurrent neu- ral networks (Yuan and Briscoe, 2016) and re- cent powerful Transformer-based seq2seq mod- els. Transformer-based models autoregressively capture the full dependency among output tokens; however, inference can be slow due to sequential decoding. Grundkiewicz et al. (2019) leveraged a Transformer model (Vaswani et al., 2017) that was pre-trained on synthetic GEC data and right-to- left re-ranking for ensemble. Kaneko et al. (2020) adopted several strategies of BERT (Devlin et al., 2018) usage for GEC. Recently, Rothe et al. (2021) built their system on top of T5 (Xue et al., 2021), a xxl version of the T5 Transformer encoder-decoder model and reached new state-of-the-art results (11B parameters). While still not as widespread as MT, the se- quence tagging approach for GEC, which generates a sequence of text edit operations encoded by tags for …

Content truncated to 500 words