Grammatical Error Correction via Supervised Attention in the Vicinity of Errors

Grammatical Error Correction via Supervised Attention in the Vicinity of Errors Hiromichi Ishii1∗ Akihiro Tamura2 Takashi Ninomiya1 1Ehime University 2Doshisha University ishii@ai.cs.ehime-u.ac.jp aktamura@mail.doshisha.ac.jp ninomiya@cs.ehime-u.ac.jp Abstract This study proposes a new supervised atten- tion mechanism for grammatical error correc- tion, which is trained to focus the encoder- decoder attention to each word in the corrected sentence on words adjacent to corresponding words in the error sentence. Experiments on the CoNLL 2014 test set show that the per- formance of a Transformer-based grammati- cal error correction model, Copy-Augmented Transformer, can be improved by 0.73 F0.5 points by incorporating our proposed attention mechanism. 1 Introduction Extensive research has taken place on grammati- cal error correction techniques which automatically correct grammatically erroneous sentences (error sentences) into accurate sentences (corrected sen- tences), due in part to their usefulness as tools for foreign language learning. A variety of methods have been proposed in the past. In recent years, methods using neural networks have achieved the highest accuracy and have become mainstream in the field. Among grammatical error correction mod- els using neural networks, those based on Trans- former (Vaswani et al., 2017) architecture have per- formed well. Transformer-based models’ character- istics include a self-attention mechanism that grasps relationships between words within the same sen- tence (error sentence or corrected sentence), as well as an encoder-decoder attention mechanism that ∗Present affiliation is Daihatsu Motor Co., Ltd. identifies the error sentence words on which to fo- cus when producing the words of the corrected sen- tence (relationship between the words of the error sentence and those of the corrected sentence). In general, the relationships between words captured by these attention mechanisms are learned automat- ically. Chollampatt and Ng (2018) compare and ana- lyze encoder-decoder attention automatically cap- tured via grammatical error correction models based on CNN and LSTM. They consider the attention given to each word in the corrected sentence, con- cluding that focusing on the words adjacent to cor- responding words in the error sentence may be more beneficial than focusing on corresponding words in the error sentence themselves. Using a Copy-Augmented Transformer (Zhao et al., 2019) — a model of grammatical error correc- tion based on Transformer architecture — this study proposes a method of machine learning whereby one of the heads of a multi-head encoder-decoder atten- tion mechanism is programmed to focus the atten- tion given to each word in the corrected sentence only on words adjacent to corresponding words in the error sentence. Through the proposed method, the model is expected to be able to learn a gram- matical error correction model by focusing on error- adjacent words. An evaluation experiment on grammatical error correction was conducted using the CoNLL-2014 test-set (Ng et al., 2014). A Copy-Augmented Transformer was programmed to focus its attention only on words adjacent to corrected words, resulting in a confirmed increase in F0.5 by 0.73 points. Figure 1: Schematic diagram of the Transformer 2 Copy-Augumented Transformer This section illustrates the Copy-Augmented Trans- former (Zhao et al., 2019) on which …

Content truncated to 500 words