Exploring Methods for Generating Feedback Comments for Writing Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9719–9730 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9719 Exploring Methods for Generating Feedback Comments for Writing Learning Kazuaki Hanawa1,2, Ryo Nagata3,4, Kentaro Inui2,1 1RIKEN Center for Advanced Intelligence Project, 2Tohoku University 3Konan University, 4JST, PRESTO kazuaki.hanawa@riken.jp nagata-emnlp2021 @ ml.hyogo-u.ac.jp. inui@ecei.tohoku.ac.jp Abstract The task of generating explanatory notes for language learners is known as feedback com- ment generation. Although various generation techniques are available, little is known about which methods are appropriate for this task. Nagata (2019) demonstrates the effectiveness of neural-retrieval-based methods in generat- ing feedback comments for preposition use. Retrieval-based methods have limitations in that they can only output feedback comments existing in a given training data. Furthermore, feedback comments can be made on other grammatical and writing items than preposi- tion use, which is still unaddressed. To shed light on these points, we investigate a wider range of methods for generating many feed- back comments in this study. Our close anal- ysis of the type of task leads us to investigate three different architectures for comment gen- eration: (i) a neural-retrieval-based method as a baseline, (ii) a pointer-generator-based gener- ation method as a neural seq2seq method, (iii) a retrieve-and-edit method, a hybrid of (i) and (ii). Intuitively, the pointer-generator should outperform neural-retrieval, and retrieve-and- edit should perform best. However, in our ex- periments, this expectation is completely over- turned. We closely analyze the results to reveal the major causes of these counter-intuitive re- sults and report on our findings from the exper- iments. 1 1 Introduction Feedback comment generation is the task of gen- erating explanatory notes for writing learning. An example of a feedback comment would be: (1) Target sentence: *We discussed about it. Feedback comment: Since discuss is a tran- sitive verb, about is not required.2 1Our source code is available at https://github.c om/k-hanawa/fcg_emnlp2021 2Note that feedback comments are actually written in Japanese but we show English translation for clarity in this paper. This type of feedback can assist the writer in deter- mining why their writing was wrong and how to fix it. While datasets (Nagata, 2019; Nagata et al., 2020; Pilan et al., 2020) for this task have be- come available, very little is known about the feed- back comment generation methods. Nagata (2019) demonstrates deep neural network (DNN)-based re- trieval methods are effective in generating feedback comments for preposition use, which is almost the only knowledge available. Recently, a wide variety of DNN-based generation methods are available and they can improve feedback comment genera- tion performance significantly. Furthermore, Na- gata (2019) focused on feedback comments for preposition use. DNN-based retrieval methods might not perform well on more general feedback comments (Nagata et al., 2020; Pilan et al., 2020) because they involve a wide range of writing items and rules. More sophisticated methods will likely perform better in such cases. Given these contexts, we investigate feedback comment generation methods for preposition and general feedback comments in this study to …

Content truncated to 500 words