Toward a Task of Feedback Comment Generation for Writing Learning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3206–3215, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3206 Toward a Task of Feedback Comment Generation for Writing Learning Ryo Nagata Konan University / 8-9-1 Okamoto, Kobe, 658-8501 Hyogo, Japan Japan Science and Technology Agency, PRESTO / 4-1-8 Honcho, Kawaguchi, Saitama 332-0012, Japan nagata-emnlp2019 @ ml.hyogo-u.ac.jp. Abstract In this paper, we introduce a novel task called feedback comment generation — a task of automatically generating feedback comments such as a hint or an explanatory note for writ- ing learning for non-native learners of En- glish. There has been almost no work on this task nor corpus annotated with feedback com- ments. We have taken the first step by creating learner corpora consisting of approximately 1,900 essays where all preposition errors are manually annotated with feedback comments. We have tested three baseline methods on the dataset, showing that a simple neural retrieval- based method sets a baseline performance with an F-measure of 0.34 to 0.41. Finally, we have looked into the results to explore what mod- ifications we need to make to achieve better performance. We also have explored problems unaddressed in this work. 1 Introduction In this paper, we introduce a novel task called Feedback Comment Generation. Feedback com- ment generation is1 the task of generating hints or explanatory notes (hereafter, feedback comments) for language learners2. Figure 1 shows an exam- ple of feedback comments on preposition use. As in this example, feedback comments are typically given to erroneous words in a given text so that the writer can understand why their writing is not good together with underlying grammatical rules and more importantly so that they can improve their writing skill. Feedback comment generation complements grammatical error detection/correction in lan- guage learning. With the advent of Deep Neural Networks (DNN), their performance has dramati- cally improved in recent years. However, they are 1The strict definition will be introduced in Subsect. 3. 2In this paper, language learners refer to learners of En- glish as Foreign Language (EFL). not capable of explaining why detected words are erroneous nor why they should be corrected as in- dicated. This limitation is particularly problematic for beginner to intermediate learners. They might not be able to understand and acquire the under- lying rules; if so, they will likely make similar er- rors in the future. This is exactly where feedback comment generation comes in. Namely, it com- plements grammatical error detection/correction by generating feedback comments to help learners understand and acquire the underlying rules. This form of feedback should be useful for language learners as Bitchener et al. (2005)3 and Sheen (2007) show. Despite its usefulness, there has been almost no work on feedback comment generation as Sect. 2 will describe. One of the major reasons is that there exists no publicly available dataset for re- search on feedback comment generation. It is costly and time-consuming to annotate …

Content truncated to 500 words