TYPIC: A Corpus of Template-Based Diagnostic Comments on Argumentation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5916–5928 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5916 TYPIC: A Corpus of Template-Based Diagnostic Comments on Argumentation Shoichi Naito1,2,5, Shintaro Sawada3, Chihiro Nakagawa3,2,Naoya Inoue4,2,∗, Kenshi Yamaguchi1, Iori Shimizu3, Farjana Sultana Mim1, Keshav Singh1, Kentaro Inui1,2 1Tohoku University, 2RIKEN, 3Osaka Prefecture University, 4Stony Brook University, 5Ricoh Company, Ltd. shohichi.naitoh@jp.ricoh.com, {szb03072, scb03059}@edu.osakafu-u.ac.jp, chihiro@me.osakafu-u.ac.jp, {naoya.inoue.lab, keshav.singh29}@gmail.com {kenshi.yamaguchi.e7, inui}@tohoku.ac.jp, mim.farjana.sultana.t3@dc.tohoku.ac.jp Abstract Providing feedback on the argumentation of the learner is essential for developing critical thinking skills, however, it requires a lot of time and effort. To mitigate the overload on teachers, we aim to automate a process of providing feedback, especially giving diagnostic comments which point out the weaknesses inherent in the argumentation. It is recommended to give specific diagnostic comments so that learners can recognize the diagnosis without misinterpretation. However, it is not obvious how the task of providing specific diagnostic comments should be formulated. We present a formulation of the task as template selection and slot filling to make an automatic evaluation easier and the behavior of the model more tractable. The key to the formulation is the possibility of creating a template set that is sufficient for practical use. In this paper, we define three criteria that a template set should satisfy: expressiveness, informativeness, and uniqueness, and verify the feasibility of creating a template set that satisfies these criteria as a first trial. We will show that it is feasible through an annotation study that converts diagnostic comments given in a text to a template format. The corpus used in the annotation study is publicly available. Keywords: argument, argumentation, debate, formative feedback, diagnostic comment 1. Introduction Argumentation and debate are known to be effective tools for developing critical thinking skills (Roy and Macchiette, 2005). In such argumentation-based ed- ucation, teachers’ feedback helps learners efficiently develop their skills (Dur´on et al., 2006; Tsui, 1999; Paulus, 1999). Learning becomes more efficient when specific feedback is given to the learners (Shute, 2008). However, it is impractical to ask all teachers to do this because they will need a substantial amount of time to provide specific feedback to each student, and not all teachers have been trained to teach argumenta- tion (Driver et al., 2000). It would be highly beneficial to develop a technology that automatically gives spe- cific feedback to students. Essay Scoring aims for assessing students’ arguments and giving feedback as a score. Some studies give a single holistic score for the entire essay (Burstein and Chodorow, 1999; Dong and Zhang, 2016), while others give multi-dimensional scores such as organi- zation, clarity, and justification (Persing et al., 2010; Persing and Ng, 2013; Persing and Ng, 2014; Persing and Ng, 2015; Wachsmuth et al., 2017b; Carlile et al., 2018). However, we argue that these score-based feed- back is not specific enough for students to develop their skills. Consider an example argument in Fig. 1. Where we imagine students receive feedback such as Partially justified: …

Content truncated to 500 words