Automating Idea Unit Segmentation and Alignment for Assessing Reading Comprehension via Summary Protocol Analysis

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4663–4673 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4663 Automating Idea Unit Segmentation and Alignment for Assessing Reading Comprehension via Summary Protocol Analysis Marcello Gecchele†, Hiroaki Yamada†, Takenobu Tokunaga† Yasuyo Sawaki‡, Mika Ishizuka§ † Tokyo Institute of Technology, ‡ Waseda University, § Tokyo University of Technology Tokyo, Japan gecchele.m.aa@m.titech.ac.jp, yamada@c.titech.ac.jp, take@c.titech.ac.jp, ysawaki@waseda.jp, ishizuka@stf.teu.ac.jp Abstract In this paper, we approach summary evaluation from an applied linguistics (AL) point of view. We provide computational tools to AL researchers to simplify the process of Idea Unit (IU) segmentation. The IU is a segmentation unit that can identify chunks of information. These chunks can be compared across documents to measure the content overlap between a summary and its source text. We propose a full revision of the annotation guidelines to allow machine implementation. The new guideline also improves the inter-annotator agreement, rising from 0.547 to 0.785 (Cohen’s “κ”). We release L2WS 2021, a IU gold standard corpus composed of 40 manually annotated student summaries. We propose IUExtract; i.e. the first automatic segmentation algorithm based on the IU. The algorithm was tested over the L2WS 2021 corpus. Our results are promising, achieving a precision of 0.789 and a recall of 0.844. We tested an existing approach to IU alignment via word embeddings with the state of the art model SBERT. The recorded precision for the top 1 aligned pair of IUs was 0.375. We deemed this result insufficient for effective automatic alignment. We propose “SAT”, an online tool to facilitate the collection of alignment gold standards for future training. Keywords: Idea Unit, Student Summary Evaluation, Summary Response Analysis, Segmentation, Alignment 1. Introduction In second language learning, summaries are among the most popular type of student assignments. Summaries are effective tools to teach techniques for the identi- fication and concise delivery of information (Graham and Perin, 2007), to assess the language comprehension ability of students (Westby et al., 2010) and to teach paraphrasing skills; i.e., the ability to rewrite informa- tion adopting a different expression. Such techniques are paramount for academic writing as they are neces- sary to avoid plagiarism (Keck, 2014). While writing a summary, students are asked to read a source text, pick the core information and rewrite it in a shorter text. In this paper, we are looking to evaluate English lan- guage summaries in regards to their content, assess- ing whether a summary preserves the gist of the source text; i.e. the core information relayed by the source text. This is challenging, as paraphrased texts obfus- cate the connection between the information contained across the source and summary texts. Students can commit mistakes by misunderstanding the original text, by outright fabricating new information or by misre- porting facts (Winograd, 1984). Teachers have to read a summary multiple times to correctly assess the level of content overlap between two texts. In a classroom setting, each teacher adopts their own personal tech- nique for summary …

Content truncated to 500 words