A Dataset of Argumentative Dialogues on Scientific Papers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7684–7699 July 9-14, 2023 ©2023 Association for Computational Linguistics A Dataset of Argumentative Dialogues on Scientific Papers Federico Ruggeri1 ∗ Mohsen Mesgar2 † 1DISI, University of Bologna 2Bosch Center for Artificial Intelligence, Renningen, Germany 3Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt 1federico.ruggeri6@unibo.it 2mohsen.mesgar@bosch.com 3 www.ukp.tu-darmstadt.de Iryna Gurevych3 Abstract With recent advances in question-answering models, various datasets have been collected to improve and study the effectiveness of these models on scientific texts. Questions and an- swers in these datasets explore a scientific pa- per by seeking factual information from the pa- per’s content. However, these datasets do not tackle the argumentative content of scientific papers, which is of huge importance in persua- siveness of a scientific discussion. We intro- duce ArgSciChat, a dataset of 41 argumenta- tive dialogues between scientists on 20 NLP papers. The unique property of our dataset is that it includes both exploratory and argumen- tative questions and answers in a dialogue dis- course on a scientific paper. Moreover, the size of ArgSciChat demonstrates the difficulties in collecting dialogues for specialized domains. Thus, our dataset is a challenging resource to evaluate dialogue agents in low-resource domains, in which collecting training data is costly. We annotate all sentences of dialogues in ArgSciChat and analyze them extensively. The results confirm that dialogues in ArgSci- Chat include exploratory and argumentative in- teractions. Furthermore, we use our dataset to fine-tune and evaluate a pre-trained document- grounded dialogue agent. The agent achieves a low performance on our dataset, motivating a need for dialogue agents with a capability to reason and argue about their answers. We pub- licly release ArgSciChat1. 1 Introduction The enormous and ever-growing number of scien- tific papers (Munroe, 2013; Ronzano and Saggion, 2015) make scientific text processing imperative. A considerable body of research in NLP is dedicated to developing methods to provide insights from ∗ Work done while conducting an internship at UKP Lab. † Work done while being a postdoc at UKP Lab. 1https://github.com/UKPLab/ acl2023-argscichat such texts (Beltagy et al., 2019; Zhang et al., 2021; Wadden et al., 2022; Zhang et al., 2021; Parveen et al., 2016; Mysore et al., 2022). Recent advances in Question-Answering (QA) models made them a great asset in accessing the content of scientific papers (Dasigi et al., 2021). To develop and evalu- ate such QA models, various datasets (Section 2) have been collected (Dasigi et al., 2021; Pampari et al., 2018; Pappas et al., 2018; Jin et al., 2019; Tsatsaronis et al., 2015; Krallinger et al., 2020). For each scientific paper in these datasets, a set of questions with the exploratory intention is defined. These questions explore the content of the paper by seeking factual information. Thus, answers in these datasets are mainly limited to yes/no, named entities, or extracted from the text of the paper. Scientific answers should be explained with ar- gumentation …

Content truncated to 500 words