DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1234 - 1249 July 10-15, 2022 ©2022 Association for Computational Linguistics DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions Neha Nayak Kennard Tim O’Gorman Rajarshi Das Akshay Sharma Chhandak Bagchi Matthew Clinton Pranay Kumar Yelugam Hamed Zamani University of Massachusetts AmherstAndrew McCallum {kennard, togorman, rajarshi, akshaysharma, cbagchi, mfclinton, pyelugam, zamani, mccallum}@cs.umass.edu Abstract At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DIS- APERE synthesizes label sets from prior work and extends them to include fine-grained an- notation of the rebuttal sentences, characteriz- ing their context in the review and the authors’ stance towards review arguments. Further, we annotate every review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argu- mentative strategies employed by the review- ers and authors provides useful signal for area chairs and other decision makers. 1 Introduction Peer review performs the essential role of quality control in the dissemination of scientific knowl- edge. The recent rapid increase in academic output places an immense burden on decision makers such as area chairs and editors, as their decisions must take into account not only extensive manuscripts, but enormous additional amounts of technical text including reviews, rebuttals, and other discussions. One long term goal of research in peer review is to support decision makers in managing their work- load by providing tools to help them efficiently absorb the discussions they must read. While ma- chine learning should not be used to produce con- densed accounts of the peer review text due to the risk of amplifying biases (Zhao et al., 2017), ML tools could nevertheless help manage information overload by identifying patterns in the data, such as argumentative strategies, goals, and intentions. Any such research requires an extensive labeled dataset. While the OpenReview platform (Soergel et al., 2013) has made it easy to obtain unlabeled public peer review text, labeling this data for su- pervised NLP requires highly qualified annotators. Correct interpretation of the discourse structure of the text requires an understanding of the technical content, precluding the use of standard crowdsourc- ing techniques. Prior work on discourse in peer review has focused this qualified labor force on labeling arguments extracted from the text, which enables the complete annotation of more examples, at the expense of research on non-argumentative behaviors in peer review. While there has been extensive research and deep analysis of different aspects of peer review, the taxonomies used to de- scribe review argumentation are disparate and not directly compatible. Finally, there …

Content truncated to 500 words