Argument Mining for Understanding Peer Reviews

Proceedings of NAACL-HLT 2019, pages 2131–2137 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2131 Argument Mining for Understanding Peer Reviews Xinyu Hua, Mitko Nikolov, Nikhil Badugu, Lu Wang Khoury College of Computer Sciences Northeastern University Boston, MA 02115 {hua.x, nikolov.m, badugu.n}@husky.neu.edu luwang@ccs.neu.edu Abstract Peer-review plays a critical role in the scien- tific writing and publication ecosystem. To as- sess the efficiency and efficacy of the review- ing process, one essential element is to un- derstand and evaluate the reviews themselves. In this work, we study the content and struc- ture of peer reviews under the argument min- ing framework, through automatically detect- ing (1) argumentative propositions put forward by reviewers, and (2) their types (e.g., evalu- ating the work or making suggestions for im- provement). We first collect 14.2K reviews from major machine learning and natural lan- guage processing venues. 400 reviews are an- notated with 10, 386 propositions and corre- sponding types of EVALUATION, REQUEST, FACT, REFERENCE, or QUOTE. We then train state-of-the-art proposition segmentation and classification models on the data to evalu- ate their utilities and identify new challenges for this new domain, motivating future di- rections for argument mining. Further ex- periments show that proposition usage varies across venues in amount, type, and topic. 1 Introduction Peer review is a process where domain experts scrutinize the quality of research work in their field, and it is a cornerstone of scientific discov- ery (Hettich and Pazzani, 2006; Kelly et al., 2014; Price and Flach, 2017). In 2015 alone, approxi- mately 63.4 million hours were spent on peer re- views (Kovanis et al., 2016). To maximize their benefit to the scientific community, it is crucial to understand and evaluate the construction and lim- itation of reviews themselves. However, minimal work has been done to analyze reviews’ content and structure, let alone to evaluate their qualities. As seen in Figure 1, peer reviews resemble arguments: they contain argumentative propo- sitions (henceforth propositions) that convey re- Review #1 (rating: 5, # sentences: 11) [Quality: This paper demonstrates that convolutional and re- lational neural networks fail to solve visual relation prob- lems . . . ]FACT [This points at important limitations of cur- rent neural network architectures where architectures depend mainly on rote memorization.]EVAL . . . [Significance: This work demonstrates failures of relational networks on relational tasks. . .]FACT [Pros: Important message about network limita- tions.]EVAL [Cons: Straightforward testing of network perfor- mance on specific visual relation tasks.]EVAL . . . Review #2 (rating: 5, # sentences: 10) [The authors present two autoregressive models . . .]FACT. . . [In that context , this work can be viewed as applying deep autore- gressive density estimators to policy gradient methods.]EVAL. . . [At least one of those papers ought to be cited.]REQ [It also seems like a simple, obvious baseline is missing from their experi- ments . . .]EVAL. . . [The method could even be made to capture dependencies between different actions by adding a …

Content truncated to 500 words