DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1120–1130 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1120 DeepSentiPeer: Harnessing Sentiment in Review Texts To Recommend Peer Review Decisions Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology Patna, India (tirthankar.pcs16,rajeev.ee15,asif,pb)@iitp.ac.in Abstract Automatically validating a research artefact is one of the frontiers in Artificial Intelligence (AI) that directly brings it close to competing with human intellect and intuition. Although criticized sometimes, the existing peer review system still stands as the benchmark of re- search validation. The present-day peer review process is not straightforward and demands profound domain knowledge, expertise, and intelligence of human reviewer(s), which is somewhat elusive with the current state of AI. However, the peer review texts, which contains rich sentiment information of the reviewer, re- flecting his/her overall attitude towards the re- search in the paper, could be a valuable en- tity to predict the acceptance or rejection of the manuscript under consideration. Here in this work, we investigate the role of reviewers sentiments embedded within peer review texts to predict the peer review outcome. Our pro- posed deep neural architecture takes into ac- count three channels of information: the pa- per, the corresponding reviews, and the review polarity to predict the overall recommenda- tion score as well as the final decision. We achieve significant performance improvement over the baselines (∼ 29% error reduction) proposed in a recently released dataset of peer reviews. An AI of this kind could assist the ed- itors/program chairs as an additional layer of confidence in the final decision making, espe- cially when non-responding/missing reviewers are frequent in present day peer review. 1 Introduction The rapid increase in research article submis- sions across different venues is posing a signif- icant management challenge for the journal ed- itors and conference program chairs1. Among 1Apparently CVPR, NIPS, AAAI 2019 received over 5100, 4900, 7000 submissions respectively! the load of works like assigning reviewers, ensur- ing timely receipt of reviews, slot-filling against the non-responding reviewer, taking informed de- cisions, communicating to the authors, etc., edi- tors/program chairs are usually overwhelmed with many such demanding yet crucial tasks. However, the major hurdle lies in to decide the acceptance and rejection of the manuscripts based on the re- views received from the reviewers. The quality, randomness, bias, inconsistencies in peer reviews is well-debated across the aca- demic community (Bornmann and Daniel, 2010). Due to the rise in article submissions and non- availability of expert reviewers, editors/program chairs are sometimes left with no other options than to assign papers to the novice, out of domain reviewers which sometimes results in more incon- sistencies and poor quality reviews. To study the arbitrariness inherent in the existing peer review system, organisers of the NIPS 2014 conference assigned 10% submissions to two different sets of reviewers and observed that the two commit- tees disagreed for more than quarter of the papers (Langford …

Content truncated to 500 words