Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation

Proceedings of the 28th International Conference on Computational Linguistics, pages 6302–6314 Barcelona, Spain (Online), December 8-13, 2020 6302 Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation Zhongfen Deng1, Hao Peng2, 3, Congying Xia1 , Jianxin Li2, Lifang He4 , Philip S. Yu1 1Department of Computer Science, University of Illinois at Chicago, Chicago, USA 2BDBC, Beihang University, Beijing, China 3School of Cyber Science and Technology, Beihang University, Beijing, China 4Department of Computer Science and Engineering, Lehigh University, Bethlehem, USA {zdeng21,cxia8,psyu}@uic.edu, {penghao,lijx}@act.buaa.edu.cn lih319@lehigh.edu Abstract Review rating prediction of text reviews is a rapidly growing technology with a wide range of applications in natural language processing. However, most existing methods either use hand- crafted features or learn features using deep learning with simple text corpus as input for review rating prediction, ignoring the hierarchies among data. In this paper, we propose a Hierarchical bi-directional self-attention Network framework (HabNet) for paper review rating prediction and recommendation, which can serve as an effective decision-making tool for the academic paper review process. Specifically, we leverage the hierarchical structure of the paper reviews with three levels of encoders: sentence encoder (level one), intra-review encoder (level two) and inter- review encoder (level three). Each encoder first derives contextual representation of each level, then generates a higher-level representation, and after the learning process, we are able to identify useful predictors to make the final acceptance decision, as well as to help discover the inconsis- tency between numerical review ratings and text sentiment conveyed by reviewers. Furthermore, we introduce two new metrics to evaluate models in data imbalance situations. Extensive exper- iments on a publicly available dataset (PeerRead) and our own collected dataset (OpenReview) demonstrate the superiority of the proposed approach compared with state-of-the-art methods. 1 Introduction With an increasing submission of academic papers in recent years, the task of making final decisions manually incurs significant overheads to the program chairs, it is desirable to automate the process. In this study, we aim at utilizing document-level semantic analysis for paper review rating prediction and recommendation. Given the reviews of each paper from several reviewers as input, our goal is to infer the final acceptance decision for that paper and the reviewers’ evaluation with respect to a numeric rating (e.g., 1-10 points). Paper review rating prediction and recommendation is a practical and important task in AI applications which will help improve the efficiency of the paper review process. It is also intended to enhance the consistency of the assessment procedures and outcomes, and to diversify the paper review process by comparing human recommended rating with machine recommended rating. In the literature, most of existing studies cast review rating prediction as a multi-class classification/regression task (Pang and Lee, 2005). They build a predictor by using supervised machine learning models with review texts and corresponding ratings. Due to the importance of features, most researches focus on extracting effec- tive features such as context-level features (Qu et al., 2010) and user features (Gao et al., 2013) to boost prediction performance. However, feature engineering …

Content truncated to 500 words