Uncertainty Aware Review Hallucination for Science Article Classification

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5004–5009 August 1–6, 2021. ©2021 Association for Computational Linguistics 5004 Uncertainty Aware Review Hallucination for Science Article Classification Korbinian Friedl1,∗, Georgios Rizos1,∗, Lukas Stappen2, Madina Hasan3, Lucia Specia1, Thomas Hain3, Bj¨orn W. Schuller1,2 1GLAM – Group on Language, Audio, & Music, Imperial College London, UK 2EIHW – Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany 3SpandH – Speech and Hearing Research Group, University of Sheffield, UK friedlkorbinian@gmail.com, gr912@ic.ac.uk Abstract The high subjectivity and costs inherent in peer reviewing have recently motivated the preliminary design of machine learning-based acceptance decision methods. However, such approaches are limited in that they: a) do not explore the usage of both the reviewer and area chair recommendations, b) do not explicitly model subjectivity on a per submission basis, and c) are not applicable in realistic settings, by assuming that review texts are available at test time, when these are exactly the inputs that should be considered to be missing in this ap- plication. We propose to utilise methods that model the aleatory uncertainty of the submis- sions, while also exploring different loss im- portance interpolations between area chair and reviewers’ recommendations. We also propose a modality hallucination approach to impute review representations at test time, providing the first realistic evaluation framework for this challenging task. 1 Introduction An analysis (Langford and Guzdial, 2015) of the NeurIPS 2014 experiment shows that 60% of the selected accepted papers were rejected by a second, independent review committee. Such significant reviewer disagreement makes the task of the area chair harder, and may even invite questioning of their decision. Software tools have been piloted in an effort to aid the human reviewers with a compu- tational recommendation on aspects like absence of bias and proper statistical reporting in scientific submissions (Sizo et al., 2019). Natural Language Understanding (NLU) could also offer decision support to the area chair, as ar- gued in (Ghosal et al., 2019; Stappen et al., 2020). Such systems jointly model the entire or part of the article and one (Kang et al., 2018; Wang and Wan, 2018; Ghosal et al., 2019), or a variable number ∗KF and GR contributed equally to this work. of potentially contradicting reviews (Stappen et al., 2020). We adopt the latter, review-aggregating ap- proach, that resembles the editorial process more. 1.1 Contributions In this short paper, we offer solutions to three par- ticularities of this task that the above approaches do not address: a) Often, the recommendations given by the area chair and the reviewers are in disagreement. Whereas previous studies have used either the former (Kang et al., 2018; Wang and Wan, 2018; Ghosal et al., 2019) or a soft label average of the latter (Stappen et al., 2020) for supervision, we show that both signals comprise complementary information. b) Whereas soft labels de-emphasise subjective articles with disagreeing reviews during training (Stappen et al., 2020), we manage to out- perform the latter study by explicitly modelling aleatory …

Content truncated to 500 words