A Rationale-Centric Framework for Human-in-the-loop Machine Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6986 - 6996 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Rationale-Centric Framework for Human-in-the-loop Machine Learning Jinghui Lu∗ 1,2,5, Linyi Yang∗ 3,4, Brian Mac Namee 1,2, Yue Zhang 3,4 1 The Insight Centre for Data Analytics, University College Dublin 2 School of Computer Science, University College Dublin 3 School of Engineering, Westlake University 4 Institute of Advanced Technology, Westlake Institute for Advanced Study 5 SenseTime Research {jinghui.lu, brian.macnamee}@ucd.ie {yanglinyi, zhangyue}@westlake.edu.cn Abstract We present a novel rationale-centric framework with human-in-the-loop – Rationales-centric Double-robustness Learning (RDL) – to boost model out-of-distribution performance few-shot learning scenarios. By using staticin semi-factual generation and dynamic human- intervened correction, RDL exploits rationales (i.e. phrases that cause the prediction), human interventions and semi-factual augmenta- tions to decouple spurious associations and bias models towards generally applicable underlying distributions, which enables fast and accurate generalisation. Experimental results show that RDL leads to significant prediction benefits on both in-distribution and out-of-distribution tests compared to many state-of-the-art benchmarks—especially for few-shot learning scenarios. We also perform extensive ablation studies to support in-depth analyses of each component in our framework. 1 Introduction Recent work finds that natural artefacts (Guru- rangan et al., 2018) or spurious patterns (Keith et al., 2020; Srivastava et al., 2020) in datasets can cause sub-optimal model performance for neural networks. As shown in Figure 1, the bold phrases—“100% bad” and “brain cell killing”—are underlying causes for a negative sen- timent prediction that most human readers would recognise. These are defined as rationales in this paper. The underlined phrase—“acting and plot”— has been incorrectly recognised as a causal term by the model used fort this example, and is referred to as a spurious pattern. Spurious patterns (or associations) are caused by natural artefacts or biases in training data (Lertvittayakumjorn and Toni, 2021), and are usually useless, or even harmful, at test time. This issue can be severe in few-shot learning (FSL) *These authors contributed equally to this work. This video is 100% bad with the brain cell killing acting and plot. Model Human Figure 1: A negative movie review with human an- notated causal terms (bold text) and spurious patterns recognised by the model (underlined text). scenarios. For instance, Kulesza et al. (2010) suggests that when a model is trained with a small subset of labelled data, it is prone to exploiting spurious patterns leading to poor generalisability that is evident in the performance decay in out- of-distribution (OOD) datasets. In spite of these issues, training deep neural networks using few labelled examples is a compelling scenario since unlabelled data may be abundant but labelled data is expensive to obtain in real-world applications (Lu and MacNamee, 2020; Lu et al., 2021). There is a strand of research addressing this sce- nario that seeks to improve model performance by “introducing methods and resources for train- ing models less sensitive to spurious patterns” (Kaushik et al., 2020). Most of this work relies on …

Content truncated to 500 words