Explanation-Based Human Debugging of NLP Models: A Survey

Explanation-Based Human Debugging of NLP Models: A Survey Piyawat Lertvittayakumjorn and Francesca Toni Department of Computing Imperial College London, UK {pl1515, ft}@imperial.ac.uk Abstract Debugging a machine learning model is hard since the bug usually involves the training data and the learning process. This becomes even harder for an opaque deep learning model if we have no clue about how the model actually works. In this survey, we review papers that exploit explanations to enable humans to give feedback and debug NLP models. We call this problem explanation-based human debugging (EBHD). In particular, we categorize and dis- cuss existing work along three dimensions of EBHD (the bug context, the workflow, and the experimental setting), compile findings on how EBHD components affect the feedback providers, and highlight open problems that could be future research directions. 1 Introduction Explainable AI focuses on generating explana- tions for AI models as well as for their predictions. It is gaining more and more attention these days since explanations are necessary in many appli- cations, especially in high-stake domains such as healthcare, law, transportation, and finance (Adadi and Berrada, 2018). Some researchers have ex- plored various merits of explanations to humans, such as supporting human decision making (Lai and Tan, 2019; Lertvittayakumjorn et al., 2021), increasing human trust in AI (Jacovi et al., 2020), and even teaching humans to perform challenging tasks (Lai et al., 2020). On the other hand, ex- planations can benefit the AI systems as well, for example, when explanations are used to promote system acceptance (Cramer et al., 2008), to verify the model reasoning (Caruana et al., 2015), and to find potential causes of errors (Han et al., 2020). In this paper, we review progress to date spe- cifically on how explanations have been used in the literature to enable humans to fix bugs in NLP models. We refer to this research area as explanation-based human debugging (EBHD), as a general umbrella term encompassing explana- tory debugging (Kulesza et al., 2010) and human- in-the-loop debugging (Lertvittayakumjorn et al., 2020). We define EBHD as the process of fixing or mitigating bugs in a trained model using human feedback given in response to explanations for the model. EBHD is helpful when the training data at hand leads to suboptimal models (due, for in- stance, to biases or artifacts in the data), and hence human knowledge is needed to verify and improve the trained models. In fact, EBHD is related to three challenging and intertwined issues in NLP: explainability (Danilevsky et al., 2020), inter- active and human-in-the-loop learning (Amershi et al., 2014; Wang et al., 2021), and knowledge integration (von Rueden et al., 2021; Kim et al., 2021). Although there are overviews for each of these topics (as cited above), our paper is the first to draw connections among the three towards the final application of model debugging in NLP. Whereas most people agree on the meaning of the term bug in software engineering, vari- ous meanings have been ascribed to this term in machine learning …

Content truncated to 500 words