Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inference

Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inference Mokanarangan Thayaparan†‡, Marco Valentino†‡, Deborah Ferreira†, Julia Rozanova†, Andr´e Freitas†‡ Department of Computer Science, University of Manchester, United Kingdom† Idiap Research Institute, Switzerland‡ {mokanarangan.thayaparan,marco.valentino,deborah.ferreira, julia.rozanova,andre.freitas}@manchester.ac.uk Abstract This paper presents Diff-Explainer, the first hybrid framework for explainable multi-hop inference that integrates explicit constraints with neural architectures through differen- tiable convex optimization. Specifically, Diff- Explainer allows for the fine-tuning of neural representations within a constrained optimi- zation framework to answer and explain multi-hop questions in natural language. To demonstrate the efficacy of the hybrid frame- work, we combine existing ILP-based solvers for multi-hop Question Answering (QA) with Transformer-based representations. An exten- sive empirical evaluation on scientific and commonsense QA tasks demonstrates that the integration of explicit constraints in a end-to-end differentiable framework can sig- nificantly improve the performance of non- differentiable ILP solvers (8.91%–13.3%). Moreover, additional analysis reveals that Diff-Explainer is able to achieve strong perfor- mance when compared to standalone Trans- formers and previous multi-hop approaches while still providing structured explanations in support of its predictions. 1 Introduction Explainable Question Answering (QA) in com- plex domains is often modeled as a multi-hop inference problem (Thayaparan et al., 2020; Valentino et al., 2021; Jansen et al., 2021). In this context, the goal is to answer a given question through the construction of an explanation, typi- cally represented as a graph of multiple intercon- nected sentences supporting the answer (Figure 1). (Khashabi et al., 2018; Jansen, 2018; Kundu et al., 2019; Thayaparan et al., 2021). However, explainable QA models exhibit lower performance when compared to state-of-the-art approaches, which are generally represented by Transformer-based architectures (Khashabi et al., 2020; Devlin et al., 2019; Khot et al., 2020). While Transformers are able to achieve high ac- curacy due to their ability to transfer linguistic and semantic information to downstream tasks, they are typically regarded as black-boxes (Liang et al., 2021), posing concerns about the inter- pretability and transparency of their predictions (Rudin, 2019; Guidotti et al., 2018). To alleviate the aforementioned limitations and find a better trade-off between explainability and inference performance, this paper proposes Diff- Explainer (∂-Explainer), a novel hybrid frame- work for multi-hop and explainable QA that combines constraint satisfaction layers with pre- trained neural representations, enabling end-to- end differentiability. Recent works have shown that certain convex optimization problems can be represented as in- dividual layers in larger end-to-end differentia- ble networks (Agrawal et al., 2019a,b; Amos and Kolter, 2017), demonstrating that these layers can be adapted to encode constraints and dependen- cies between hidden states that are hard to capture via standard neural networks. In this paper, we build upon this line of re- search, showing that convex optimization layers can be integrated with Transformers to improve explainability and robustness in multi-hop in- ference problems. To illustrate the impact of end-to-end differentiability, we integrate the con- straints of existing ILP solvers (i.e., TupleILP [Khot et al., 2017], ExplanationLP [Thayaparan et al., 2021]) into a hybrid framework. Specif- ically, we propose a methodology to transform …

Content truncated to 500 words