Graph-Induced Transformers for Efficient Multi-Hop Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10288 - 10294 December 7-11, 2022 ©2022 Association for Computational Linguistics Graph-Induced Transformers for Efficient Multi-Hop Question Answering Giwon Hong1 Jeonghwan Kim1 Junmo Kang2∗ Sung-Hyon Myaeng1 1School of Computing, KAIST 2School of Interactive Computing, Georgia Institute of Technology {giwon.hong, jeonghwankim123, myaeng}@kaist.ac.kr junmo.kang@gatech.edu Abstract A graph is a suitable data structure to repre- sent the structural information of text. Re- cently, multi-hop question answering (MHQA) tasks, which require inter-paragraph/sentence linkages, have come to exploit such properties of a graph. Previous approaches to MHQA re- lied on leveraging the graph information along with the pre-trained language model (PLM) en- coders. However, this trend exhibits the follow- ing drawbacks: (i) sample inefficiency while training in a low-resource setting; (ii) lack of reusability due to changes in the model struc- ture or input. Our work proposes the Graph- Induced Transformer (GIT) that applies graph- derived attention patterns directly into a PLM, without the need to employ external graph mod- ules. GIT can leverage the useful inductive bias of graphs while retaining the unperturbed Transformer structure and parameters. Our ex- periments on HotpotQA successfully demon- strate both the sample efficient characteristic of GIT and its capacity to replace the graph modules while preserving model performance. 1 Introduction Graphs are a widely employed data structure from fake news detection in social media (Monti et al., 2019) to molecular graph generation in bio- medicine (Jin et al., 2018). More recently, graphs have come to manifest themselves in many subdo- mains of natural language processing (NLP) for its topological properties that provide useful con- nectivity inductive bias for model reasoning. Par- ticularly, some models in the multi-hop question answering (MHQA) tasks like HotpotQA (Yang et al., 2018) and MuSiQue (Trivedi et al., 2022) that require textual reasoning through multiple para- graphs (i.e., multi-hop) make use of such graph structures to model the internal structural infor- mation within the given document. The external graphs used in these studies have demonstrated the ∗Work was done while working at KAIST. 𝑛1 𝑛2 𝑛3 Language Model Graph Module 𝑙𝑎𝑦𝑒𝑟&𝑖 + 1 𝑙𝑎𝑦𝑒𝑟&𝑖 ... ... 𝑛1 Graph-Induced Transformer (GIT) 𝑛2 𝑛3 + Figure 1: Existing graph-based MHQA model vs. Graph-Induced Transformer (GIT). effectiveness of harnessing the information from textual units such as paragraphs and sentences to perform multi-hop reasoning in the QA setting. The graphs used in MHQA models like HGN (Fang et al., 2020), SAE (Tu et al., 2020) and DFGN (Qiu et al., 2019) are typically placed on top of the pre-trained language models (PLM) like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). These graphs use graph neural net- work’s (GNN) message propagation mechanism to update the node and edge parameters, learning to extract useful inductive, structural bias from the graph. These features are then fused with the con- textualized representations from the PLM to lever- age the information derived from this connectivity (e.g., paragraph-to-sentence) of these graphs. De- spite the merits of these …

Content truncated to 500 words