Weakly Supervised Pre-Training for Multi-Hop Retriever

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 694–704 August 1–6, 2021. ©2021 Association for Computational Linguistics 694 Weakly Supervised Pre-Training for Multi-Hop Retriever Yeon Seonwoo†, Sang-Woo Lee‡§, Ji-Hoon Kim‡§, Jung-Woo Ha‡§, Alice Oh† †KAIST ‡NAVER AI Lab, §NAVER Clova yeon.seonwoo@kaist.ac.kr {sang.woo.lee,genesis.kim,jungwoo.ha}@navercorp.com alice.oh@kaist.edu Abstract In multi-hop QA, answering complex ques- tions entails iterative document retrieval for finding the missing entity of the question. The main steps of this process are sub-question detection, document retrieval for the sub- question, and generation of a new query for the final document retrieval. However, building a dataset that contains complex questions with sub-questions and their corresponding docu- ments requires costly human annotation. To address the issue, we propose a new method for weakly supervised multi-hop retriever pre- training without human efforts. Our method includes 1) a pre-training task for generating vector representations of complex questions, 2) a scalable data generation method that pro- duces the nested structure of question and sub- question as weak supervision for pre-training, and 3) a pre-training model structure based on dense encoders. We conduct experiments to compare the performance of our pre-trained re- triever with several state-of-the-art models on end-to-end multi-hop QA as well as document retrieval. The experimental results show that our pre-trained retriever is effective and also robust on limited data and computational re- sources. 1 Introduction Multi-hop QA is the task of answering complex questions that requires reasoning across multiple documents (Nogueira and Cho, 2017; Nie et al., 2019; Sun et al., 2019; Fang et al., 2020; Zhao et al., 2020). The core components of multi-hop reasoning are identifying the missing entity in the question and generating a new query with the miss- ing entity. Figure 1 shows an example of the reason- ing process in multi-hop QA. In the example, the missing entity, which we call bridge entity, of the question is “Jupiter”. To answer the question, the correct document for the sub-question “the largest planet in the Solar System,” should be retrieved. Supervised training of the multi-hop QA models for these intermediate reasoning steps requires a dataset of complex questions, sub-questions, and their corresponding documents. However, building such dataset requires costly human annotation and cannot be done at scale (Min et al., 2019; Wolfson et al., 2020). When there is limited annotated supervision sig- nal, weakly supervised pre-training can be a solu- tion (Devlin et al., 2019; Liu et al., 2019) which has shown effectiveness in open-domain QA (Lee et al., 2019; Guu et al., 2020). Unlike open-domain QA, it is not trivial to apply a pre-training method to multi-hop QA due to the complexity in generating weak supervision data. In open-domain QA, weak supervision is generated by selecting a document from a corpus and extracting a sentence from the document. The sentence becomes a pseudo ques- tion, and the document becomes a pseudo support- ing document to be predicted by retrievers. This two-step process for weak supervision cannot be directly applied in multi-hop QA since each multi- hop question refers to …

Content truncated to 500 words