Neural Ranking with Weak Supervision for Open-Domain Question Answering : A Survey

Findings of the Association for Computational Linguistics: EACL 2023, pages 1736–1750 May 2-6, 2023 ©2023 Association for Computational Linguistics Neural Ranking with Weak Supervision for Open-Domain Question Answering : A Survey Xiaoyu Shen†1, Svitlana Vakulenko1, Marco del Tredici1, Gianni Barlacchi1 Bill Byrne1,2 and Adrià de Gispert1 1Amazon Alexa AI 2Univeristy of Cambridge †gyouu@amazon.com Abstract Neural ranking (NR) has become a key com- ponent for open-domain question-answering in order to access external knowledge. However, training a good NR model requires substan- tial amounts of relevance annotations, which is very costly to scale. To address this, a growing body of research works have been proposed to reduce the annotation cost by training the NR model with weak supervision (WS) in- stead. These works differ in what resources they require and employ a diverse set of WS signals to train the model. Understanding such differences is crucial for choosing the right WS technique. To facilitate this understand- ing, we provide a structured overview of stan- dard WS signals used for training a NR model. Based on their required resources, we divide them into three main categories: (1) only docu- ments are needed; (2) documents and questions are needed; and (3) documents and question- answer pairs are needed. For every WS signal, we review its general idea and choices. Promis- ing directions are outlined for future research. 1 Introduction Open-Domain Question Answering (ODQA) aims to provide precise answers in response to the user’s questions by drawing on a large collection of doc- uments (Voorhees et al., 1999). The majority of modern ODQA models follow the retrieve(-rerank)- read architecture: 1) given a question, a set of rele- vant documents are selected from a large document collection, and 2) the reader model produces an an- swer given this selected set and the question (Chen et al., 2017; Verga et al., 2021; Lee et al., 2021). Compared with parametric models without access to external knowledge, this architecture can better adapt to updated knowledge, offer easier interpre- tation and reduce hallucination (Zhu et al., 2021a; Shuster et al., 2021; Guo et al., 2022). Conventional methods use sparse retrievers (SRs) such as TF-IDF and BM-25 in the first stage Resource Weak-Supervision Signal Documents (§3) Self Contrastive Learning (§3.1) Question Generation (§3.2) Documents +Questions (§4) Sparse Retriever (§4) Pre-trained Language Model (§4) Supervised Teacher Model (§4) Documents +QA Pairs (§5) Answer as Document (§5.1) Answer-Document Mapping (§5.2) Latent-Variable Model (§5.3) Table 1: Overview of different weak-supervision signals, together with their required resources, that we can apply to train NR models for open-domain question-answering. to match questions and documents via lexical over- lap (Robertson and Walker, 1994), a process that may overlook semantically relevant documents with low lexical overlap with the question. Neu- ral ranking (NR) models resolve this issue by en- coding the questions and documents into dense vectors so that synonyms and paraphrases can be mapped to similar vectors through task-specific fine-tuning (Das et al., 2019; Karpukhin et al., 2020). However, training good NR models re- quires substantial amount …

Content truncated to 500 words