Knowledge Transfer from Answer Ranking to Answer Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9481 - 9495 December 7-11, 2022 ©2022 Association for Computational Linguistics Knowledge Transfer from Answer Ranking to Answer Generation Matteo Gabburo1∗, Rik Koncel-Kedziorski2, Siddhant Garg2, Luca Soldaini3†, Alessandro Moschitti2 1University of Trento , 2Amazon Alexa AI, 3Allen Institute for AI matteo.gabburo@unitn.it {rikdz,sidgarg,amosch}@amazon.com lucas@allenai.org Abstract Recent studies show that Question Answer- ing (QA) based on Answer Sentence Selec- tion (AS2) can be improved by generating an improved answer from the top-k ranked an- swer sentences (termed GenQA). This allows for synthesizing the information from multi- ple candidates into a concise, natural-sounding answer. However, creating large-scale super- vised training data for GenQA models is very challenging. In this paper, we propose to train a GenQA model by transferring knowledge from a trained AS2 model, to overcome the aforementioned issue. First, we use an AS2 model to produce a ranking over answer candi- dates for a set of questions. Then, we use the top ranked candidate as the generation target, and the next k top ranked candidates as con- text for training a GenQA model. We also pro- pose to use the AS2 model prediction scores for loss weighting and score-conditioned in- put/output shaping, to aid the knowledge trans- fer. Our evaluation on three public and one large industrial datasets demonstrates the supe- riority of our approach over the AS2 baseline, and GenQA trained using supervised data. 1 Introduction In recent times, extractive QA research can be cat- egorized into two broad directions for the task of producing the final answer for a question: (i) An- swer Sentence Selection (AS2), which, given a question and a set of answer-sentence candidates, selects sentences (e.g., retrieved by a search en- gine) that correctly answer the question; and (ii) Machine Reading (MR), e.g., (Chen et al., 2017), which, given a question and a reference text, in- volves finding an exact text span that answers the question. AS2 models can perform more efficiently with large text databases (as they originated from the TREC-QA track (Voorhees, 1999)), and there ∗Work done as an intern at Amazon Alexa AI †Work completed at Amazon Alexa AI seems a renewed research interest in these models for applications to personal assistants, e.g., Alexa (Garg et al., 2020; Matsubara et al., 2020a; Garg and Moschitti, 2021). Both approaches (AS2 and MR) when applied for QA over unstructured web text, while effective, may have certain drawbacks. Arbitrary web sen- tences may not contain all the information needed to answer a question, or may contain distracting ex- traneous information. Moreover, they may have a particular sentiment or style that is not suited to QA, or be too structurally reliant on longer discourse context to serve as a standalone answer. In light of this, researchers have been exploring text gen- eration systems for writing ‘better’ answers. For example, in MR, RAG (Lewis et al., 2020b) gener- ates an answer from a set of documents selected by dense passage retrieval models. For AS2 systems, …

Content truncated to 500 words