Learning Answer Generation using Supervision from Automatic Question Answering Evaluators

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8389–8403 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Answer Generation using Supervision from Automatic Question Answering Evaluators Matteo Gabburo1∗, Siddhant Garg2, Rik Koncel Kedziorski3†, Alessandro Moschitti2 1University of Trento , 2Amazon Alexa AI, 3Kensho Technologies, Inc. matteo.gabburo@unitn.it {sidgarg,amosch}@amazon.com rikka@kensho.com Abstract Recent studies show that sentence-level extrac- tive QA, i.e., based on Answer Sentence Se- lection (AS2), is outperformed by Generation- based QA (GenQA) models, which generate an- swers using the top-k answer sentences ranked by AS2 models (a la retrieval-augmented gen- eration style). In this paper, we propose a novel training paradigm for GenQA using supervi- sion from automatic QA evaluation models (GAVA). Specifically, we propose three strate- gies to transfer knowledge from these QA eval- uation models to a GenQA model: (i) augment- ing training data with answers generated by the GenQA model and labelled by GAVA (either statically, before training, or (ii) dynamically, at every training epoch); and (iii) using the GAVA score for weighting the generator loss during the learning of the GenQA model. We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art. 1 Introduction Recent research on retrieval-based Question An- swering (QA) systems has been focused on two main tasks: (i) Answer Sentence Selection (AS2) e.g., (Garg et al., 2020), which, given a question and a list of answer candidates, chooses the most relevant answer that correctly answers the question; and (ii) Machine Reading (MR) e.g., (Chen et al., 2017), which, given a question and a reference text, involves finding a text span that directly answers the question. While effective, both the strategies (AS2 and MR) have some limitations: (i) the text might not include all the information necessary to answer a question, (ii) the text might include un- necessary, distracting information, or (iii) the text expresses the answer in a convoluted (indirect) for- mat. Additionally, the text style and sentiment may ∗Work done as an intern at Amazon Alexa AI †Work completed at Amazon Alexa AI be inappropriate for answering, or might be struc- turally too dependent on longer discourse context to enable usage as a stand-alone answer. These drawbacks have motivated researchers to explore text generation systems for writing ‘better’ answers in the open-domain abstractive QA set- ting. For example, in the MR domain, RAG (Lewis et al., 2020b) generates an answer from a set of documents which are selected by dense passage retrieval models. For the domain of AS2, previous research has focused on summarizing answers from relevant paragraphs/evidences (Lewis et al., 2020a), or synthesizing information from the top ranked answer candidates of an AS2 system (Hsu et al., 2021; Muller et al., 2022; Gabburo et al., 2022). The latter, termed as GenQA, has shown im- provements in answer generation from the perspec- tive of both answering accuracy and style suitabil- ity. The main distinguishing feature of …

Content truncated to 500 words