Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11806 - 11816 December 7-11, 2022 ©2022 Association for Computational Linguistics Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection Luca Di Liello1∗, Siddhant Garg2, Luca Soldaini3†, Alessandro Moschitti2 1University of Trento , 2Amazon Alexa AI, 3Allen Institute for AI luca.diliello@unitn.it {sidgarg,amosch}@amazon.com lucas@allenai.org Abstract An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the an- swer to a question from a set of retrieved rele- vant documents. In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level se- mantics within and across documents, to im- prove the performance of transformers for AS2, and mitigate the requirement of large labeled datasets. Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document. Our experiments on three public and one industrial AS2 datasets demonstrate the em- pirical superiority of our pre-trained transform- ers over baseline models such as RoBERTa and ELECTRA for AS2. 1 Introduction Question Answering (QA) finds itself at the core of several commercial applications, for e.g., virtual assistants such as Google Home, Alexa and Siri. Answer Sentence Selection (AS2) is an important task for QA Systems operating on unstructured text such as web documents. When presented with a set of relevant documents for a question (retrieved from a web index), AS2 aims to find the best an- swer sentence for the question. The recent popularity of pre-trained transform- ers (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020), has made them the de-facto approach for most QA tasks, including AS2. Several research works (Garg et al., 2020; Laskar et al., 2020; Lauri- ola and Moschitti, 2021) fine-tune transformers for AS2, by posing it as a sentence-pair task and per- forming inference over the encoded representations of the question and answer candidates. ∗Work done as an intern at Amazon Alexa AI †Work completed at Amazon Alexa AI AS2 is a knowledge-intensive complex reason- ing task, where the answer candidates for a ques- tion can stem from multiple documents, possibly on different topics linked to concepts in the ques- tion. While there have been recent works (Ginzburg et al., 2021; Caciularu et al., 2021) proposing pre- training strategies for obtaining multi-document aware document representations over long input encoders such as the Longformer (Beltagy et al., 2020), there has been limited research (Giorgi et al., 2021) on enhancing sentence-pair representations with paragraph and document level semantics. Furthermore, obtaining high quality human la- beled examples for AS2 is expensive and time con- suming, due to the large number of answer candi- dates to be annotated for each question. Domain- specific AS2 datasets such as WikiQA (Yang et al., 2015) and TREC-QA (Wang et al., 2007) only con- tain a few thousand questions. Garg et al. (2020) show …

Content truncated to 500 words