A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selection

Proceedings of the 28th International Conference on Computational Linguistics, pages 5211–5222 Barcelona, Spain (Online), December 8-13, 2020 5211 A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selection Daniele Bonadiman∗ Amazon AI dbonadim@amazon.com Alessandro Moschitti Amazon Alexa amosch@amazon.com Abstract An essential task of most Question Answering (QA) systems is to re-rank the set of answer candidates, i.e., Answer Sentence Selection (AS2). These candidates are typically sentences either extracted from one or more documents preserving their natural order or retrieved by a search engine. Most state-of-the-art approaches to the task use huge neural models, such as BERT, or complex attentive architectures. In this paper, we argue that by exploiting the intrinsic structure of the original rank together with an effective word-relatedness encoder, we achieve the highest accuracy among the cost-efficient models, with two orders of magnitude fewer parameters than the current state of the art. Our model takes 9.5 seconds to train on the WikiQA dataset, i.e., very fast in comparison with the ∼ 18 minutes required by a standard BERT-base fine-tuning. 1 Introduction In recent years, there has been a renewed interest in Question Answering (QA) led by industrial needs, e.g., the development of personal assistants and academic research on neural networks. Regarding the latter, AS2 (Wang et al., 2007; Yang et al., 2015) and Machine Reading Comprehension (MRC) (Richardson et al., 2013; Rajpurkar et al., 2016) have been largely explored. AS2 consists of selecting sentences that are answers to a target question from documents or paragraphs retrieved from the web by a search engine. MRC regards the extraction of an exact text span from a document answering the question, where the document is usually provided with the target question. Even though MRC is gaining more and more popularity, AS2 is more suitable for a production scenario, where a combination of a retrieval engine and an automatic sentence selector can already constitute a QA system. In contrast, MRC has been mainly developed to find answers in a paragraph or a text of limited size. Several models for adapting MRC to an end-to-end retrieval setting have been proposed, e.g., Chen et al. (2017a) and Kratzwald and Feuerriegel (2018), the deployment of MRC systems in production is challenged by two key factors: the lack of datasets for training MRC with realistic retrieval data, and the large volume of content needed to be processed, i.e., MRC cannot efficiently process a large amount of retrieved data. In contrast, AS2 research originated from the TREC competitions (Wang et al., 2007); thus, it has targeted large databases of unstructured text from the beginning of its development. Neural models have significantly contributed to AS2 with new techniques, e.g., Wang and Jiang (2016), Qiao et al. (2019) and Nogueira and Cho (2019). Recently, new approaches for pre-training neural language models on large amount of data, e.g., ELMO (Peters et al., 2018), GPT (Radford et al., ), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), have led to major advancements in several NLP subfields. These …

Content truncated to 500 words