A Neural Model for Joint Document and Snippet Ranking in Question Answering for Large Document Collections

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3896–3907 August 1–6, 2021. ©2021 Association for Computational Linguistics 3896 A Neural Model for Joint Document and Snippet Ranking in Question Answering for Large Document Collections Dimitris Pappas1,2 and Ion Androutsopoulos1 1Department of Informatics, Athens University of Economics and Business, Greece 1pappasd@aueb.gr,ion@aueb.gr 2Institute for Language and Speech Processing, Research Center ‘Athena’, Greece 2dpappas@athenarc.gr Abstract Question answering (QA) systems for large document collections typically use pipelines that (i) retrieve possibly relevant documents, (ii) re-rank them, (iii) rank paragraphs or other snippets of the top-ranked documents, and (iv) select spans of the top-ranked snippets as ex- act answers. Pipelines are conceptually simple, but errors propagate from one component to the next, without later components being able to revise earlier decisions. We present an ar- chitecture for joint document and snippet rank- ing, the two middle stages, which leverages the intuition that relevant documents have good snippets and good snippets come from rele- vant documents. The architecture is general and can be used with any neural text relevance ranker. We experiment with two main instan- tiations of the architecture, based on POSIT- DRMM (PDRMM) and a BERT-based ranker. Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval, the main goal for QA, with fewer trainable parameters, also remaining competitive in document retrieval. Furthermore, our joint PDRMM-based model is competitive with BERT-based models, de- spite using orders of magnitude fewer param- eters. These claims are also supported by hu- man evaluation on two test batches of BIOASQ. To test our key findings on another dataset, we modified the Natural Questions dataset so that it can also be used for document and snip- pet retrieval. Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset, even though it performs worse than the pipeline in document retrieval. We make our code and the modified Natural Questions dataset publicly available. 1 Introduction Question answering (QA) systems that search large document collections (Voorhees, 2001; Tsatsaro- nis et al., 2015; Chen et al., 2017) typically use pipelines operating at gradually finer text granulari- ties. A fully-fledged pipeline includes components that (i) retrieve possibly relevant documents typi- cally using conventional information retrieval (IR); (ii) re-rank the retrieved documents employing a computationally more expensive document ranker; (iii) rank the passages, sentences, or other ‘snip- pets’ of the top-ranked documents; and (iv) select spans of the top-ranked snippets as ‘exact’ answers. Recently, stages (ii)–(iv) are often pipelined neural models, trained individually (Hui et al., 2017; Pang et al., 2017; Lee et al., 2018; McDonald et al., 2018; Pandey et al., 2019; Mackenzie et al., 2020; Sekuli´c et al., 2020). Although pipelines are conceptually simple, errors propagate from one component to the next (Hosein et al., 2019), without later com- ponents being able to revise earlier decisions. For example, once a document has …

Content truncated to 500 words