Ranking and Sampling in Open-Domain Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2412–2421, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2412 Ranking and Sampling in Open-Domain Question Answering Yanfu Xu1,2, Zheng Lin1∗, Yuanxin Liu1,2, Rui Liu1,2, Weiping Wang1 and Dan Meng1 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {xuyanfu,linzheng,liuyuanxin,liurui1995,wangweiping,mengdan}@iie.ac.cn Abstract Open-domain question answering (OpenQA) aims to answer questions based on a num- ber of unlabeled paragraphs. Existing ap- proaches always follow the distantly super- vised setup where some of the paragraphs are wrong-labeled (noisy), and mainly utilize the paragraph-question relevance to denoise. However, the paragraph-paragraph relevance, which may aggregate the evidence among rel- evant paragraphs, can also be utilized to dis- cover more useful paragraphs. Moreover, cur- rent approaches mainly focus on the posi- tive paragraphs which are known to contain the answer during training. This will affect the generalization ability of the model and make it be disturbed by the similar but ir- relevant (distracting) paragraphs during test- ing. In this paper, we first introduce a ranking model leveraging the paragraph-question and the paragraph-paragraph relevance to compute a confidence score for each paragraph. Fur- thermore, based on the scores, we design a modified weighted sampling strategy for train- ing to mitigate the influence of the noisy and distracting paragraphs. Experiments on three public datasets (Quasar-T, SearchQA and Triv- iaQA) show that our model advances the state of the art. 1 Introduction Different from the traditional reading comprehen- sion (RC) task, which aims to answer a question based on an off-the-peg paragraph, the reading paragraphs on OpenQA datasets are always col- lected via an information retrieval (IR) system. For example, given a question as shown in Fig- ure 1, an OpenQA system usually coarsely re- trieves paragraphs that are similar to the question. Therefore, the OpenQA model has to answer a question based on numerous paragraphs. As can ∗Corresponding author Question: Who is Donald Duck’s uncle ? Answer: Scrooge Paragraph1 (positive, relevant): Donald’s maternal un- cle Scrooge McDuck made his first appearance ... Paragraph2 (positive, relevant): This story was reis- sued as a single record entitled “Donald Duck and Uncle Scrooge’s Money Rocket” ... Paragraph3 (positive, noisy): In Scrooge’s stories, Don- ald Duck was cast as Scrooge’s assistant ... Paragraph4 (negative, distracting): Donald Duck is the uncle of Huey, Dewey, and Louie. Figure 1: An example of OpenQA. The key infor- mation, answers in correct-labeled and wrong-labeled contexts are marked in blue, green and red respectively. be seen in Figure 1, some of the negative para- graphs are similar to the question whereas the an- swer string is missed, e.g. Paragraph4. We con- sider these paragraphs as distracting ones. Be- sides, in the distantly supervised setup (Mintz et al., 2009), it is postulated that the paragraphs that contain the answer string are ground truths, while even some of the positive …

Content truncated to 500 words