Improving Answer Selection and Answer Triggering using Hard Negatives

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5911–5917, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5911 Improving Answer Selection and Answer Triggering using Hard Negatives Sawan Kumar ∗ Indian Institute of Science, Bangalore sawankumar@iisc.ac.in Shweta Garg † Search Customer Experience, Amazon shwegarg@amazon.com Kartik Mehta India Machine Learning, Amazon kartim@amazon.com Nikhil Rasiwasia India Machine Learning, Amazon rasiwasi@amazon.com Abstract In this paper, we establish the effectiveness of using hard negatives, coupled with a siamese network and a suitable loss function, for the tasks of answer selection and answer trigger- ing. We show that the choice of sampling strategy is key for achieving improved perfor- mance on these tasks. Evaluating on recent an- swer selection datasets - InsuranceQA, SelQA, and an internal QA dataset, we show that using hard negatives with relatively simple model ar- chitectures (bag of words and LSTM-CNN) drives significant performance gains. On In- suranceQA, this strategy alone improves over previously reported results by a minimum of 1.6 points in P@1. Using hard negatives with a Transformer encoder provides a further im- provement of 2.3 points. Further, we propose to use quadruplet loss for answer triggering, with the aim of producing globally meaning- ful similarity scores. We show that quadru- plet loss function coupled with the selection of hard negatives enables bag-of-words mod- els to improve F1 score by 2.3 points over pre- vious baselines, on SelQA answer triggering dataset. Our results provide key insights into answer selection and answer triggering tasks. 1 Introduction Question answering (QA) is an active field of re- search, drawing attention from the natural lan- guage processing (NLP) and information retrieval (IR) community. Selection-based QA is the task of selecting an answer for a given question, from a set of candidate answers. Two tasks have been proposed for selection-based QA. Given a ques- tion and an answer candidate pool, answer selec- tion is the task of ranking valid answers higher than irrelevant answers, where it is assumed that there is at least one valid answer in the candidate ∗Work done during an internship at Amazon †Work done at India Machine Learning, Amazon pool. Answer triggering, defined for cases where the candidate pool may or may not have a valid answer, is the task of finding a valid answer, while being allowed to abstain. Recently released datasets (Feng et al., 2015; Yang et al., 2015; Jurczyk et al., 2016) have led to significant development in end-to-end neu- ral network Selection-based QA. These networks are learned by posing answer selection as ei- ther a ranking problem or a classification prob- lem. When posed as a ranking problem, a com- mon choice is to use a triplet loss defined over a question, a correct answer and a (usually sam- pled) negative answer. Triplet loss penalises rela- tive distances between positive and negative pairs, making the choice of negatives critical. Recent progress in image understanding tasks suggests that learning …

Content truncated to 500 words