Tackling Adversarial Examples in QA via Answer Sentence Selection

Proceedings of the Workshop on Machine Reading for Question Answering, pages 31–36 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 31 Tackling Adversarial Examples in QA via Answer Sentence Selection Yuanhang Ren1, Ye Du2, Di Wang2,3 1University of Electronic Science and Technology of China 2Southwestern University of Finance and Economics, China 3Vector Lab, JD Finance, Beijing, China {ryuanhang,henry.duye,albertwang0921}@gmail.com Abstract Question answering systems deteriorate dramatically in the presence of adversar- ial sentences in articles. According to Jia and Liang (2017), the single BiDAF sys- tem (Seo et al., 2016) only achieves an F1 score of 4.8 on the ADDANY adver- sarial dataset. In this paper, we present a method to tackle this problem via answer sentence selection. Given a paragraph of an article and a corresponding query, in- stead of directly feeding the whole para- graph to the single BiDAF system, a sen- tence that most likely contains the answer to the query is first selected, which is done via a deep neural network based on Tree- LSTM (Tai et al., 2015). Experiments on ADDANY adversarial dataset validate the effectiveness of our method. The F1 score has been improved to 52.3. 1 Introduction Question answering is an important task in evalu- ating the ability of language understanding of ma- chines. Usually, given a paragraph and a corre- sponding question, a question answering system is supposed to generate the answer of this ques- tion from the paragraph. By comparing the pre- dicted answer with human-approved answers, the performance of the system can be assessed. Re- cently, many systems have achieved great results on this task (Shen et al., 2017b; Wang and Jiang, 2016; Hu et al., 2017). However, Jia and Liang (2017) show that these systems are very vulnera- ble to paragraphs with adversarial sentences. For instance, the single BiDAF system (Seo et al., 2016), which achieves an F1 of 75.5 on Standford Question Answering Dataset (SQuAD), deterio- rates significantly to an F1 of 4.8 on the ADDANY adversarial dataset. Besides the single BiDAF, the single Match LSTM, the ensemble Match LSTM, and the ensemble BiDAF achieve an F1 of 7.6, 11.7, and 2.7 respectively in question answering on ADDANY adversarial dataset (Jia and Liang, 2017). Therefore, question answering with adver- sarial sentences in paragraphs is a prominent issue and is the focus of this study. In this paper, we propose a method to improve the performance of the single BiDAF system1 on ADDANY adversarial dataset. Given a paragraph and a corresponding question, our method works in two steps to generate an answer. In the first step, a deep neural network named the QA Likelihood neural network is deployed to predict the likeli- hood of each sentence in the paragraph to be an answer sentence, i.e., the sentence that contains the answer. The architecture and the loss of the QA Likelihood neural network follow the neural network for semantic relatedness proposed by Tai et al. (2015). Its main ingredient is the Tree-LSTM model. While the neural network for semantic re- …

Content truncated to 500 words