Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726–2736 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2726 Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA Yichen Jiang and Mohit Bansal UNC Chapel Hill {yichenj, mbansal}@cs.unc.edu Abstract Multi-hop question answering requires model to connect multiple pieces of evidencea scattered in a long context to answer the ques- tion. In this paper, we show that in the multi- hop HotpotQA (Yang et al., 2018) dataset, the examples often contain reasoning shortcuts through which models can directly locate the answer by word-matching the question with a sentence in the context. We demonstrate this issue by constructing adversarial documents that create contradicting answers to the short- cut but do not affect the validity of the origi- nal answer. The performance of strong base- line models drops significantly on our adver- sarial evaluation, indicating that they are in- deed exploiting the shortcuts rather than per- forming multi-hop reasoning. After adversar- ial training, the baseline’s performance im- proves but is still limited on the adversarial evaluation. Hence, we use a control unit that dynamically attends to the question at differ- ent reasoning hops to guide the model’s multi- hop reasoning. We show that this 2-hop model trained on the regular data is more robust to the adversaries than the baseline model. Af- ter adversarial training, this 2-hop model not only achieves improvements over its counter- part trained on regular data, but also outper- forms the adversarially-trained 1-hop baseline. We hope that these insights and initial im- provements will motivate the development of new models that combine explicit composi- tional reasoning with adversarial training.1 1 Introduction The task of question answering (QA) requires the model to answer a natural language question by finding relevant information in a given natural lan- guage context. Most QA datasets require single- hop reasoning only, which means that the evidence 1Our code and data are publicly available at: https://github.com/jiangycTarheel/ Adversarial-MultiHopQA What was the father of Kasper Schmeichel voted to be by the IFFHS in 1992? R. Bolesław Kelly MBE (] ; born 18 November 1963) is a Danish former professional footballer who played as a Defender, and was voted the IFFHS World's Best Defender in 1992 and 1993. Kasper Peter Schmeichel (] ; born 5 November 1986) is a Danish professional footballer who plays as a goalkeeper ... . He is the son of former Manchester United and Danish international goalkeeper Peter Schmeichel. Edson Arantes do Nascimento (] ; born 23 October 1940), known as Pelé (] ), is a retired Brazilian professional footballer who played as a forward. In 1999, he was voted World Player of the Century by IFFHS. Peter Bolesław Schmeichel MBE (] ; born 18 November 1963) is a Danish former professional footballer who played as a goalkeeper, and was voted the IFFHS World's Best Goalkeeper in 1992 and 1993. Kasper Hvidt (born 6 February 1976 in Copenhagen) is a Danish retired handball …

Content truncated to 500 words