Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5642–5650 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5642 Logic-Guided Data Augmentation and Regularization for Consistent Question Answering Akari Asai† and Hannaneh Hajishirzi†‡ †University of Washington ‡Allen Institute for AI {akari, hannaneh}@cs.washington.edu Abstract Many natural language questions require qual- itative, quantitative or logical comparisons be- tween two entities or events. This paper ad- dresses the problem of improving the accuracy and consistency of responses to comparison questions by integrating logic rules and neu- ral models. Our method leverages logical and linguistic knowledge to augment labeled train- ing data and then uses a consistency-based regularizer to train the model. Improving the global consistency of predictions, our ap- proach achieves large improvements over pre- vious methods in a variety of question answer- ing (QA) tasks including multiple-choice qual- itative reasoning, cause-effect reasoning, and extractive machine reading comprehension. In particular, our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets. We advance state of the art by around 5-8% on WIQA and QuaRel and reduce consistency violations by 58% on HotpotQA. We further demonstrate that our approach can learn effectively from limited data.1 1 Introduction Comparison-type questions (Tandon et al., 2019; Tafjord et al., 2019; Yang et al., 2018) ask about re- lationships between properties of entities or events such as cause-effect, qualitative or quantitative rea- soning. To create comparison questions that re- quire inferential knowledge and reasoning ability, annotators need to understand context presented in multiple paragraphs or carefully ground a ques- tion to the given situation. This makes it chal- lenging to annotate a large number of comparison questions. Most current datasets on comparison questions are much smaller than standard machine reading comprehension (MRC) datasets (Rajpurkar 1Our code and data is available at https://github. com/AkariAsai/logic_guided_qa. Q: The ceramic vase was less flexible than the plastic ball so it was A: more breakable Q: The ceramic vase was more flexible than the plastic ball so it was A: less breakable Q: If it is silent, does the outer ear collect less sound waves? A: more [positive causal relationship] Q: If the outer ear collect less sound waves, is less sound being detected? A: more [positive causal relationship] Q: If it is silent, is less sound being detected? A: more [positive causal relationship] RoBERTa more breakable more breakable RoBERTa more more less Conflict Conflict Figure 1: Inconsistent predictions by RoBERTa. Top row shows an example of symmetric inconsistency and the second row shows an example of transitive incon- sistency. The examples are partially modified. et al., 2016; Joshi et al., 2017). This poses new challenges to standard models, which are known to exploit statistical patterns or annotation artifacts in these datasets (Sugawara et al., 2018; Min et al., 2019a). Importantly, state-of-the-art models show inconsistent comparison predictions as shown in Figure 1. Improving the consistency of predictions has been previously studied in natural language in- ference (NLI) tasks (Minervini and Riedel, 2018; Li et al., …

Content truncated to 500 words