EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 349–361 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 349 EQUATE : A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference Abhilasha Ravichander∗, Aakanksha Naik∗, Carolyn Rose, Eduard Hovy Language Technologies Institute, Carnegie Mellon University {aravicha, anaik, cprose, hovy}@cs.cmu.edu Abstract Quantitative reasoning is a higher-order reaso- ning skill that any intelligent natural language understanding system can reasonably be ex- pected to handle. We present EQUATE1 (Eva- luating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on ave- rage, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quan- tities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2%), but has limited verbal reasoning capabilities (-8.1%). We hope our evaluation framework will sup- port the development of models of quantitative reasoning in language understanding. 1 Introduction Numbers play a vital role in our lives. We re- ason with numbers in day-to-day tasks ranging from handling currency to reading news articles to understanding sports results, elections and stock markets. As numbers are used to communicate in- formation accurately, reasoning with them is an essential core competence in understanding natu- ral language (Levinson, 2001; Frank et al., 2008; Dehaene, 2011). A benchmark task in natural lan- guage understanding is natural language inference (NLI)(or recognizing textual entailment (RTE)) (Cooper et al., 1996; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006), wherein a model determines if a natural language hypothesis ∗*The first two authors contributed equally to this work. 1Code and data available at https://github.com/ AbhilashaRavichander/EQUATE. RTE-QUANT P: After the deal closes, Teva will generate sales of about $ 7 billion a year, the company said. H: Teva earns $ 7 billion a year. AWP-NLI P: Each of farmer Cunningham’s 6048 lambs is either black or white and there are 193 white ones. H: 5855 of Farmer Cunningham’s lambs are black. NEWSNLI P: Emmanuel Miller, 16, and Zachary Watson, 17, are charged as adults, police said. H: Two teen suspects charged as adults. REDDITNLI P: Oxfam says richest one percent to own more than rest by 2016. H: Richest 1% To Own More Than Half Worlds Wealth By 2016 Oxfam. Table 1: Examples from evaluation sets in EQUATE can be justifiably inferred from a given premise2. Making such inferences often necessitates reaso- ning about quantities. Consider the following example from Table 1, P: With 99.6% of precincts counted , Dewhurst held 48% of the vote to 30% for Cruz . H: Lt. Gov. David Dewhurst fails to get 50% of primary vote. To conclude the hypothesis is inferable, a model must reason that since 99.6% precincts are coun- ted, even if all remaining precincts …

Content truncated to 500 words