HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 250–255 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 250 HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning Hitomi Yanaka1,2, Koji Mineshima2, Daisuke Bekki2, Kentaro Inui1,3, Satoshi Sekine1, Lasha Abzianidze4, and Johan Bos4 1RIKEN, 2Ochanomizu University, 3Tohoku University, Japan 4University of Groningen, Netherlands {hitomi.yanaka, satoshi.sekine}@riken.jp, mineshima.koji@ocha.ac.jp, bekki@is.ocha.ac.jp, inui@ecei.tohoku.ac.jp, {l.abzianidze, johan.bos}@rug.nl Abstract Large crowdsourced datasets are widely used for training and evaluating neural models on natural language inference (NLI). De- spite these efforts, neural models have a hard time capturing logical inferences, including those licensed by phrase replacements, so- called monotonicity reasoning. Since no large dataset has been developed for monotonicity reasoning, it is still unclear whether the main obstacle is the size of datasets or the model ar- chitectures themselves. To investigate this is- sue, we introduce a new dataset, called HELP, for handling entailments with lexical and logi- cal phenomena. We add it to training data for the state-of-the-art neural models and evaluate them on test sets for monotonicity phenomena. The results showed that our data augmentation improved the overall accuracy. We also find that the improvement is better on monotonic- ity inferences with lexical replacements than on downward inferences with disjunction and modification. This suggests that some types of inferences can be improved by our data aug- mentation while others are immune to it. 1 Introduction Natural language inference (NLI) has been pro- posed as a benchmark task for natural language understanding. This task is to determine whether a given statement (premise) semantically entails an- other statement (hypothesis) (Dagan et al., 2013). Large crowdsourced datasets such as SNLI (Bow- man et al., 2015a) and MultiNLI (Williams et al., 2018) have been created from naturally-occurring texts for training and testing neural models on NLI. Recent reports showed that these crowd- sourced datasets contain undesired biases that al- low prediction of entailment labels only from hy- pothesis sentences (Gururangan et al., 2018; Po- liak et al., 2018b; Tsuchiya, 2018). Moreover, these standard datasets come with the so-called Upward Some changes in personal values are simply part of growing older (MultiNLI) ⇒ Some changes in values are a part of growing older Downward At most ten commissioners spend time at home (FraCaS) ⇒ At most ten female commissioners spend time at home Table 1: Upward and downward inferences. upward monotonicity inferences (see Table 1), i.e., inferences from subsets to supersets (changes in personal values ⊑ changes in values), but they rarely come with downward monotonicity infer- ences, i.e., inferences from supersets to subsets (commissioners ⊒ female commissioners). Down- ward monotonicity inferences are interesting in that they allow to replace a phrase with a more specific one and thus the resulting sentence can be- come longer, yet the inference is valid. FraCaS (Cooper et al., 1994) contains such log- ically challenging problems as downward infer- ences. However, it is small in size (only 346 exam- ples) for training neural models, and it covers only …

Content truncated to 500 words