Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6105–6117 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6105 Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? Hitomi Yanaka1, Koji Mineshima2, Daisuke Bekki3, and Kentaro Inui4,1 1RIKEN, 2Keio University, 3Ochanomizu University, 4Tohoku University hitomi.yanaka@riken.jp, minesima@abelard.flet.keio.ac.jp, bekki@is.ocha.ac.jp, inui@ecei.tohoku.ac.jp Abstract Despite the success of language models us- ing neural networks, it remains unclear to what extent neural models have the general- ization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbi- trary inferences with generalization on com- position. We consider four aspects of mono- tonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on un- seen combinations of lexical and logical phe- nomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the struc- tures are slightly changed in the test set while retaining all vocabularies and constituents al- ready appearing in the training set. This indi- cates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set. 1 Introduction Natural language inference (NLI), a task whereby a system judges whether given a set of premises P semantically entails a hypothesis H (Dagan et al., 2013; Bowman et al., 2015), is a fundamental task for natural language understanding. As with other NLP tasks, recent studies have shown a remarkable impact of deep neural networks in NLI (Williams et al., 2018; Wang et al., 2019; Devlin et al., 2019). However, it remains unclear to what extent DNN-based models are capable of learning the compositional generalization underly- ing NLI from given labeled training instances. Systematicity of inference (or inferential sys- tematicity) (Fodor and Pylyshyn, 1988; Aydede, 1997) in natural language has been intensively studied in the field of formal semantics. From among the various aspects of inferential system- aticity, in the context of NLI, we focus on mono- tonicity (van Benthem, 1983; Icard and Moss, 2014) and its productivity. Consider the follow- ing premise–hypothesis pairs (1)–(3), which have the target label entailment: (1) P: Some [puppies ↑] ran. H: Some dogs ran. (2) P: No [cats ↓] ran. H: No small cats ran. (3) P: Some [puppies which chased no [cats ↓]] ran. H: Some dogs which chased no small cats ran. As in (1), for example, quantifiers such as some exhibit upward monotone (shown as [... ↑]), and replacing a phrase in an upward-entailing con- text in a sentence with a more general phrase (re- placing puppies in P with dogs as in H) yields a sentence inferable from the original sentence. In contrast, as in (2), quantifiers such as no …

Content truncated to 500 words