Exploring Transitivity in Neural NLI Models through Veridicality

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 920–934 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 920 Exploring Transitivity in Neural NLI Models through Veridicality Hitomi Yanaka1, Koji Mineshima2, and Kentaro Inui3,1 1RIKEN, 2Keio University, 3Tohoku University hitomi.yanaka@riken.jp, minesima@abelard.flet.keio.ac.jp, inui@ecei.tohoku.ac.jp Abstract Despite the recent success of deep neural net- works in natural language processing, the ex- tent to which they can demonstrate human-like generalization capacities for natural language understanding remains unclear. We explore this issue in the domain of natural language inference (NLI), focusing on the transitivity of inference relations, a fundamental property for systematically drawing inferences. A model capturing transitivity can compose basic infer- ence patterns and draw new inferences. We introduce an analysis method using synthetic and naturalistic NLI datasets involving clause- embedding verbs to evaluate whether models can perform transitivity inferences composed of veridical inferences and arbitrary inference types. We find that current NLI models do not perform consistently well on transitivity inference tasks, suggesting that they lack the generalization capacity for drawing compos- ite inferences from provided training exam- ples. The data and code for our analysis are publicly available at https://github.com/ verypluming/transitivity. 1 Introduction Deep neural networks (DNNs) have shown impres- sive performance in many natural language pro- cessing tasks. In particular, DNN models pre- trained with large-scale data such as BERT (De- vlin et al., 2019) have achieved high accuracy in various benchmark tasks (Wang et al., 2019a,b), which suggests that they might possess some gen- eralization capacities that are a hallmark of human cognition. However, recent analyses (Talmor and Berant, 2019; Liu et al., 2019; McCoy et al., 2019) have shown that high accuracy on a test set drawn from the same distribution as the training set does not always indicate that the model has obtained the intended ability, so it remains unclear to what ex- Veridical inference Boolean inference A: Jo knows that Ann and Bob left. B: Ann and Bob left. A′: Jo hopes that Ann and Bob left. C: Ann left. Figure 1: Illustration of transitivity inferences (indi- cated by ) composed of two basic inferences, veridi- cal and Boolean. Arrows indicate entailment and ar- rows with a cross ( ) indicate non-entailment. tent DNN models can learn the systematic general- ization in natural language from training instances. Central to human-like generalization capacities is the fact that ability to understand a given sen- tence is related to ability to understand other sen- tences, called systematicity of human cognition in Fodor and Pylyshyn (1988). Thus, if speakers un- derstand the meaning of the sentence Ann loves Bob, they must also understand the meaning of structurally related sentences such as Bob loves Ann. We explore whether DNN models possess this type of generalization capacity in the domain of natural language inference (NLI), which is the task to judge whether a premise entails a hypothe- sis (Dagan et al., 2013; Bowman et al., 2015a). A key property underlying systematicity of drawing inferences …

Content truncated to 500 words