Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 65–74 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 65 Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge Pasquale Minervini University College London p.minervini@cs.ucl.ac.uk Sebastian Riedel University College London s.riedel@cs.ucl.ac.uk Abstract Adversarial examples are inputs to machine learning models designed to cause the model to make a mistake. They are useful for under- standing the shortcomings of machine learn- ing models, interpreting their results, and for regularisation. In NLP, however, most ex- ample generation strategies produce input text by using known, pre-specified semantic trans- formations, requiring significant manual ef- fort and in-depth understanding of the prob- lem and domain. In this paper, we investi- gate the problem of automatically generating adversarial examples that violate a set of given First-Order Logic constraints in Natural Lan- guage Inference (NLI). We reduce the prob- lem of identifying such adversarial examples to a combinatorial optimisation problem, by maximising a quantity measuring the degree of violation of such constraints and by using a language model for generating linguistically- plausible examples. Furthermore, we propose a method for adversarially regularising neu- ral NLI models for incorporating background knowledge. Our results show that, while the proposed method does not always improve results on the SNLI and MultiNLI datasets, it significantly and consistently increases the predictive accuracy on adversarially-crafted datasets – up to a 79.6% relative improve- ment – while drastically reducing the num- ber of background knowledge violations. Fur- thermore, we show that adversarial examples transfer among model architectures, and that the proposed adversarial training procedure improves the robustness of NLI models to ad- versarial examples. 1 Introduction An open problem in Artificial Intelligence is quan- tifying the extent to which algorithms exhibit in- telligent behaviour (Levesque, 2014). In Machine Learning, a standard procedure consists in esti- mating the generalisation error, i.e. the predic- tion error over an independent test sample (Hastie et al., 2001). However, machine learning models can succeed simply by recognising patterns that happen to be predictive on instances in the test sample, while ignoring deeper phenomena (Rimell and Clark, 2009; Paperno et al., 2016). Adversarial examples are inputs to machine learning models designed to cause the model to make a mistake (Szegedy et al., 2014; Goodfel- low et al., 2014). In Natural Language Processing (NLP) and Machine Reading, generating adversar- ial examples can be really useful for understanding the shortcomings of NLP models (Jia and Liang, 2017; Kannan and Vinyals, 2017) and for regular- isation (Minervini et al., 2017). In this paper, we focus on the problem of gener- ating adversarial examples for Natural Language Inference (NLI) models in order to gain insights about the inner workings of such systems, and reg- ularising them. NLI, also referred to as Recog- nising Textual Entailment (Fyodorov et al., 2000; Condoravdi et al., 2003; Dagan et al., 2005), is a central problem in language understanding (Katz, 1972; Bos and Markert, 2005; van Benthem, 2008; MacCartney and Manning, …

Content truncated to 500 words