Posing Fair Generalization Tasks for Natural Language Inference

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4485–4495, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4485 Posing Fair Generalization Tasks for Natural Language Inference Atticus Geiger Stanford Symbolic Systems Program atticusg@stanford.edu Lauri Karttunen Stanford Linguistics laurik@stanford.edu Ignacio Cases Stanford Linguistics cases@stanford.edu Christopher Potts Stanford Linguistics cgpotts@stanford.edu Abstract Deep learning models for semantics are gener- ally evaluated using naturalistic corpora. Ad- versarial methods, in which models are eval- uated on new examples with known semantic properties, have begun to reveal that good per- formance at these naturalistic tasks can hide serious shortcomings. However, we should in- sist that these evaluations be fair – that the models are given data sufficient to support the requisite kinds of generalization. In this pa- per, we define and motivate a formal notion of fairness in this sense. We then apply these ideas to natural language inference by con- structing very challenging but provably fair ar- tificial datasets and showing that standard neu- ral models fail to generalize in the required ways; only task-specific models that jointly compose the premise and hypothesis are able to achieve high performance, and even these models do not solve the task perfectly. 1 Introduction Evaluations of deep learning approaches to seman- tics generally rely on corpora of naturalistic exam- ples, with quantitative metrics serving as a proxy for the underlying capacity of the models to learn rich meaning representations and find generalized solutions. From this perspective, when a model achieves human-level performance on a task ac- cording to a chosen metric, one might be tempted to say that the task is “solved”. However, recent adversarial testing methods, in which models are evaluated on new examples with known semantic properties, have begun to reveal that even these state-of-the-art models often rely on brittle, local solutions that fail to generalize even to examples that are similar to those they saw in training. These findings indicate that we need a broad and deep range of evaluation methods to fully characterize the capacities of our models. However, for any evaluation method, we should ask whether it is fair. Has the model been shown data sufficient to support the kind of generaliza- tion we are asking of it? Unless we can say “yes” with complete certainty, we can’t be sure whether a failed evaluation traces to a model limitation or a data limitation that no model could overcome. In this paper, we seek to address this issue by defining a formal notion of fairness for these eval- uations. The definition is quite general and can be used to create fair evaluations for a wide range of tasks. We apply it to Natural Language Infer- ence (NLI) by constructing very challenging but provably fair artificial datasets. We evaluate a number of different standard architectures (vari- ants of LSTM sequence models with attention and tree-structured neural networks) as well as NLI- specific tree-structured neural networks that pro- …

Content truncated to 500 words