Natural Language Inference with Mixed Effects

Natural Language Inference with Mixed Effects William Gantt* Department of Computer Science University of Rochester wgantt@cs.rochester.edu Benjamin Kane* Department of Computer Science University of Rochester bkane2@cs.rochester.edu Aaron Steven White Department of Linguistics University of Rochester aaron.white@rochester.edu Abstract There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a mixed effects model by incorporating an- notator random effects into any existing neu- ral model, improves performance over models that do not incorporate such effects. 1 Introduction A common method for constructing natural lan- guage inference (NLI) datasets is (i) to gener- ate text-hypothesis pairs using some method— commonly, crowd-sourced hypothesis elicitation given a text from some existing resource (Bowman et al., 2015; Williams et al., 2018) or automated text-hypothesis generation (Zhang et al., 2017); (ii) to collect crowd-sourced judgments about infer- ence from the text to the hypothesis; and (iii) to ag- gregate the possibly multiple annotations provided for a single text-hypothesis pair into a single label. This final step follows common practice across an- notation tasks in NLP; but for NLI in particular, there is growing evidence that it is problematic due to disagreement among annotators that is not cap- tured by the probabilistic outputs of standard NLI models (Pavlick and Kwiatkowski, 2019). One way to capture this disagreement would be to directly model the variability in the raw annota- tions. But this approach presents a challenge: it can *Equal contribution. This work is licensed under a Creative Commons Attribution 4.0 International License. License de- tails: http://creativecommons.org/licenses/ by/4.0/. Figure 1: Distribution of [-50, 50] slider ratings (by annotator) for the same 20 NLI pairs in Pavlick and Kwiatkowski’s dataset (batch 1, described in their §3). be difficult to assess how much disagreement arises from disagreement about the interpretation of a text-hypothesis pair and how much is due to biases that annotators bring to the task. Such biases can be extreme. For instance, Figure 1 plots the distri- bution by annotator of [-50, 50] ratings—with -50 clear contradiction and 50 clear entailment—for the same 20 NLI pairs in Pavlick and Kwiatkowski’s dataset. Despite describing responses to the same items, the distributions are quite variable, sug- gesting variability in how annotators approach the task. This difference in approach may be relatively shallow—e.g. given some true label (or distribution thereon), annotators merely differ in their mapping of that value to the response scale—or they may be quite deep—e.g. annotators differ in how they inter- pret the relationship between texts and hypotheses. We investigate both of these possibilities within a mixed effects modeling framework (Gelman and Hill, 2014). The core idea is to incorporate annotator-specific parameters into …

Content truncated to 500 words