Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning

Findings of the Association for Computational Linguistics: ACL 2022, pages 972 - 987 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning Xiang Zhou∗ Yixin Nie∗ Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill {xzh, yixin1, mbansal}@cs.unc.edu Abstract We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language in- ference. We show that by applying addi- tional distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation, models can capture human judgement distribu- tion more effectively than the softmax baseline. We show that MC Dropout is able to achieve decent performance without any distribution annotations while Re-Calibration can give fur- ther improvements with extra distribution an- notations, suggesting the value of multiple an- notations for one example in modeling the dis- tribution of human judgements. Despite these improvements, the best results are still far be- low the estimated human upper-bound, indicat- ing that predicting the distribution of human judgements is still an open, challenging prob- lem with a large room for improvements. We showcase the common errors for MC Dropout and Re-Calibration. Finally, we give guide- lines on the usage of these methods with dif- ferent levels of data availability and encourage future work on modeling the human opinion distribution for language reasoning.1 1 Introduction Natural Language Understanding (NLU) and Rea- soning play a fundamental role in Natural Lan- guage Processing (NLP) research. It has almost be- come the de facto rule that newly proposed generic language models will be tested on NLU tasks and progress obtained on general NLU often bring po- tential improvement on other aspects of NLP re- search (Wang et al., 2019). The well-known NLU tasks include Sentiment Analysis (Socher et al., 2013), Natural Language Inference (NLI) (Bow- man et al., 2015; Nie et al., 2020a), Commonsense ∗ Equal contribution. 1Our code and data are publicly available at https:// github.com/easonnie/ChaosNLI. Reasoning (Talmor et al., 2019), etc., covering a representative set of problems for NLP. One common practice shared by most of the lan- guage understanding and reasoning tasks is that they are formalized as a classification problem, where the model is required to predict a single most preferable label from a predefined candidate set, and the goal is to reverse-engineer how a reason- able human chooses the best one. This simplifica- tion not only helps standardize the evaluation, i.e., accuracy could become the canonical measure, but also help make the annotation task more straight- forward during crowdsourcing data collection. However, recent findings suggest that inher- ent disagreements exist in both the Natural Lan- guage Inference (NLI) and Commonsense Reason- ing datasets (Pavlick and Kwiatkowski, 2019; Chen et al., 2020; Nie et al., 2020b) and advocate that NLU evaluation should explicitly incentivize mod- els to predict distributions of human judgments. Similarly, Gantt et al. (2020) suggest that NLI should account for annotator random effects. This is intuitive …

Content truncated to 500 words