Deep Model Compression Also Helps Models Capture Ambiguity

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6893–6905 July 9-14, 2023 ©2023 Association for Computational Linguistics Deep Model Compression Also Helps Models Capture Ambiguity Hancheol Park Jong C. Park School of Computing Korea Advanced Institute of Science and Technology {hancheol.park,jongpark}@kaist.ac.kr Abstract Natural language understanding (NLU) tasks face a non-trivial amount of ambiguous sam- ples where veracity of their labels is debatable among annotators. NLU models should thus ac- count for such ambiguity, but they approximate the human opinion distributions quite poorly and tend to produce over-confident predictions. To address this problem, we must consider how to exactly capture the degree of relationship between each sample and its candidate classes. In this work, we propose a novel method with deep model compression and show how such relationship can be accounted for. We see that more reasonably represented relationships can be discovered in the lower layers and that vali- dation accuracies are converging at these layers, which naturally leads to layer pruning. We also see that distilling the relationship knowledge from a lower layer helps models produce better distribution. Experimental results demonstrate that our method makes substantial improve- ment on quantifying ambiguity without gold distribution labels. As positive side-effects, our method is found to reduce the model size sig- nificantly and improve latency, both attractive aspects of NLU products.1 1 Introduction Datasets constructed for natural language under- standing (NLU) tasks, such as natural language inference (NLI) and text emotion analysis, contain a large amount of ambiguous samples (Nie et al., 2020; Uma et al., 2021). As exemplified in Ta- ble 1, each ambiguous sample is too debatable to be assigned a single gold label. Recent work has revealed that these disagreements among annota- tors are not annotation noise, which could have simply been resolved by aggregating more anno- tations, but rather a reproducible signal (Pavlick and Kwiatkowski, 2019; Nie et al., 2020). This 1Code is available at https://github.com/hancheolp/ compression_for_capturing_ambiguity. Premise It’s summer time and two girls play with bubbles near a boat dock. Hypothesis It is warm outside. Label distribution Entailment: 0.52 Neutral: 0.46 Contradiction: 0.02 News headline Amateur rocket scientists reach for space. Label distribution Joy: 0.57 Surprise: 0.43 Anger / Disgust / Fear / Sadness: 0.00 Table 1: Ambiguous samples from datasets for NLI (ChaosSNLI (Nie et al., 2020)) and emotion analysis (SemEval-2007 Task 14 dataset (Strapparava and Mihal- cea, 2007)) suggests that NLU models should predict not only majority labels, but also label distributions that re- spect such ambiguity. Since Transformer-based (Vaswani et al., 2017) pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019) have become popular for NLU tasks, the accuracies of various NLU models have been substantially improved. Nevertheless, they are still not good at approximating the human opinion distributions (Pavlick and Kwiatkowski, 2019; Nie et al., 2020), or label distributions drawn from a larger number of annotators, and their pre- dictions tend to be over-confident (Zhang et al., 2021). If NLU products frequently …

Content truncated to 500 words