Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast—Choose Three

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5583–5595, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5583 Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast—Choose Three Steven Reich1 David Mueller2,3 Nicholas Andrews3 1Department of Mathematics, University of Maryland 2Center for Language and Speech Processing, Johns Hopkins University 3Human Language Technology Center of Excellence, Johns Hopkins University sreich47@umd.edu {dam,noa}@jhu.edu Abstract Modern neural networks do not always pro- duce well-calibrated predictions, even when trained with a proper scoring function such as cross-entropy. In classification settings, sim- ple methods such as isotonic regression or temperature scaling may be used in conjunc- tion with a held-out dataset to calibrate model outputs. However, extending these methods to structured prediction is not always straight- forward or effective; furthermore, a held-out calibration set may not always be available. In this paper, we study ensemble distillation as a general framework for producing well- calibrated structured prediction models while avoiding the prohibitive inference-time cost of ensembles. We validate this framework on two tasks: named-entity recognition and machine translation. We find that, across both tasks, en- semble distillation produces models which re- tain much of, and occasionally improve upon, the performance and calibration benefits of en- sembles, while only requiring a single model during test-time. 1 Introduction For a calibrated model, an event with a forecast confidence p occurs in held-out data with prob- ability p. Calibrated probabilities enable mean- ingful decision making, either by machines such as downstream probabilistic systems (Nguyen and O’Connor, 2015), or by end-users who must inter- pret and trust system outputs (Jiang et al., 2012). The calibration of modern neural models has re- cently received increased attention in both the natural language processing and machine learn- ing communities. A major finding is that mod- ern neural networks do not always produce well- calibrated predictions. As a result, much re- cent work has focused on improving model cal- ibration, predominantly with post-hoc calibration methods (Guo et al., 2017). However, post-hoc calibration methods have primarily been developed in the context of classi- fication tasks. Thus, it is unclear how these meth- ods will affect the performance of sequence-level structured prediction tasks (Kumar and Sarawagi, 2019). Additionally, post-hoc calibration meth- ods require a held out calibration dataset, which may not be available in all circumstances. To im- prove calibration, an alternate approach is model ensembling, which is closely related to approxi- mating the intractable posterior distribution over model parameters (Lakshminarayanan et al., 2017; Pearce et al., 2018; Dusenberry et al., 2020). Al- though computationally expensive, both at train- ing and inference time, ensembling does not re- quire a separate calibration set. Furthermore, en- sembles have been found to be competitive or even outperform other calibration methods, partic- ularly in more challenging settings such as dataset shift (Snoek et al., 2019). In this paper, we study ensemble distillation as a means of achieving calibrated and accurate struc- tured models while avoiding the high cost of naive ensembles at inference time (Hinton et al., …

Content truncated to 500 words