How Certain is Your Transformer?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1833–1840 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1833 How Certain is Your Transformer? Artem Shelmanov‡, Evgenii Tsymbalov†,‡, Dmitri Puzyrev‡,⋆, Kirill Fedyanin‡, Alexander Panchenko‡, and Maxim Panov‡ ‡Skolkovo Institute of Science and Technology, Moscow, Russia †Huawei Industry Video Application Lab, Moscow, Russia ⋆Mobile TeleSystems (MTS), Moscow, Russia {a.shelmanov,evgenii.tsymbalov,dmitry.puzyrev,k.fedyanin,a.panchenko,m.panov}@skoltech.ru Abstract In this work, we consider the problem of uncer- tainty estimation for Transformer-based mod- els. We investigate the applicability of uncer- tainty estimates based on dropout usage at the inference stage (Monte Carlo dropout). The series of experiments on natural language un- derstanding tasks shows that the resulting un- certainty estimates improve the quality of de- tection of error-prone instances. Special atten- tion is paid to the construction of computation- ally inexpensive estimates via Monte Carlo dropout and Determinantal Point Processes. 1 Introduction Quantifying the uncertainty of machine learning models is an important aspect of trustworthy, reli- able, and accountable natural language understand- ing (NLU) systems. Obtaining measures of uncer- tainty in predictions (also known as uncertainty es- timations, UE) helps to detect out-of-domain (Ma- linin and Gales, 2018), adversarial, or error-prone instances that require special treatment. For exam- ple, such instances can be additionally checked by human experts or another more advanced system or alternatively rejected from classification (Her- bei and Wegkamp, 2006). Besides, uncertainty estimation is an essential component of various ap- plications such as active learning (Shelmanov et al., 2021) and outlier/error detection in a dataset (Lar- son et al., 2019). Many modern NLU methods take advantage of deep pre-trained models that are based on the Trans- former architecture (Vaswani et al., 2017) (e.g., BERT (Devlin et al., 2019) or ELECTRA (Clark et al., 2020)). Obtaining reliable uncertainty estima- tions for such neural networks (NNs) can, therefore, directly benefit a wide range of NLU tasks, yet im- plementing UEs, in this case, is challenging due to the huge number of parameters in these deep learn- ing models. The approximations of Bayesian infer- ence based on dropout usage at the inference stage – Monte Carlo (MC) dropout (Gal and Ghahramani, 2016), provide a realizable approach to quantifying UEs of deep models. However, they are usually ac- companied by serious computational overhead due to the necessity of performing multiple stochastic predictions. Importantly, training ensembles of in- dependent models (Lakshminarayanan et al., 2017) leads to even more prohibitive overheads. In this work, we investigate various MC dropout- based approaches to uncertainty quantification of NLU models on the widely-used General Lan- guage Understanding Evaluation (GLUE) bench- mark (Wang et al., 2018). The main contributions of our work are two-fold:1 • We show that the use of the MC dropout with pre-trained Transformer models significantly improves the quality of UEs in NLU tasks compared to deterministic baselines. • We are the first to our knowledge to apply a modification of the MC dropout based on determinantal point processes (DPP; Tsym- balov et al. (2020)) …

Content truncated to 500 words