Knowledge Distillation for Quality Estimation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5091–5099 August 1–6, 2021. ©2021 Association for Computational Linguistics 5091 Knowledge Distillation for Quality Estimation Amit Gajbhiye,1 Marina Fomicheva,1 Fernando Alva-Manchego,1 Fr´ed´eric Blain,1,2 Abiola Obamuyide,1 Nikolaos Aletras,1 Lucia Specia1,3 1University of Sheffield, 2University of Wolverhampton, 3Imperial College London {a.gajbhiye,m.fomicheva,f.alva,a.obamuyide,n.aletras}@shef.ac.uk f.blain@wlv.ac.uk l.specia@imperial.ac.uk Abstract Quality Estimation (QE) is the task of automat- ically predicting Machine Translation quality in the absence of reference translations, mak- ing it applicable in real-time settings, such as translating online social media conversa- tions. Recent success in QE stems from the use of multilingual pre-trained representations, where very large models lead to impressive re- sults. However, the inference time, disk and memory requirements of such models do not allow for wide usage in the real world. Mod- els trained on distilled pre-trained representa- tions remain prohibitively large for many us- age scenarios. We instead propose to directly transfer knowledge from a strong QE teacher model to a much smaller model with a differ- ent, shallower architecture. We show that this approach, in combination with data augmenta- tion, leads to light-weight QE models that per- form competitively with distilled pre-trained representations with 8x fewer parameters. 1 Introduction Quality Estimation (QE) aims to predict the quality of the output of Machine Translation (MT) systems when no gold-standard translations are available. It can make MT useful in real-world applications by informing end-users on the translation quality. We focus on sentence-level QE, usually formulated as a regression task where quality is required to be predicted on an continuous scale, e.g. 0-100. The high performances achieved in the most re- cent shared task on sentence-level QE (Specia et al., 2020) have been attributed to the use of strong pre- trained language models, namely BERT (Devlin et al., 2018) and its multilingual variants, especially XLM-Roberta (Conneau et al., 2020a). These mod- els have an extremely large number of parameters and, since they are required at training and infer- ence time, they are very disk and RAM-hungry, also making inference slow. This poses challenges for real-time inference, and prohibits deployment on client machines with limited resources. Making models based on pre-trained representa- tions smaller and more usable in practice is an ac- tive area of research. One approach is Knowledge Distillation (KD), aiming to extract knowledge from a top-performing large model (the teacher) into a smaller (in terms of memory print, com- putational power and prediction latency) yet well- performing model (the student) (Hinton et al., 2015; Gou et al., 2020). KD techniques have been used to make BERT and similar models smaller. For exam- ple, DistilBERT (Sanh et al., 2019) and TinyBERT (Jiao et al., 2020) follow the same general archi- tecture as the teacher BERT, but with a reduced number of layers. However, these student models are also based on Transformers and, as such, they still have too large memory and disk footprints. For instance, the number of parameters in the multilin- gual DistilBERT-based TransQuest model for QE (Ranasinghe et al., 2020) is …

Content truncated to 500 words