Chinese WPLC: A Chinese Dataset for Evaluating Pretrained Language Models on Word Prediction Given Long-Range Context

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 364–378 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 364 Distilling Linguistic Context for Language Model Compression Geondo Park1 Gyeongman Kim1 Eunho Yang1,2 KAIST1, Daejeon, South Korea AITRICS2, Seoul, South Korea {geondopark, gmkim, eunhoy}@kaist.ac.kr Abstract A computationally expensive and memory in- tensive neural network lies behind the re- cent success of language representation learn- ing. Knowledge distillation, a major technique for deploying such a vast language model in resource-scarce environments, transfers the knowledge on individual word representations learned without restrictions. In this paper, in- spired by the recent observations that language representations are relatively positioned and have more semantic knowledge as a whole, we present a new knowledge distillation objective for language representation learning that trans- fers the contextual knowledge via two types of relationships across representations: Word Re- lation and Layer Transforming Relation. Un- like other recent distillation techniques for the language models, our contextual distillation does not have any restrictions on architectural changes between teacher and student. We vali- date the effectiveness of our method on chal- lenging benchmarks of language understand- ing tasks, not only in architectures of various sizes, but also in combination with DynaBERT, the recently proposed adaptive size pruning method. 1 Introduction Since the Transformer, a simple architecture based on attention mechanism, succeeded in machine translation tasks, Transformer-based models have become a new state of the arts that takes over more complex structures based on recurrent or convo- lution networks on various language tasks, e.g., language understanding and question answering, etc (Devlin et al., 2018; Lan et al., 2019; Liu et al., 2019; Raffel et al., 2019; Yang et al., 2019). How- ever, in exchange for high performance, these mod- els suffer from a major drawback: tremendous com- putational and memory costs. In particular, it is not possible to deploy such large models on platforms with limited resources such as mobile and wearable devices, and it is an urgent research topic with im- pact to keep up with the performance of the latest models from a small-size network. As the main method for this purpose, Knowl- edge Distillation (KD) transfers knowledge from the large and well-performing network (teacher) to a smaller network (student). There have been some efforts that distill Transformer-based mod- els into compact networks (Sanh et al., 2019; Turc et al., 2019; Sun et al., 2019, 2020; Jiao et al., 2019; Wang et al., 2020). However, they all build on the idea that each word representation is independent, ignoring relationships between words that could be more informative than individual representations. In this paper, we pay attention to the fact that word representations from language mod- els are very structured and capture certain types of semantic and syntactic relationships. - Word2Vec (Mikolov et al., 2013) and Glove (Pen- nington et al., 2014) demonstrated that trained em- bedding of words contains the linguistic patterns as linear relationships between word vectors. Re- cently, Reif et al. (2019) found out that the dis- …

Content truncated to 500 words