Self-Knowledge Distillation in Natural Language Processing

Proceedings of Recent Advances in Natural Language Processing, pages 423–430, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_050 423 Self-Knowledge Distillation in Natural Language Processing Sangchul Hahn Handong Global University Pohang, South Korea schahn21@gmail.com Heeyoul Choi Handong Global University Pohang, South Korea heeyoul@gmail.com Abstract Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learn- ing models. While many methods have been proposed to learn more efficient repre- sentation, knowledge distillation from pre- trained deep networks suggest that we can use more information from the soft target probability to train other neural networks. In this paper, we propose a new knowledge distillation method self-knowledge distilla- tion, based on the soft target probabilities of the training model itself, where mul- timode information is distilled from the word embedding space right below the soft- max layer. Due to the time complexity, our method approximates the soft target probabilities. In experiments, we applied the proposed method to two different and fundamental NLP tasks: language model and neural machine translation. The ex- periment results show that our proposed method improves performance on the tasks. 1 Introduction Deep learning has achieved the state-of-the-art per- formance on many machine learning tasks, such as image classification, object recognition, and neural machine translation (He et al., 2016; Redmon and Farhadi, 2017; Vaswani et al., 2017) and outper- formed humans on some tasks. In deep learning, one of the critical points for success is to learn bet- ter representation of data with many layers (Ben- gio et al., 2013) than other machine learning algo- rithms. In other words, if we make a model to learn better representation of data, the model can show better performance. In natural language processing (NLP) tasks like language modeling (LM) (Bengio et al., 2003; Mikolov et al., 2013) and neural machine trans- lation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015), when the models are trained, they are to generate many words in sentence, which is a sequence of classification steps, for each of which they choose a target word among the whole words in the dictionary. That is why LM and NMT are usually trained with the sum of cross-entropies over the target sentence. Thus, although language re- lated tasks are more of generation rather than clas- sification, the models estimate target probabilities with the softmax operation on the previous neu- ral network layers and the target distributions are provided as one-hot representations. As data rep- resentation in NLP models, word symbols should also be represented as vectors. In this paper, we focus on the word embedding and the estimation of the target distribution. In NLP, word embedding is a step to translate word symbols (indices in the vocabulary) to vectors in a contin- uous vector space and is considered as a standard approach to handle …

Content truncated to 500 words