Distilling Knowledge Learned in BERT for Text Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7893–7905 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7893 Distilling Knowledge Learned in BERT for Text Generation Yen-Chun Chen1, Zhe Gan1, Yu Cheng1, Jingzhou Liu2, Jingjing Liu1 1Microsoft Dynamics 365 AI Research 2Carnegie Mellon University {yen-chun.chen,zhe.gan,yu.cheng,jinjl}@microsoft.com; liujingzhou@cs.cmu.edu Abstract Large-scale pre-trained language model such as BERT has achieved great success in lan- guage understanding tasks. However, it re- mains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text gen- eration performance. By leveraging BERT’s idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as ma- chine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English- Vietnamese MT datasets.1 1 Introduction Large-scale pre-trained language model, such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), has become the de facto first encoding step for many natural language processing (NLP) tasks. For example, BERT, pre-trained with deep bidirectional Trans- former (Vaswani et al., 2017) via masked language modeling and next sentence prediction, has revo- lutionized the state of the art in many language understanding tasks, such as natural language infer- ence (Bowman et al., 2015) and question answer- ing (Rajpurkar et al., 2016). 1Code is available at https://github.com/ChenRocks/Distill- BERT-Textgen. However, beyond common practice of finetun- ing BERT for language understanding (Wang et al., 2019), applying BERT to language generation still remains an open question. Text generation aims to generate natural language sentences conditioned on certain input, with applications ranging from machine translation (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015), text sum- marization (Nallapati et al., 2016; Gehring et al., 2017; Chen and Bansal, 2018), to image caption- ing (Vinyals et al., 2015; Xu et al., 2015; Gan et al., 2017). In this work, we study how to use BERT for better text generation, which is still a relatively unexplored territory. Intuitively, as BERT is learned with a generative objective via Masked Language Modeling (MLM) during the pre-training stage, a natural assumption is that this training objective should have learned essential, bidirectional, contextual knowledge that can help enhance text generation. Unfortunately, this MLM objective is not auto-regressive, which encumbers its direct application to auto-regressive text generation in practice. We tackle this challenge by proposing a novel and generalizable approach to distilling knowledge learned in BERT for text generation tasks. We first propose a new Conditional Masked Language Modeling (C-MLM) task, inspired by MLM but re- quiring additional conditional input, which enables …

Content truncated to 500 words