Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2686–2694 August 1–6, 2021. ©2021 Association for Computational Linguistics 2686 Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model Jeonghyeok Park and Hai Zhao∗ Department of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 117033990011@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn Abstract This work empirically explores effective ex- ploiting of intermediate output from pre- trained language models (PrLMs) for language generation tasks. For this purpose, we propose an improved method to integrate public check- points of PrLMs for the most convenience and perform extensive experiments on 6 differ- ent kinds of PrLMs, including BERT, ELEC- TRA, GPT2, Multi-lingual BERT, and XLM RoBERTa. Evaluation with automatic met- rics shows that our approach significantly im- proves the generation quality on the generation tasks, up to 1.8 BLEU points for neural ma- chine translation (Korean-to-English, Korean- to-Chinese) and 1.8 ROUGE points improve- ments for text summarization. 1 Introduction Pre-trained Language Models (PrLMs), such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and ELECTRA (Clark et al., 2020), have thoroughly changed the landscape of state-of-the- art performance on many Natural Language Un- derstanding (NLU) tasks. Also, publicly released checkpoints of the PrLMs allow natural language processing (NLP) researchers to gain SOTA results while saving vast compute and time resources. The widely used method to exploit PrLM is fine-tuning. However, for Natural Language Generation (NLG) tasks, such methods do not get as much perfor- mance gain as in the NLU task. Several previ- ous studies proposed methods that better use prior knowledge of the PrLM for NLG tasks (Yang et al., 2020; Zhu et al., 2020; Chen et al., 2020). Expend- ing the previous studies, in this paper, we propose ∗ Corresponding author. This paper was partially sup- ported by National Key Research and Development Program of China (No. 2017YFB0304100) and Key Projects of Na- tional Natural Science Foundation of China (U1836222 and 61733011). This work was supported by Huawei Noah’s Ark Lab. an improved method that exploits the checkpoint of the PrLM into the Transformer models (Vaswani et al., 2017). The existing methods for leveraging PrLM in NLG tasks can be roughly classified into two cate- gories: Reusing the PrLM as a starting point and In- tegrating the intermediate output of the PrLM. The former, the widely used in various NLP tasks, de- notes to initialize the part of Transformer from the PrLM for generation tasks (Clinchant et al., 2019; Edunov et al., 2019; Rothe et al., 2020) or replace the input embedding with the PrLM. The latter is an approach that first extracts the contextualized representation from a LM for an input sentence and fuses it into a neural model (Yang et al., 2020; Zhu et al., 2020; Chen et al., 2020). As our preliminary experiment shows, we expand this approach and ex- plore …

Content truncated to 500 words