Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6001–6011 August 1–6, 2021. ©2021 Association for Computational Linguistics 6001 Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation Xin Liu1,2 Baosong Yang3 Dayiheng Liu3 Haibo Zhang3 Weihua Luo3 Min Zhang4 Haiying Zhang1,2 Jinsong Su1,2,5∗ 1School of Informatics, Xiamen University 2Institute of Artificial Intelligence, Xiamen University 3Alibaba Group 4Soochow University, China 5Pengcheng Lab, Shenzhen liuxin@stu.xmu.edu.cn {yangbaosong.ybs,liudayiheng.ldyh,zhanhui.zhb,weihua.luowh}@alibaba-inc.com minzhang@suda.edu.cn {zhang2002,jssu}@xmu.edu.cn Abstract A well-known limitation in pretrain-finetune paradigm lies in its inflexibility caused by the one-size-fits-all vocabulary. This poten- tially weakens the effect when applying pre- trained models into natural language genera- tion (NLG) tasks, especially for the subword distributions between upstream and down- stream tasks with significant discrepancy. To- wards approaching this problem, we extend the vanilla pretrain-finetune pipeline with an extra embedding transfer step. Specifically, a plug-and-play embedding generator is intro- duced to produce the representation of any input token, according to pre-trained embed- dings of its morphologically similar ones. Thus, embeddings of mismatch tokens in downstream tasks can also be efficiently ini- tialized. We conduct experiments on a variety of NLG tasks under the pretrain-finetune fash- ion. Experimental results and extensive anal- yses show that the proposed strategy offers us opportunities to feel free to transfer the vocab- ulary, leading to more efficient and better per- formed downstream NLG models. 1 1 Introduction Pretrain-finetune paradigm has been highly suc- cessful on tackling challenging problems in natural language processing, e.g., domain adaptation (Sato et al., 2020; Yao et al., 2020), incremental learn- ing (Khayrallah et al., 2018; Wan et al., 2020), as well as knowledge transferring (Liu et al., 2020b). The rise of large-scale pre-trained language mod- els further attracts increasing attention towards this strategy (Devlin et al., 2019; Edunov et al., 2019). Typically, these methods first pretrain a universal 1We release the code at https://github.com/ DeepLearnXMU/embedding-transfer *Jinsong Su is the corresponding author. This work was done when Xin Liu was interning at DAMO Academy, Alibaba Group. M-BERT Ce no zo ic pala eo hy dro dyn ami c Out-of-Domain Cen ozo ic pal a e o hydro dynamic Thesis Cenozoic palaeohydrodynamic Table 1: Segmentation of English sequence “Cenozoic palaeohydrodynamic” learned from different data dis- tribution as described in § 4. High frequent words in thesis domain are split into fine-grained and under- represented tokens in pre-trained models. model using a large-scale corpus, which is then finetuned to various downstream tasks via a few adjustments. Due to its simplicity yet impressive performance, pretrain-finetune paradigm becomes the undoubtedly dominant solution for building state-of-the-art models in many natural language understanding tasks (Xu et al., 2019; Yang et al., 2019a; Liu et al., 2020b). In comparison, this strategy often achieves disap- pointing or barely satisfactory performance in natu- ral language generation (NLG) tasks. For example, several studies observe that M-BERT (Devlin et al., 2019) fails to enhance the decoder of a translation model (Edunov et al., 2019; Zhu et al., …

Content truncated to 500 words