Bridging the Gap between Pre-Training and Fine-Tuning for Commonsense Generation

Findings of the Association for Computational Linguistics: EACL 2023, pages 376–383 May 2-6, 2023 ©2023 Association for Computational Linguistics Bridging the Gap between Pre-Training and Fine-Tuning for Commonsense Generation ∗ Haoran Yang1, Yan Wang2, Piji Li2, Wei Bi2, Wai Lam1, Chen Xu3 1The Chinese University of Hong Kong 2Tencent AI Lab 3Beijing University of Technology hryang@se.cuhk.edu.hk Abstract Commonsense generation aims to generate a plausible sentence containing all given un- ordered concept words. Previous methods fo- cusing on this task usually directly concatenate these words as the input of a pre-trained lan- guage model (PLM). However, in PLMs’ pre- training process, the inputs are often corrupted sentences with correct word order. This input distribution discrepancy between pre-training and fine-tuning makes the model difficult to fully utilize the knowledge of PLMs. In this pa- per, we propose a two-stage framework to alle- viate this issue. Firstly, in pre-training stage, we design a new format of input to endow PLMs the ability to deal with masked sentences with incorrect word order. Secondly, during fine- tuning, we insert the special token [MASK] be- tween two consecutive concept words to make the input distribution more similar to the in- put distribution in pre-training. We conduct extensive experiments and provide a thorough analysis to demonstrate the effectiveness of our proposed method. The code is available at https://github.com/LHRYANG/CommonGen. 1 Introduction To investigate machines’ ability of generating logi- cal sentences, Lin et al. (2020) propose the Com- monsense Generation task. Given a set of concept words, this task is designed to generate a sentence which not only contains the given concepts but also can correctly describe the relations between concepts. An example is shown in Table 1. Existing methods employ the Pre-trained Languege Models (PLMs) such as BART (Lewis et al., 2020), GPT-2 (Radford et al., 2019) as the backbone to solve this problem. They (Liu et al., 2021; Fan et al., 2020; Wang et al., 2021; Li et al., 2021) usually take the concatenated concepts words as the inputs. However, such processing of inputs ∗This work was done during an internship at Tencent. concept words {wear, player, field, jersey} references The player will wear a jersey while on the field. A soccer player wears a jersey on the field. · · · output of our model football player wears a jersey on the field. Table 1: An example of Commonsense Generation task causes a huge gap between pre-training and fine- tuning. Specifically, these concept words are un- ordered which means the order of the input words is inconsistent with the order of these words in the ref- erences. It seems incompatible to PLMs pre-trained with ordered words (For BART (Lewis et al., 2020), sentence permutation is adopted, nevertheless, the word order within a sentence remains correct.). As studied by Zhao et al. (2022) and Ou et al. (2022), the word order of inputs can hinder the exploita- tion of knowledge existing in PLMs. Moreover, even if the word order of inputs is correct, for some LMs (e.g., …

Content truncated to 500 words