An Investigation of Suitability of Pre-Trained Language Models for Dialogue Generation – Avoiding Discrepancies

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4481–4494 August 1–6, 2021. ©2021 Association for Computational Linguistics 4481 An Investigation of Suitability of Pre-Trained Language Models for Dialogue Generation – Avoiding Discrepancies Yan Zeng DIRO, Universit´e de Montr´eal yan.zeng@umontreal.ca Jian-Yun Nie DIRO, Universit´e de Montr´eal nie@iro.umontreal.ca Abstract Pre-trained language models have been widely used in response generation for open-domain dialogue. These approaches are built within 4 frameworks: Transformer-ED, Transformer- Dec, Transformer-MLM and Transformer-AR. In this study, we experimentally compare them using both large and small-scale data. This re- veals that decoder-only architecture is better than stacked encoder-decoder, and both left- to-right and bi-directional attention have their own advantages. We further define two con- cepts of model discrepancy, which provides a new explanation to the model performance. As discrepancies may hinder performance, we propose two solutions to reduce them, which successfully improve the model performance. 1 Introduction It has been shown (Wolf et al., 2019) that lever- aging a pre-trained Language Model (LM) based on transformer can achieve excellent performance for dialogue generation. Different approaches have been proposed recently, which can be catego- rized into 4 frameworks: Transformer-ED(Zheng et al., 2019), an encoder-decoder Transformer, Transformer-Dec (Wolf et al., 2019; Lin et al., 2020), Transformer-MLM (Dong et al., 2019) and Transformer-AR (Bao et al., 2019; Shuster et al., 2019). The latter three all utilize a decoder-only architecture. Besides, Trans-Dec uses left-to-right attention for both source and target side, while Trans-MLM and Trans-AR employ bi-directional attention on the source side to encode dialogue his- tory. Due to this difference, Trans-Dec only utilizes left-to-right pre-trained models, e.g. GPT-2 (Rad- ford et al., 2019), while Trans-MLM/AR are based on the pre-trained models applying bi-directional attention (on the source side), e.g. BERT (Devlin et al., 2018). The difference between Trans-MLM and Trans-AR is that Trans-MLM uses masked language modeling while Trans-AR uses auto- regressive objective. Recent studies have explored pre-training dia- logue models using large-scale Reddit/Twitter data (Adiwardana et al., 2020; Roller et al., 2020). It is then straightforward to fine-tune the models for a specific dialogue task. However, in practice, there may not always be enough data for pre-training. In some cases, we still need to exploit a pre-trained LM. For example, some studies do further pre- training for dialogue based on a pre-trained LM (Zhang et al., 2019; Dong et al., 2019; Bao et al., 2019; Shuster et al., 2019), and some studies that do multi-task learning (e.g. on dialogue and ques- tion answering) can only fine-tune based on a pre- trained LM (Lin et al., 2020; Zeng and Nie, 2021). Then, a critical question is how to best exploit a pre-trained LM for dialogue generation. On this question, we have contradictory beliefs in the liter- ature: some researchers believe that Trans-Dec is appropriate because it uses a left-to-right language model that corresponds well to the dialogue gen- eration task (Zhang et al., 2019; Lin et al., 2020), while some others (Dong et al., 2019; Bao et al., 2019) show that Trans-MLM/AR …

Content truncated to 500 words