Dialog-Post: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10134–10148 July 9-14, 2023 ©2023 Association for Computational Linguistics DIALOG-POST: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training Zhenyu Zhang, Lei Shen, Yuming Zhao, Meng Chen∗, Xiaodong He JD AI Research, Beijing, China {zhangzhenyu47,shenlei20,zhaoyuming3}@jd.com {chenmeng20,xiaodong.he}@jd.com Abstract Dialogue representation and understanding aim to convert conversational inputs into embed- dings and fulfill discriminative tasks. Com- pared with free-form text, dialogue has two important characteristics, hierarchical seman- tic structure and multi-facet attributes. There- fore, directly applying the pretrained language models (PLMs) might result in unsatisfactory performance. Recently, several work focused on the dialogue-adaptive post-training (Dial- Post) that further trains PLMs to fit dialogues. To model dialogues more comprehensively, we propose a DialPost method, DIALOG-POST, with multi-level self-supervised objectives and a hierarchical model. These objectives lever- age dialogue-specific attributes and use self- supervised signals to fully facilitate the repre- sentation and understanding of dialogues. The novel model is a hierarchical segment-wise self-attention network, which contains inner- segment and inter-segment self-attention sub- layers followed by an aggregation and updating module. To evaluate the effectiveness of our methods, we first apply two public datasets for the verification of representation ability. Then we conduct experiments on a newly-labelled dataset that is annotated with 4 dialogue under- standing tasks. Experimental results show that our method outperforms existing SOTA models and achieves a 3.3% improvement on average. 1 Introduction As an indispensable way of communication, dia- logue is related to many research and application scenarios in academia and industry. Better dia- logue representation and understanding serve for several tasks, including intent classification, emo- tion recognition, and response selection, thus how to represent and model dialogues is an essential topic. Compared with free-form text, dialogue modeling has to pay more attention to the following ∗Corresponding author. characteristics: (1) hierarchical semantic structure (Serban et al., 2016; Xing et al., 2018; Zhang et al., 2019), i.e., dialogue → utterance → token, and (2) multi-facet attributes (See et al., 2019; Shen et al., 2021a), such as speaker-shift, content-relatedness, fact-awareness, and coherence. Therefore, directly applying pre-trained language models (PLMs) to the dialogue understanding tasks is inappropriate. To better utilize PLMs for dialogue representa- tion and understanding, researchers use data sam- ples from dialogue corpora to conduct a second phase pre-training of PLMs, i.e, dialogue-adaptive post-training (DialPost). At first, the training objec- tives were just those for general language modeling (Masked Language Modeling and Next Sentence Prediction) (Whang et al., 2020, 2021; Xu et al., 2021). After that, researchers tried to design some novel objectives that fit dialogue characteristics more. For example, Wu et al. (2021) utilized Span Boundary Objective and Perturbation Masking Ob- jective in post-training to capture the dialogue se- mantics in span and token levels. Liu et al. (2021) and Wu et al. (2020) constructed positive and neg- ative samples for context-response pairs, and con- tinued training PLMs with contrastive learning to better maintain the dialogue coherence. Existing DialPost methods either focus on token- …

Content truncated to 500 words