Representation Learning for Conversational Data using Discourse Mutual Information Maximization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1718 - 1734 July 10-15, 2022 ©2022 Association for Computational Linguistics Representation Learning for Conversational Data using Discourse Mutual Information Maximization Bishal Santra♠ IIT Kharagpur Sumegh Roychowdhury IIT Kharagpur Aishik Mandal IIT Kharagpur Vasu Gurram IIT Kharagpur Atharva Naik IIT Kharagpur Manish Gupta♣ Microsoft, India Pawan Goyal♦ IIT Kharagpur ♠bsantraigi ✉ gmail.com, ♣ gmanish ✉ microsoft.com, ♦ pawang ✉ cse.iitkgp.ac.in Abstract Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But such language modeling pre- training objectives do not take the structural information of conversational text into consid- eration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation model- ing. We empirically demonstrate that such rep- resentations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inher- ent uncertainty in response prediction. Exten- sive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based mod- els outperform strong baselines by significant margins. 1 Introduction Representation learning has transformed how we can apply machine learning to solve real-world problems. However, despite a vast body of research on pretrained language representations, there have been relatively fewer attempts to train representa- tions specifically for dialog understanding. Prior works mostly relied on finetuned representations based on generic models like BERT (Devlin et al., 2019) or GPT-2 (Radford et al., 2019). In our exper- iments, we demonstrate that such representations do not perform uniformly across various dialog un- derstanding tasks such as dialog-act classification, intent detection or dialog evaluation. On the other hand, prior works on pretraining large-scale dialog models focused mainly on open- domain generation. These works evaluated their models only on dialog generation (Zhang et al., 2020; Roller et al., 2021; Adiwardana et al., 2020) or tasks related directly to the pretraining objective (Henderson et al., 2020; Gao et al., 2020). Their ef- fectiveness on other dialog understanding tasks like act classification or intent detection remains unex- plored. So we ask the following research question: Can we learn enriched representations directly at the pretraining phase that are specifically helpful for dialog understanding? Existing language modeling (causal or masked) pretraining objectives unfortunately are not the best to model dialogs for these reasons: (1) The model is not directly trained to learn the content discourse structure (e.g., context-response in dialogs). (2) Such models are trained to generate the response word-by-word rather than predicting a larger unit. (3) The inherent one-to-many nature of dialog gen- eration implies that the encoding model should be able to capture uncertainty in the response predic- tion task, that such models ignore. Hence, in this …

Content truncated to 500 words