Unknown Intent Detection Using Gaussian Mixture Model with an Application to Zero-shot Intent Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85–96 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 85 PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable Siqi Bao∗, Huang He∗, Fan Wang, Hua Wu and Haifeng Wang Baidu Inc., China {baosiqi, hehuang, wangfan04, wu hua, wanghaifeng}@baidu.com Abstract Pre-training models have been proved effec- tive for a wide range of natural language pro- cessing tasks. Inspired by this, we propose a novel dialogue generation pre-training frame- work to support various kinds of conversations, including chit-chat, knowledge grounded di- alogues, and conversational question answer- ing. In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation. We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in re- sponse generation. Two reciprocal tasks of re- sponse generation and latent act recognition are designed and carried out simultaneously within a shared network. Comprehensive ex- periments on three publicly available datasets verify the effectiveness and superiority of the proposed framework. 1 Introduction Dialogue generation is a challenging task due to the limited corpus of human conversations, com- plex background knowledge, and diverse relation- ships between utterances. Recently, pre-trained large-scale language models, such as BERT (De- vlin et al., 2019) and XLNet (Yang et al., 2019), have achieved prominent success in natural lan- guage processing. Such models are usually con- structed based on a massive scale of general text corpora, like English Wikipedia or BooksCorpus (Zhu et al., 2015), where distributed representa- tions can be learned automatically from the raw text. With these representations being fine-tuned, breakthroughs have been continuously reported for various downstream tasks, especially those on nat- ural language understanding, such as question an- swering, natural language inference, and so on. ∗First two authors contributed equally to this work. This pre-training and fine-tuning paradigm also sheds light on the tasks of natural language gen- eration, like dialogue generation. However, the previous study demonstrates that there are some de- ficiencies in performance while directly fine-tuning BERT on small conversation datasets (Rashkin et al., 2019; Wolf et al., 2019). Possible reasons are three-fold: 1) the underlying linguistic patterns in human conversations can be highly different from those in general text, which suggests a potentially large gap in knowledge or data distribution; 2) the training mode of uni-directional dialogue genera- tion is also distinct from that of bi-directional nat- ural language understating as applied in BERT; 3) unlike most of the general NLP tasks, there exists a one-to-many relationship in dialogue generation, where the dialogue context may correspond to mul- tiple appropriate replies. In this paper, we propose a new method to tackle the above challenges, aiming to obtain a high- quality pre-training model for dialogue generation. First of all, to reduce the gap between data dis- tributions, large-scale Reddit and Twitter conver- sations are utilized to further pre-train the gener- ation model (upon the basis of language models pre-trained with general text). Secondly, to …

Content truncated to 500 words