Automated Crossword Solving

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 256 - 269 May 22-27, 2022 c⃝2022 Association for Computational Linguistics New Intent Discovery with Pre-training and Contrastive Learning Yuwei Zhang2∗ Haode Zhang1 Li-Ming Zhan1 Xiao-Ming Wu1† Albert Y.S. Lam3 Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R.1 University of California, San Diego2 Fano Labs, Hong Kong S.A.R.3 zhangyuwei.work@gmail.com {haode.zhang, lmzhan.zhan}@connect.polyu.edu.hk csxmwu@comp.polyu.edu.hk, albert@fano.ai Abstract New intent discovery aims to uncover novel intent categories from user utterances to ex- pand the set of supported intent classes. It is a critical task for the development and ser- vice expansion of a practical dialogue system. Despite its importance, this problem remains under-explored in the literature. Existing ap- proaches typically rely on a large amount of labeled utterances and employ pseudo-labeling methods for representation learning and clus- tering, which are label-intensive, inefficient, and inaccurate. In this paper, we provide new solutions to two important research questions for new intent discovery: (1) how to learn se- mantic utterance representations and (2) how to better cluster utterances. Particularly, we first propose a multi-task pre-training strat- egy to leverage rich unlabeled data along with external labeled data for representation learn- ing. Then, we design a new contrastive loss to exploit self-supervisory signals in unlabeled data for clustering. Extensive experiments on three intent recognition benchmarks demon- strate the high effectiveness of our proposed method, which outperforms state-of-the-art methods by a large margin in both unsupervised and semi-supervised scenarios. The source code will be available at https://github. com/zhang-yu-wei/MTP-CLNN. 1 Introduction Why Study New Intent Discovery (NID)? Recent years have witnessed the rapid growth of conversa- tional AI applications. To design a natural language understanding system, a set of expected customer intentions are collected beforehand to train an in- tent recognition model. However, the pre-defined intents cannot fully meet customer needs. This implies the necessity of expanding the intent recog- nition model by repeatedly integrating new intents discovered from unlabeled user utterances (Fig. 1). ∗Work done while the author was with HK PolyU. † Corresponding author. Figure 1: New Intent Discovery. To reduce the effort in manually identifying un- known intents from a mass of utterances, previous works commonly employ clustering algorithms to group utterances of similar intents (Cheung and Li, 2012; Hakkani-Tür et al., 2015; Padmasundari, 2018). The cluster assignments thereafter can ei- ther be directly used as new intent labels or as heuristics for faster annotations. Research Questions (RQ) and Challenges. Current study of NID centers around two basic research questions: 1) How to learn semantic ut- terance representations to provide proper cues for clustering? 2) How to better cluster the utterances? The study of the two questions are often interwoven in existing research. Utterances can be represented according to different aspects such as the style of language, the related topics, or even the length of sentences. It is important to learn semantic ut- terance representations to provide proper cues for clustering. Simply applying a vanilla pre-trained language …

Content truncated to 500 words