Going beyond research datasets: Novel intent discovery in the industry setting

Findings of the Association for Computational Linguistics: EACL 2023, pages 925–941 May 2-6, 2023 ©2023 Association for Computational Linguistics Going beyond research datasets: Novel intent discovery in the industry setting Aleksandra Chrabrowa, Tsimur Hadeliya, Dariusz Kajtoch, Robert Mroczkowski, Piotr Rybak ML Research at Allegro, Pozna´n, Poland {firstname.lastname}@allegro.pl Abstract Novel intent discovery automates the process of grouping similar messages (questions) to identify previously unknown intents. However, current research focuses on publicly available datasets which have only the question field and significantly differ from real-life datasets. This paper proposes methods to improve the intent discovery pipeline deployed in a large e-commerce platform. We show the benefit of pre-training language models on in-domain data: both self-supervised and with weak su- pervision. We also devise the best method to utilize the conversational structure (i.e., ques- tion and answer) of real-life datasets during fine-tuning for clustering tasks, which we call Conv. All our methods combined to fully uti- lize real-life datasets give up to 33pp perfor- mance boost over state-of-the-art Constrained Deep Adaptive Clustering (CDAC) (Lin et al., 2020) model for question only. By comparison CDAC model for the question data only gives only up to 13pp performance boost over the naive baseline. 1 Introduction Allegro is one of largest the e-commerce market- place in Central Eastern Europe region that con- nects buyers and merchants. It has millions of active users. Therefore, the good functioning of the Customer Experience (CX) department is cru- cial as it provides the necessary support, resolves emerging issues, and answers user questions. Task-oriented chatbots relieve humans by auto- matically resolving the most repetitive and triv- ial issues. They usually have a pre-defined set of user intents with matching template answers. Then, when a user asks a question, the intent classifier detects the question intent and returns the matching response. Creating a reliable and comprehensive chatbot requires massive work to discover, define, and maintain a set of intents with training examples. With the continuous development of marketplace platforms, new intents constantly appear as new features are introduced. Therefore, the automated intent discovery system becomes a critical compo- nent. Novel intent discovery is performed offline on historical data. In the context of personalized in- telligence assistants existing approaches (Lin et al., 2020; Gao et al., 2021; Vedula et al., 2022) focus on learning transferable features with utterance en- coders that guide the discovery on unlabeled data with a handful of labeled examples belonging to known intents. However, at Allegro our main com- munication form is emails, and we have access to much richer conversational data that can improve discovery performance. A large body of historical conversational data (user questions and consultants’ answers) can be leveraged in two ways. Firstly, to better initialize message encoders and secondly by performing intent discovery on conversational data as an additional signal. Additionally, a form of weak supervision is available: keywords (or tags) added by the consultants that help them understand past cases. The paper’s main contribution is the demonstra- tion that incorporating additional signals like con- …

Content truncated to 500 words