Using Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7014–7027 August 1–6, 2021. ©2021 Association for Computational Linguistics 7014 Using Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems Claudio S. Pinhanez, Paulo Cavalin, Victor Ribeiro, Ana Paula Appel, Heloisa Candello, Julio Nogima, Mauro Pichiliani, Melina Guerra, Maira Gatti de Bayser, Gabriel Malfatti, Henrique Ferreira IBM Research - Brazil csantosp@br.ibm.com Abstract In this paper we explore the improvement of in- tent recognition in conversational systems by the use of meta-knowledge embedded in in- tent identifiers. Developers often include such knowledge, structure as taxonomies, in the documentation of chatbots. By using neuro- symbolic algorithms to incorporate those tax- onomies into embeddings of the output space, we were able to improve accuracy in intent recognition. In datasets with intents and ex- ample utterances from 200 professional chat- bots, we saw decreases in the equal error rate (EER) in more than 40% of the chatbots in comparison to the baseline of the same algo- rithm without the meta-knowledge. The meta- knowledge proved also to be effective in de- tecting out-of-scope utterances, improving the false acceptance rate (FAR) in two thirds of the chatbots, with decreases of 0.05 or more in FAR in almost 40% of the chatbots. When con- sidering only the well-developed workspaces with a high level use of taxonomies, FAR de- creased more than 0.05 in 77% of them, and more than 0.1 in 39% of the chatbots. 1 Introduction Classification of sentences into a discrete set of classes is a key part of professional conversational systems. In fact, most of those systems require developers to define the different classes, or in- tents, by enumerating exemplars of each of them, since classification is often performed using ma- chine learning (ML) methods. The process of clas- sifying an input sentence into a specific intent or signaling it as out-of-scope (OOS) of the system is often referred to as intent recognition. Determining a class solely on a list of exemplars is a practical method to implement ML systems but it is hardly a natural way for human beings to de- fine a class. In real life, people define classes often using a rich mix of symbolic definitions, sometimes taxonomic in nature, such as in “a credit card is a type of bank card”, coupled with its sub-classes, for instance, “basic”, “premium”, and typical features such as “international”. People also use exemplars, “card X of bank Y is a credit card”, as well as par- ticular examples to describe a sub-class, such as in “card W is an international card”. They also use counter examples, either categorically, “a debit card is not a credit card”, or in examples, “card Z is not a credit card”. Defining and specifying classes in the real world is, in fact, a cultural, con- textual, and linguistic construct, and how people and societies perform this process is a traditional research subject in social …

Content truncated to 500 words