Bot2Vec: Learning Representations of Chatbots

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 75–84 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 75 Bot2Vec: Learning Representations of Chatbots Jonathan Herzig1 Tommy Sandbank∗,2 Michal Shmueli-Scheuer1 David Konopnicki1 John Richards1 1IBM Research 2Similari {hjon,shmueli,davidko}.il.ibm.com, tommy@similari.com, ajtr@us.ibm.com Abstract Chatbots (i.e., bots) are becoming widely used in multiple domains, along with supporting bot programming platforms. These platforms are equipped with novel testing tools aimed at im- proving the quality of individual chatbots. Do- ing so requires an understanding of what sort of bots are being built (captured by their under- lying conversation graphs) and how well they perform (derived through analysis of conver- sation logs). In this paper, we propose a new model, BOT2VEC, that embeds bots to a com- pact representation based on their structure and usage logs. Then, we utilize BOT2VEC representations to improve the quality of two bot analysis tasks. Using conversation data and graphs of over than 90 bots, we show that BOT2VEC representations improve detection performance by more than 16% for both tasks. 1 Introduction As conversational systems (i.e., chatbots) become more pervasive, careful analysis of their capabil- ities becomes important. Conversational systems are being used for a variety of support, service, and sales applications that were formerly handled by human agents. Thus, organizations deploying such systems must be able to understand bots be- havior to improve their performance. In many cases, such an analysis can be viewed as a classi- fication task whose goal is to check whether a bot or a particular instance of a conversation satisfies some property (e.g., is the conversation success- ful?). Models for these downstream classification tasks should benefit from conditioning on repre- sentations that capture global bot behavior. For a conversation itself, there exists a natural way to represent it as the concatenation of the hu- man and bot utterances. As for a bot, the question of its representation is more complicated: bots are ∗ Work was done while working at IBM. complex objects that execute logic in order to drive conversations with users. How should they best be represented? Many commercial companies provide bot pro- gramming platforms. These platforms provide tools and services to develop bots, monitor and improve their quality. Due to the increasing pop- ularity of bots, thousands or tens of thousands of bots could be deployed by different companies on each platform1. Although bots might have differ- ent purposes and different underlying structures, the ability to understand bot behavior at a high level could inspire new tools and services bene- fitting all bots on the platform. In this work, we explore a commercial platform, and study differ- ent bot representations. Inspired by the success of recently proposed learned embeddings for objects such as graphs (Narayanan et al., 2017), nodes (Grover and Leskovec, 2016), documents (Le and Mikolov, 2014) and words (Mikolov et al., 2013a), we pro- pose a new model, BOT2VEC, that learns bot embeddings, and propose both content and graph based representations. While previous graph em- …

Content truncated to 500 words