Topic Spotting using Hierarchical Networks with Self Attention

Proceedings of NAACL-HLT 2019, pages 3755–3761 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3755 Topic Spotting using Hierarchical Networks with Self Attention Pooja Chitkara1, Ashutosh Modi1, Pravalika Avvaru1, Sepehr Janghorbani1,2, Mubbasir Kapadia1,2 1Disney Research, 2Rutgers University pchitkar@andrew.cmu.edu ashutosh.modi@disneyresearch.com pavvaru@andrew.cmu.edu sepehr.janghorbani@rutgers.edu mubbasir.kapadia@rutgers.edu Abstract Success of deep learning techniques have re- newed the interest in development of dialogue systems. However, current systems struggle to have consistent long term conversations with the users and fail to build rapport. Topic spot- ting, the task of automatically inferring the topic of a conversation, has been shown to be helpful in making a dialog system more en- gaging and efficient. We propose a hierarchi- cal model with self attention for topic spotting. Experiments on the Switchboard corpus show the superior performance of our model over previously proposed techniques for topic spot- ting and deep models for text classification. Additionally, in contrast to offline processing of dialog, we also analyze the performance of our model in a more realistic setting i.e. in an online setting where the topic is identified in real time as the dialog progresses. Results show that our model is able to generalize even with limited information in the online setting. 1 Introduction Recently, a number of commercial conversation systems have been introduced e.g. Alexa, Google Assistant, Siri, Cortana, etc. Most of the avail- able systems perform well on goal-oriented con- versations which spans over few utterances in a dialogue. However, with longer conversations (in open domains), existing systems struggle to re- main consistent and tend to deviate from the cur- rent topic during the conversation. This hinders the establishment of long term social relationship with the users (Dehn and Van Mulken, 2000). In order to have coherent and engaging conversations with humans, besides other relevant natural lan- guage understanding (NLU) techniques (Jokinen and McTear, 2009), a system, while responding, should take into account the topic of the current conversation i.e. Topic Spotting. Topic spotting has been shown to be important in commercial dialog systems (Bost et al., 2013; Jokinen et al., 2002) directly dealing with the cus- tomers. Topical information is useful for speech recognition systems (Iyer and Ostendorf, 1999) as well as in audio document retrieval systems (Hazen et al., 2007; Hazen, 2011). Importance of topic spotting can be gauged from the work of Alexa team (Guo et al., 2018), who have pro- posed topic based metrics for evaluating the qual- ity of conversational bots. The authors empirically show that topic based metrics correlate with hu- man judgments. Given the importance of topical information in a dialog system, this paper proposes self attention based hierarchical model for predicting topics in a dialog. We evaluate our model on Switchboard (SWBD) corpus (Godfrey et al., 1992) and show that our model supersedes previously applied tech- niques for topic spotting. We address the evalu- ative limitations of the current SWBD corpus by creating a new version of the corpus referred as SWBD2. We hope that SWBD2 corpus would provide …

Content truncated to 500 words