What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

Findings of the Association for Computational Linguistics: ACL 2022, pages 383 - 395 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems Yi-Lin Tuan1, Sajjad Beygi2, Maryam Fazel-Zarandi2 Qiaozi Gao2, Alessandra Cervone2, William Yang Wang1 1 University of California, Santa Barbara 2 Amazon Alexa AI {ytuan, william}@cs.ucsb.edu {beygi, fazelzar, qzgao, cervon}@amazon.com Abstract Users interacting with voice assistants today need to phrase their requests in a very specific manner to elicit an appropriate response. This limits the user experience, and is partly due to the lack of reasoning capabilities of dialogue platforms and the hand-crafted rules that re- quire extensive labor. One possible way to improve user experience and relieve the man- ual efforts of designers is to build an end-to- end dialogue system that can do reasoning it- self while perceiving user’s utterances. In this work, we propose a novel method to incorpo- rate the knowledge reasoning capability into dialogue systems in a more scalable and gener- alizable manner. Our proposed method allows a single transformer model to directly walk on a large-scale knowledge graph to generate re- sponses. To the best of our knowledge, this is the first work to have transformer models generate responses by reasoning over differen- tiable knowledge graphs. We investigate the reasoning abilities of the proposed method on both task-oriented and domain-specific chit- chat dialogues. Empirical results show that this method can effectively and efficiently incorpo- rate a knowledge graph into a dialogue system with fully-interpretable reasoning paths. 1 Introduction Nowadays, dialogue systems are ubiquitous in cus- tomer service and voice-based assistants. One of the main uses of this technology is supporting hu- mans in accomplishing tasks that might require accessing and navigating large knowledge bases (e.g., movies search). A dialogue system architec- ture is typically composed of a natural language understanding (NLU) module, a dialogue manage- ment (DM) module, and a natural language gener- ation (NLG) module (Jurafsky and Martin, 2009; Williams et al., 2016). First, the NLU component extracts a meaning representation from the user utterance based on which the DM generates the next system action by reasoning over the meaning representation and communicating with external applications if necessary. For example, the DM may retrieve information from external knowledge graphs (KG) to answer the user’s query based on the dialogue history. This process requires the DM to convert the output of NLU to a query to be issued to the backend. Given the difficulty of this step, which is often domain-dependent, the DM com- ponent might require the design of hand-crafted rules. However, such rules are usually not scalable to different applications. They could require con- siderable effort to cover all possible cases/dialogue flows, leading to expensive costs to design new applications. Moreover, in several cases, users in- teracting with such assistants are forced to formu- late specific queries in order to accomplish their objective, which might break user engagement. To alleviate the problem of having to design ex- pensive hand-crafted rules and breaking user …

Content truncated to 500 words