Resolving Referring Expressions in Conversational Dialogs for Natural User Interfaces

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2094–2104, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Resolving Referring Expressions in Conversational Dialogs for Natural User Interfaces Asli Celikyilmaz, Zhaleh Feizollahi, Dilek Hakkani-Tur, Ruhi Sarikaya Microsoft asli@ieee.org, zhalehf@microsoft.com dilek@ieee.org, ruhi.sarikaya@microsoft.com Abstract Unlike traditional over-the-phone spoken dialog systems (SDSs), modern dialog systems tend to have visual rendering on the device screen as an additional modal- ity to communicate the system’s response to the user. Visual display of the system’s response not only changes human behav- ior when interacting with devices, but also creates new research areas in SDSs. On- screen item identification and resolution in utterances is one critical problem to achieve a natural and accurate human- machine communication. We pose the problem as a classification task to cor- rectly identify intended on-screen item(s) from user utterances. Using syntactic, se- mantic as well as context features from the display screen, our model can resolve dif- ferent types of referring expressions with up to 90% accuracy. In the experiments we also show that the proposed model is ro- bust to domain and screen layout changes. 1 Introduction Todays natural user interfaces (NUI) for applica- tions running on smart devices, e.g, phones (SIRI, Cortana, GoogleNow), consoles (Amazon FireTV, XBOX), tablet, etc., can handle not only simple spoken commands, but also natural conversational utterances. Unlike traditional over-the-phone spo- ken dialog systems (SDSs), user hears and sees the system’s response displayed on the screen as an additional modality. Having visual access to the system’s response and results changes human be- havior when interacting with the machine, creating new and challenging problems in SDS. [System]: How can i help you today ? [User]: Find non-fiction books by Chomsky. [System]: (Fetches the following books from database) [User]: “show details for the oldest production” or “details for the syntax book” or “open the last one” or “i want to see the one on linguistics” or “bring me Jurafsky’s text book” Table 1: A sample multi-turn dialog. A list of second turn utterances referring to the last book (in bold) and a new search query (highlighted) are shown. Consider a sample dialog in Table 1 between a user and a NUI in the books domain. After the sys- tem displays results on the screen, the user may choose one or more of the on-screen items with natural language utterances as shown in Table 1. Note that, there are multiple ways of referring to the same item, (e.g. the last book)1. To achieve a natural and accurate human to machine conversa- tion, it is crucial to accurately identify and resolve referring expressions in utterances. As important as interpreting referring expressions (REs) is for modern NUI designs, relatively few studies have investigated withing the SDSs. Those that do fo- cus on the impact of the input from multimodal interfaces such as gesture for understanding (Bolt, 1980; Heck et al., 2013; Johnston et al., 2002), touch for ASR error correction (Huggins-Daines and Rudnicky, 2008), …

Content truncated to 500 words