Learning to Retrieve Engaging Follow-Up Queries

Findings of the Association for Computational Linguistics: EACL 2023, pages 2009–2016 May 2-6, 2023 ©2023 Association for Computational Linguistics Learning to Retrieve Engaging Follow-Up Queries Christopher Richardson ‡∗ Sudipta Kar † Anjishnu Kumar † Anand Ramachandran † Omar Zia Khan † Zeynab Raeesy † Abhinav Sethy † ‡ Georgia Institute of Technology † Amazon Alexa AI crichardson332@gmail.com {sudipkar,anjikum,anramac,ozkhan,raeesyzr,sethya}@amazon.com Abstract Open domain conversational agents can answer a broad range of targeted queries. However, the sequential nature of interaction with these sys- tems makes knowledge exploration a lengthy task which burdens the user with asking a chain of well phrased questions. In this paper, we present a retrieval based system and associated dataset for predicting the next questions that the user might have. Such a system can proactively assist users in knowledge exploration leading to a more engaging dialog. The retrieval system is trained on a dataset called the Follow-up Query Bank (FQ-Bank). FQ-Bank contains ≈14K multi-turn information-seeking conversations with a valid follow-up question and a set of invalid candidates. The invalid candidates are generated to simulate various syntactic and se- mantic confounders such as paraphrases, partial entity match, irrelevant entity, and ASR errors. We use confounder specific techniques to simu- late these negative examples on the OR-QuAC dataset. Then, we train ranking models on FQ- Bank and present results comparing supervised and unsupervised approaches. The results sug- gest that we can retrieve the valid follow-ups by ranking them in higher positions compared to confounders, but further knowledge grounding can improve ranking performance. FQ-Bank is publicly available at https://github.c om/amazon-science/fq-bank. 1 Introduction State of the art open domain conversational voice assistants can help users accomplish a wide range of tasks, including: factoid question answering, playing music, adding items to personal lists, con- trolling smart home appliances, and booking trans- portation. However, the linear nature of dialog with existing voice assistant technology makes it chal- lenging for users to discover and fully utilize the ∗ Work done during internship. Figure 1: An overview of the follow-up question (FQ) retrieval system. full range of these capabilities. In addition, success- ful utilization often requires exact formulation of the request, which further hinders the experience. One recent approach to addressing these issues in the voice assistant domain involves predicting rel- evant follow-up queries in order to assist the user with accomplishing their latent goals.1 Relevant follow-up queries (FQs) for typical voice assistant scenarios can range from specific command and control tasks such as “What is the temperature in New York” followed by “What is the chance of rain in New York?”, to more open-ended knowledge exploration, e.g. “What is the capital of Croatia" followed by “What is the population of Croatia?". Once a valid FQ has been identified, the system can proactively recommend it to reduce user’s cognitive load, e.g. “Would you like to know what the population of Croatia is?”. This exchange is illustrated in Figure 1. The user can then be en- gaged in follow-up dialogue without the need to ask redundant questions. Follow-up …

Content truncated to 500 words