InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions

INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions Zeqiu Wu♠ Ryu Parish♠ Hao Cheng♣ Sewon Min♠ Prithviraj Ammanabrolu♦ Mari Ostendorf♠ Hannaneh Hajishirzi♠♦ ♠University of Washington, USA ♣Microsoft Research, USA ♦Allen Institute for AI, USA {zeqiuwu1,rparish,sewon,ostendor,hannaneh}@uw.edu chehao@microsoft.com raja@allenai.org Abstract In an information-seeking conversation, a user may ask questions that are under-specified or unanswerable. An ideal agent would in- teract by initiating different response types according to the available knowledge sources. However, most current studies either fail to or artificially incorporate such agent-side ini- tiative. This work presents INSCIT, a dataset for Information-Seeking Conversations with mixed-initiative Interactions. It contains 4.7K user-agent turns from 805 human-human con- versations where the agent searches over Wikipedia and either directly answers, asks for clarification, or provides relevant informa- tion to address user queries. The data supports two subtasks, evidence passage identification and response generation, as well as a human evaluation protocol to assess model perfor- mance. We report results of two systems based on state-of-the-art models of conver- sational knowledge identification and open- domain question answering. Both systems significantly underperform humans, suggest- ing ample room for improvement in future studies.1 1 Introduction Recently, there has been increasing interest in developing conversational information-seeking systems (Choi et al., 2018; Adlakha et al., 2022; Saeidi et al., 2018; Feng et al., 2020) that as- sist users in finding information from knowledge sources (e.g., text corpus) via multi-turn con- versational interactions. One important advantage of such conversational information-seeking sys- tems is that users do not need to come up with a very descriptive query by themselves (Webb and Webber, 2009; Rieser and Lemon, 2009; 1We open-source all data and code athttps://github .com/ellenmellon/INSCIT. Konstantinova and Orasan, 2013). In realistic set- tings, as shown in Figure 1, users can start with a request that is under-specified or has no direct answer, and through conversational interactions, the agent can collaboratively guide users to refine (left) or relax their queries and proactively sug- gest relevant information that may partially satisfy the user’s information needs (right). This collabo- ration requires a mixed-initiative dialogue, where both the user and agent can direct the flow of the conversation. Handling such realistic user requests poses challenges to a conversational agent system. A comprehensive search can result in multiple pas- sages from the knowledge source, which may provide different components of an answer (pos- sibly incomplete) or multiple answers that sur- face ambiguities in the user query. Depending on the available information, the agent needs to use different strategies, which might involve summa- rizing the results, providing partial information, or trying to clarify an ambiguity. However, ex- isting information-seeking conversation datasets rarely contain conversations where agents initiate different interaction strategies. As a result, most conversational question answering (CQA) work focuses on user-initiative interactions, where the agent simply responds to user questions with direct answers or uses no answer for out-of-scope queries (Choi et al., 2018; Reddy et al., 2019; Adlakha et al., 2022). Other work studies clarifi- cation questions using artificially created data, failing to capture natural information-seeking in- teractions (Saeidi …

Content truncated to 500 words