SCAI-QReCC Shared Task on Conversational Question Answering

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4913–4922 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4913 SCAI-QReCC Shared Task on Conversational Question Answering Svitlana Vakulenko⭑♠, Johannes Kiesel♦, Maik Fr¨obe♥ ⭑Amazon, Spain, svvakul@amazon.com ♦Bauhaus-Universit¨at Weimar, Germany, johannes.kiesel@uni-weimar.de ♥Martin-Luther-Universit¨at Halle-Wittenberg, Germany, maik.froebe@informatik.uni-halle.de Abstract Search-Oriented Conversational AI (SCAI) is an established venue that regularly puts a spotlight upon the recent work advancing the field of conversational search. SCAI’21 was organised as an independent online event and featured a shared task on conversational question answering, on which this paper reports. The shared task featured three subtasks that correspond to three steps in conversational question answering: question rewriting, passage retrieval, and answer generation. This report discusses each subtask, but emphasizes the answer generation subtask as it attracted the most attention from the participants and we identified evaluation of answer correctness in the conversational settings as a major challenge and acurrent research gap. Alongside the automatic evaluation, we conducted two crowdsourcing experiments to collect annotations for answer plausibility and faithfulness. As a result of this shared task, the original conversational QA dataset used for evaluation was further extended with alternative correct answers produced by the participant systems. Keywords: Conversational Systems, Question Answering 1. Introduction Conversational Question Answering (QA) is a chal- lenging task at the current research frontier (Qiu et al., 2021; Kim et al., 2021) important for devel- oping conversational information retrieval (conversa- tional search) systems (Anand et al., 2019). In con- versational QA, a system is required to return a cor- rect answer given a question and the previous conver- sation turns. Such questions are often ambiguous out- side of the conversational context and require incorpo- rating additional information contained in the previous conversation turns. Moreover, evaluating systems for conversational QA, especially for questions that require long generative answers drawn from multiple informa- tion sources, remains an open research problem in its own right (Voorhees, 2003; Krishna et al., 2021; Sib- lini et al., 2021; Li et al., 2021). This paper provides an overview of the SCAI-QReCC 2021 shared task on conversational question answering. It reports on the extended conversational QA dataset employed in this task, participating systems and in- sights gained from their performance, and the chal- lenges we faced when evaluating the submissions and our approach for dealing with those. Specifically, we designed and tested a set of guidelines to support eval- uation of (conversational) QA models. The shared task was built around the recently intro- duced QReCC dataset (Anantha et al., 2021). QReCC contains sequences of conversational questions paired with answers produced by human annotators. Such sequences imitate a dialogue session with follow-up questions asked by the user. The annotators had ac- ♠ Research conducted when the author was at the Uni- versity of Amsterdam. cess to a web search engine and every answer is based on the content of a single web page, while several web pages may be used to answer different questions within the same sequence. …

Content truncated to 500 words