On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method

On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method Zorik GekhmanT∗ Nadav OvedT∗ Orgad KellerG Idan SzpektorG Roi ReichartT T Technion - Israel Institute of Technology, Israel GGoogle Research, Israel {zorik@campus.|nadavo@campus.|roiri@}technion.ac.il {orgad|szpektor}@google.com Abstract Most work on modeling the conversation his- tory in Conversational Question Answering (CQA) reports a single main result on a com- mon CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realis- tic ones), training data size (e.g., from large to small sets) and domain. In this work, we design and conduct the first large-scale robust- ness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robust- ness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach and demonstrate its strong robust- ness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, in- stead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy to plug into practically any model, and highly effective, thus we recom- mend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.1 1 Introduction Conversational Question Answering (CQA) in- volves a dialogue between a user who asks questions and an agent that answers them based on a given document. CQA is an extension of the ∗Authors contributed equally to this work. 1Our code and data are available at: https://github .com/zorikg/MarCQAp. traditional single-turn QA task (Rajpurkar et al., 2016), with the major difference being the pres- ence of the conversation history, which requires effective history modeling (Gupta et al., 2020). Previous work demonstrated that the straightfor- ward approach of concatenating the conversation turns to the input is lacking (Qu et al., 2019a), leading to various proposals of architecture com- ponents that explicitly model the conversation history (Choi et al., 2018; Huang et al., 2019; Yeh and Chen, 2019; Qu et al., 2019a,b; Chen et al., 2020; Kim et al., 2021). However, there is no single agreed-upon setting for evaluating the effectiveness of such methods, with the majority of prior work reporting a single main result on a CQA benchmark, such as CoQA (Reddy et al., 2019) or QuAC (Choi et al., 2018). While recent CQA models show impressive re- sults on these benchmarks, such a single-score evaluation scheme overlooks aspects that can be essential in real-world use-cases. First, QuAC and CoQA contain large annotated training sets, which makes it unclear whether existing methods can re- main effective in small-data settings, where the annotation budget is limited. In addition, the eval- uation …

Content truncated to 500 words