Saving Dense Retriever from Shortcut Dependency in Conversational Search

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10278 - 10287 December 7-11, 2022 ©2022 Association for Computational Linguistics Saving Dense Retriever from Shortcut Dependency in Conversational Search Sungdong Kim1,2 Gangwoo Kim3⋄ NAVER AI Lab1 KAIST AI2 Korea University3 sungdong.kim@navercorp.com gangwoo_kim@korea.ac.kr Abstract Conversational search (CS) needs a holistic un- derstanding of conversational inputs to retrieve relevant passages. In this paper, we demon- strate the existence of a retrieval shortcut in CS, which causes models to retrieve passages solely relying on partial history while disre- garding the latest question. With in-depth anal- ysis, we first show that naively trained dense retrievers heavily exploit the shortcut and hence perform poorly when asked to answer history- independent questions. To build more robust models against shortcut dependency, we ex- plore various hard negative mining strategies. Experimental results show that training with the model-based hard negatives (Xiong et al., 2020) effectively mitigates the dependency on the shortcut, significantly improving dense re- trievers on recent CS benchmarks. In partic- ular, our retriever outperforms the previous state-of-the-art model by 11.0 in Recall@10 on QReCC (Anantha et al., 2021).1 1 Introduction Conversational search (CS) is a task of retrieving relevant passages from a large amount of web text given the current question and its conversational history, which consists of previously asked ques- tions and their answers (Dalton et al., 2019). Un- like open-domain question answering (ODQA) tak- ing a single question (Voorhees and Tice, 2000; Chen et al., 2017), CS assumes a sequence of ques- tions interactively taken from information seekers. Hence, the questions need to be understood with the conversational history to find relevant evidence at each turn. To build a retriever that properly makes use of the conversational history, we first analyze a simple dense retriever baseline trained on one of the CS datasets, QReCC (Anantha et al., 2021). Our anal- ysis shows us the existence of a retrieval shortcut ⋄ Work done while interning at NAVER AI Lab 1The code is available at github.com/naver-ai/cs-shortcut. Figure 1: An example of a retrieval shortcut in con- versational search. While we expect the retriever to predict relevant passages by using all conversational inputs up to q3 (Blue solid line), a dense retriever often ignores current turn question q3 and only exploits previ- ous history, a2 (Red dashed line). We show the shortcut dependency is harmful to robust retrieval. in recent CS datasets, indicating dense retrievers heavily rely on the shortcut and retrieve irrelevant passages. Specifically, these shortcuts represent the spurious correlation between the conversational his- tory and the relevant passages, pushing the dense retrievers to ignore current questions. For example, as illustrated in Figure 1, a dense retriever retrieves wrong passages only paying attention to ‘Russia’ and ‘World Cup’ mentioned in the previous his- tory (a1, a2) while ignoring the crucial cue ‘win the World Cup’ in the current question q3. Motivated by our observation, we further test how much the shortcut contributes to the perfor- mance of current retrievers. First, we build …

Content truncated to 500 words