COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3030–3042 June 6–11, 2021. ©2021 Association for Computational Linguistics 3030 COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List Luyu Gao, Zhuyun Dai, Jamie Callan Language Technologies Institute Carnegie Mellon University {luyug, zhuyund, callan}@cs.cmu.edu Abstract Classical information retrieval systems such as BM25 rely on exact lexical match and carry out search efficiently with inverted list index. Recent neural IR models shifts towards soft semantic matching all query document terms, but they lose the computation efficiency of exact match systems. This paper presents COIL, a contextualized exact match retrieval architecture that brings semantic lexical match- ing. COIL scoring is based on overlapping query document tokens’ contextualized repre- sentations. The new architecture stores con- textualized token representations in inverted lists, bringing together the efficiency of exact match and the representation power of deep language models. Our experimental results show COIL outperforms classical lexical re- trievers and state-of-the-art deep LM retrievers with similar or smaller latency.1 1 Introduction Widely used, bag-of-words (BOW) information re- trieval (IR) systems such as BM25 rely on exact lexical match 2 between query and document terms. Recent study in neural IR takes a different approach and compute soft matching between all query and document terms to model complex matching. The shift to soft matching in neural IR models attempts to address vocabulary mismatch problems, that query and the relevant documents use differ- ent terms, e.g. cat v.s. kitty, for the same con- cept (Huang et al., 2013; Guo et al., 2016; Xiong et al., 2017). Later introduction of contextualized representations (Peters et al., 2018) from deep lan- guage models (LM) further address semantic mis- match, that the same term can refer to different concepts, e.g., bank of river vs. bank in finance. Fine-tuned deep LM rerankers produce token rep- resentations based on context and achieve state-of- 1Our code is available at https://github.com/ luyug/COIL. 2Exact match up to morphological changes. the-art in text ranking with huge performance leap (Nogueira and Cho, 2019; Dai and Callan, 2019b). Though the idea of soft matching all tokens is carried through the development of neural IR mod- els, seeing the success brought by deep LMs, we take a step back and ask: how much gain can we get if we introduce contextualized representations back to lexical exact match systems? In other words, can we build a system that still performs exact query- document token matching but compute matching signals with contextualized token representations instead of heuristics? This may seem a constraint on the model, but exact lexical match produce more explainable and controlled patterns than soft match- ing. It also allows search to focus on only the subset of documents that have overlapping terms with query, which can be done efficiently with in- verted list index. Meanwhile, using dense contex- tualized token representations enables the model to handle semantic mismatch, which has been a long-standing problem in classic lexical systems. …

Content truncated to 500 words