MICO: Selective Search with Mutual Information Co-training

Proceedings of the 29th International Conference on Computational Linguistics, pages 1179–1192 October 12–17, 2022. 1179 MICO: Selective Search with Mutual Information Co-training Zhanyu Wang∗, † Purdue University {wang4094}@purdue.edu Xiao Zhang† and Hyokun Yun and Choon Hui Teo and Trishul Chilimbi Amazon {zhxao, yunhyoku, choonhui, trishulc}@amazon.com Abstract In contrast to traditional exhaustive search, selective search first clusters documents into several groups before all the documents are searched exhaustively by a query, to limit the search executed within one group or only a few groups. Selective search is designed to reduce the latency and computation in mod- ern large-scale search systems. In this study, we propose MICO, a Mutual Information CO- training framework for selective search with minimal supervision using the search logs. Af- ter training, MICO does not only cluster the documents, but also routes unseen queries to the relevant clusters for efficient retrieval. In our empirical experiments, MICO signifi- cantly improves the performance on multiple metrics of selective search and outperforms a number of existing competitive baselines. 1 Introduction In information retrieval (IR), searching over all the documents is quite costly at a large scale (Risvik et al., 2013). Selective search (Kulkarni, 2013; Kulkarni and Callan, 2015) divides documents into a number of shards (clusters) to allow an incoming query to search over only a small number of these shards that are most relevant to this query. It tries to preserve the quality of search results similar to that of searching over the whole corpus, with less computation and shorter retrieval latency, achieved by limiting an incoming query to search within only a few chosen document shards. Sharding is the process of breaking a massive collection of documents into smaller chunks called shards. Ran- domly assigning documents to shards, a popular approach, cannot guarantee documents relevant to a specific query restrained within one shard or only a few shards. With the observation that semanti- cally similar documents tend to appear together in ∗ Work done while doing an internship at Amazon. † Equal contribution. the search results of the same query (van Rijsber- gen, 1979), researchers in recent years have devised multiple machine learning-based algorithms for se- lective search. These algorithms are tailored to benefit the document retrieval process by dividing a large volume of the corpus in a certain way so that semantically related documents are allocated in the same cluster and then routing new queries to the most relevant clusters subsequently. In this way, the retrieval system can return the search re- sults with both efficiency and accuracy, as shown in Figure 1. These approaches significantly differ from the traditional sharding methods that either randomly split the collection into several groups (Barroso et al., 2003) or allocate them based on simple handcrafted rules (Gravano et al., 1999), and have achieved comparable performance to ex- haustive search over the entire document corpus. q d1 d5 d4 d2 d6 d3 d7 Search Results q d1 d2 d3 d4 d5 d6 : d7 …… q q q Figure 1: Machine learning …

Content truncated to 500 words