QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14032–14047 July 9-14, 2023 ©2023 Association for Computational Linguistics QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations Chaitanya Malaviya1∗, Peter Shaw2, Ming-Wei Chang2, Kenton Lee2, Kristina Toutanova2 1University of Pennsylvania 2Google DeepMind cmalaviy@seas.upenn.edu {petershaw,mingweichang,kentonl,kristout}@google.com Abstract Formulating selective information needs re- sults in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for "shorebirds that are not sandpipers" or "science-fiction films shot in England". To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of en- tities corresponding to Wikipedia documents. The dataset challenges models to match multi- ple constraints mentioned in queries with cor- responding evidence in documents and cor- rectly perform various set operations. The dataset is constructed semi-automatically us- ing Wikipedia category names. Queries are automatically composed from individual cate- gories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of en- tities based on their documents and highlight attribution of query constraints to spans of doc- ument text. We analyze several modern re- trieval systems, finding that they often strug- gle on such queries. Queries involving nega- tion and conjunction are particularly challeng- ing and systems are further challenged with combinations of these operations.1 1 Introduction People often express their information needs with multiple preferences or constraints. Queries corre- sponding to such needs typically implicitly express set operations such as intersection, difference, and union. For example, a movie-goer might be looking for a science-fiction film from the 90s which does not feature aliens and a reader might be interested in a historical fiction novel set in France. Similarly, ∗Work done during an internship at Google. 1The dataset is available at https://github.com/ google-research/language/tree/master/language/ quest. Birds of Venezuelan Andes — Birds of Colombia Birds found in the Venezuelan Andes but not in Colombia 🗹 Fluent 🗹 Natural ➔ Rose-crowned parakeet ➔ Mérida sunangel 🗹 Complete Evidence 🗹 Definitely Relevant 🗹 Partial Evidence 🗹 Likely Relevant (1) (3) (4) Relevance & Attribution Labeling (2) Paraphrasing Figure 1: The dataset construction process for QUEST. First, (1) we sample Wikipedia category names and find their corresponding set of relevant entities. (2) Then, we compose a query with set operations and have this query paraphrased by crowdworkers. (3) These queries are then validated for fluency and naturalness. (4) Fi- nally, crowdworkers mark the entities’ relevance by highlighting attributable spans in their documents. a botanist attempting to identify a species based on their recollection might search for shrubs that are evergreen and found in Panama. Further, if the set of entities that satisfy the constraints is rela- tively small, a reader may like to see and explore an exhaustive list of these entities. In addition, to verify and trust a system’s recommendations, users benefit from being shown evidence …

Content truncated to 500 words