RELiC: Retrieving Evidence for Literary Claims

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7500 - 7518 May 22-27, 2022 c⃝2022 Association for Computational Linguistics RELI C : Retrieving Evidence for Literary Claims Katherine Thai Yapei Chang Kalpesh Krishna Mohit Iyyer University of Massachusetts Amherst, Smith College {kbthai,kalpesh,miyyer}@cs.umass.edu echang33@smith.edu Project Page: https://relic.cs.umass.edu Abstract Humanities scholars commonly provide evi- dence for claims that they make about a work of literature (e.g., a novel) in the form of quo- tations from the work. We collect a large-scale dataset (RELiC) of 78K literary quotations and surrounding critical analysis and use it to for- mulate the novel task of literary evidence re- trieval, in which models are given an excerpt of literary analysis surrounding a masked quo- tation and asked to retrieve the quoted passage from the set of all passages in the work. Solv- ing this retrieval task requires a deep under- standing of complex literary and linguistic phe- nomena, which proves challenging to meth- ods that overwhelmingly rely on lexical and semantic similarity matching. We implement a RoBERTa-based dense passage retriever for this task that outperforms existing pretrained information retrieval baselines; however, ex- periments and analysis by human domain ex- perts indicate that there is substantial room for improvement over our dense retriever. 1 Introduction When analyzing a literary work (e.g., a novel or short story), scholars make claims about the text and provide supporting evidence in the form of quo- tations from the work (Thompson, 2002; Finnegan, 2011; Graff et al., 2014). For example, Monaghan (1980) claims that Elizabeth, the main character in Jane Austen’s Pride and Prejudice, doesn’t just refuse an offer to join the standoffish bachelor Darcy and the wealthy Bingleys on their morning walk, “but does so in such a way as to group Darcy with the snobbish Bingley sisters,” and then di- rectly quotes Elizabeth’s tongue-in-cheek rejection: “No, no; stay where you are. You are charmingly grouped, and appear to uncommon advantage. The picturesque would be spoilt by admitting a fourth.” Literary scholars construct arguments like these by making complex connective inferences between their interpretations, framed as claims, and quota- tions (e.g., recognizing that Elizabeth says “charm- ingly grouped” and “picturesque” ironically in or- der to group Darcy with the snobbish Bingley sis- ters). This process requires a deep understand- ing of both literary phenomena, such as irony and metaphor, and linguistic phenomena (coreference, paraphrasing, and stylistics). In this paper, we com- putationally study the relationship between liter- ary claims and quotations by collecting a large- scale dataset for Retrieving Evidence for Literary Claims (RELiC), which contains 78K scholarly ex- cerpts of literary analysis that each directly quote a passage from one of 79 widely-read English texts. The complexity of the claims and quotations in RELiC makes it a challenging testbed for modern neural retrievers: given just the text of the claim and analysis that surrounds a masked quotation, can a model retrieve the quoted passage from the set of all possible passages in …

Content truncated to 500 words