Content-based Models of Quotation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2296–2314 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2296 Content-based Models of Quotation Ansel MacLaughlin Khoury College of Computer Science Northeastern University Boston, MA ansel@ccs.neu.edu David A. Smith Khoury College of Computer Science Northeastern University Boston, MA dasmith@ccs.neu.edu Abstract We explore the task of quotability identifica- tion, in which, given a document, we aim to identify which of its passages are the most quotable, i.e. the most likely to be directly quoted by later derived documents. We ap- proach quotability identification as a passage ranking problem and evaluate how well both feature-based and BERT-based (Devlin et al., 2019) models rank the passages in a given doc- ument by their predicted quotability. We ex- plore this problem through evaluations on five datasets that span multiple languages (English, Latin) and genres of literature (e.g. poetry, plays, novels) and whose corresponding de- rived documents are of multiple types (news, journal articles). Our experiments confirm the relatively strong performance of BERT-based models on this task, with the best model, a RoBERTA sequential sentence tagger, achiev- ing an average ρ of 0.35 and NDCG@1, 5, 50 of 0.26, 0.31 and 0.40, respectively, across all five datasets. 1 Introduction Unlike in scientific writing, where authors use terse citations due to space constraints, direct quotation of source material is an essential part of writing in many fields. Journalists, humanities scholars, and students, for instance, often quote from a range of source documents, such as interviews, speeches, and books. Quotes can be used to substantiate a claim, lend authority to an argument, or offer a viewpoint to reflect on or argue against, among many others. Modeling the process of quotation selection is thus an important step in modeling how authors compose entire new documents. In this paper, we explore the problem of quota- bility identification, identifying which passages in a source work (e.g. Hamlet) are likely to be quoted by later, derived works (e.g. humanities journal articles). Prior research has attempted to iden- tify the specific factors that influence a passage or document’s quotability. Most work, therefore, has focused on manual feature engineering and de- velopment of careful analysis frameworks to test which features have statistically significant relation- ships to quote counts (Tan et al., 2018; Danescu- Niculescu-Mizil et al., 2012). We, instead, reframe quotability identification as a practical passage ranking task and evaluate how well models can rank the passages in a given document by their pre- dicted quotability. We benchmark and analyze the performances of multiple models, exploring both BERT-based passage ranking models and feature- based models equipped with “quotability” features identified in prior work. We collect five large-scale datasets to study this problem. Each dataset consists of sets of source documents, derived documents, and alignments between them – the direct quotes. The proposed datasets are diverse, allowing us to model quota- bility dynamics across multiple source and derived document genres and languages. Uses for our proposed …

Content truncated to 500 words