NEREL: A Russian Dataset with Nested Named Entities, Relations and Events

Proceedings of Recent Advances in Natural Language Processing, pages 1–9 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_001 1 BPoMP: The Benchmark of Poetic Minimal Pairs – Limericks, Rhyme, and Narrative Coherence Almas Abdibayev Department of Computer Science Dartmouth College Hanover, NH 03755 almas.abdibayev.gr@dartmouth.edu Allen Riddell Department of English Indiana University Bloomington, IN 47408 riddella@indiana.edu Daniel Rockmore Department of Computer Science Dartmouth College Hanover, NH 03755 daniel.n.rockmore@dartmouth.edu Abstract We adapt BLiMP (Benchmark of Linguis- tic Minimal Pairs) language model evalua- tion framework to the context of poetry, in- troducing the first of a series of tasks titled Benchmark of Poetic Minimal Pairs (BPoMP). The tasks presented herein use one genre of English-language poetry, the limerick (five- lines, rhyme scheme AABBA). Following the BLiMP schema, the BPoMP tasks use 10,000 minimal pairs of limerick/corrupted limerick. The latter is created by (1) shuffling two rhyming end-of-the-line words, (2) shuffling two rhyming lines, (3) replacing end-of-the- line word by a non-rhyming synonym. Our general task is detection of the original limer- ick, which we believe tests a language model’s capacity to utilize “end rhymes”, a common feature of poetry. We evaluate Transformer- based models by checking if they assign a higher probability to the non-corrupted limer- ick in each minimal pair. We find that the mod- els identify the original limerick at rates better than chance, but with a nontrivial gap relative to human accuracy (average of 98.3% across tasks). The publicly available curated set of limericks accompanying this paper is an addi- tional contribution. In general, we see this as a first step to create a community of NLP activ- ity around the rigorous computational study of poetry. 1 Introduction Machines — i.e., artificial intelligence – con- tinue to make great progress toward the challenges of machine-assisted and machine-written litera- ture. Examples range from word auto-completion algorithms to deep learning-based methods that (sometimes) produce Turing Test-passing text (Elkins and Chun, 2020) and even credible son- nets (Ghazvininejad et al., 2016). Evaluating such work uses a range of techniques, including auto- matic metrics such as BLEU as well as – rightly or wrongly – Turing Test-inspired human evaluation. This paper focuses on testing if large models pre- trained on gigabytes of text, such as GPT-2 (Rad- ford et al., 2019) and Transformer-XL (Dai et al., 2019)), are capable of “discovering” the rhyming and basic narrative information inherently present within a simple poetic form, the limerick, a five- line poem, usually humorous, with rhyme scheme AABBA and regular – albeit not fixed – metrical structure. In particular, we test Transformer models on various (suitably contextualized) forms of the line completion problem for the limerick as well as a fundamental narrative-oriented task. These tests should be viewed as a first step toward a broader goal of framing the rigorous interrogation of the poetic (or more broadly, literary) capabilities of language models. Given their formulaic structure (e.g., AABBA rhyming scheme) and the relatively small size of each unit, limericks present a good testing ground as compared to longer or more …

Content truncated to 500 words