Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4186–4192 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4186 Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language Avia Efrat∗ Uri Shaham∗ Dan Kilman Omer Levy Tel Aviv University {avia.efrat,uri.shaham1}@gmail.com Abstract Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a mislead- ing surface reading, whose solving requires disambiguating semantic, syntactic, and pho- netic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for ex- perienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryp- tonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%). 1 Introduction The ambiguity of natural language is one of the most fundamental challenges in NLP research. While there are works and datasets specifically tar- geting ambiguity (Levesque et al., 2011; Raganato et al., 2017; Sakaguchi et al., 2020), these can be solved by a native speaker with relative ease. Can we design a dataset with ambiguities that pose a challenge even to competent native speakers? We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguis- tically complex and naturally sourced. Cryptonite’s 523K examples are taken from professionally- authored cryptic crosswords, making them less prone to artifacts and biases than examples cre- ated by crowdsourcing (Gururangan et al., 2018; Geva et al., 2019). Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a mis- leading surface reading, which poses a challenge even for humans experienced in cryptic crossword solving. A cryptic clue usually consists of two underlying parts: wordplay and definition. The ∗ Equal contribution. One doesn’t like shifting earth (5) hater wordplay defini�on synonym anagram enu. anagram indicator answer Figure 1: How to solve the cryptic clue “One doesn’t like shifting earth (5)”: Solving usually starts by fig- uring out which of the clue’s words belong to the defi- nition (blue) and which to the wordplay (orange). Next, one needs to figure out the type of wordplay, which is often hinted by an indicator (purple). In our case, “shifting” hints that the answer is an anagram of some part of the wordplay. As the enumeration (gray) states the answer is a five-letter word, “earth” is a promising candidate for anagraming. Finally, given that “hater” is both an anagram of “earth” and a synonym of the defi- nition, we conclude it to be the correct answer. clue’s answer is both a disambiguation of the word- play and, at the same time, directly answers the definition. While solving cryptic clues requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge, clues are designed to have only one …

Content truncated to 500 words