Down and Across: Introducing Crossword-Solving as a New NLP Benchmark

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2648 - 2659 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Down and Across: Introducing Crossword-Solving as a New NLP Benchmark Saurabh Kulshreshtha, Olga Kovaleva, Namrata Shivagunde, and Anna Rumshisky Department of Computer Science University of Massachusetts Lowell {skul,okovalev,nshivagu,arum}@cs.uml.edu Abstract Solving crossword puzzles requires diverse reasoning capabilities, access to a vast amount of knowledge about language and the world, and the ability to satisfy the constraints im- posed by the structure of the puzzle. In this work, we introduce solving crossword puz- zles as a new natural language understand- ing task. We release a corpus of crossword puzzles collected from the New York Times daily crossword spanning 25 years and com- prised of a total of around nine thousand puz- zles. These puzzles include a diverse set of clues: historic, factual, word meaning, syn- onyms/antonyms, fill-in-the-blank, abbrevia- tions, prefixes/suffixes, wordplay, and cross- lingual, as well as clues that depend on the answers to other clues. We separately release the clue-answer pairs from these puzzles as an open-domain question answering dataset containing over half a million unique clue- answer pairs. For the question answering task, our baselines include several sequence-to- sequence and retrieval-based generative mod- els. We also introduce a non-parametric con- straint satisfaction baseline for solving the en- tire crossword puzzle. Finally, we propose an evaluation framework which consists of sev- eral complementary performance metrics. 1 Introduction Recent breakthroughs in NLP established high stan- dards for the performance of machine learning methods across a variety of tasks. However, even state-of-the-art models demonstrate fragility (Wal- lace et al., 2019) and exhibit sensitivity to shallow data patterns (McCoy et al., 2019; Zellers et al., 2019; Jin et al., 2020; Si et al., 2019; Sugawara et al., 2020; Yogatama et al., 2019; Niven and Kao, 2019). This has led to a growing demand for suc- cessively more challenging tasks. One of the important tasks in natural language understanding is question answering (QA), with many recent datasets created to address different different aspects of this task (Yang et al., 2018; Rajpurkar et al., 2016; Kwiatkowski et al., 2019a; Zellers et al., 2019; Dua et al., 2019; Rogers et al., 2021). There are two main forms of question an- swering (QA): extractive QA and open-domain QA. In extractive QA, a passage that answers the ques- tion is provided as input to the system along with the question. In open-domain QA, only the ques- tion is provided as input, and the answer must be generated either through memorized knowledge or via some form of explicit information retrieval over a large text collection which may contain answers. The task of answering clues in a crossword is a form of open-domain question answering. Once a human or an open-domain QA system generates a few possible answer candidates for each clue, one of these candidates may form the correct answer to a word slot in the crossword grid, if the candidate meets …

Content truncated to 500 words