ELQA: A Corpus of Metalinguistic Questions and Answers about English

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2031–2047 July 9-14, 2023 ©2023 Association for Computational Linguistics ELQA: A Corpus of Metalinguistic Questions and Answers about English Shabnam Behzad Georgetown University shabnam@cs.georgetown.edu Keisuke Sakaguchi Tohoku University keisuke.sakaguchi@tohoku.ac.jp Nathan Schneider Georgetown University nathan.schneider@georgetown.edu Amir Zeldes Georgetown University amir.zeldes@georgetown.edu Abstract We present ELQA, a corpus of questions and answers in and about the English language. Collected from two online forums, the >70k questions (from English learners and others) cover wide-ranging topics including grammar, meaning, fluency, and etymology. The answers include descriptions of general properties of English vocabulary and grammar as well as ex- planations about specific (correct and incorrect) usage examples. Unlike most NLP datasets, this corpus is metalinguistic—it consists of lan- guage about language. As such, it can facilitate investigations of the metalinguistic capabilities of NLU models, as well as educational applica- tions in the language learning domain. To study this, we define a free-form question answering task on our dataset and conduct evaluations on multiple LLMs (Large Language Models) to an- alyze their capacity to generate metalinguistic answers. 1 Introduction Language is so powerful that it can be reflected back on itself. Statements like “In informal usage, a steep learning curve means something that is dif- ficult (and takes much effort) to learn” or “In some cases, an adjective has both -ic and -ical forms, with no difference in meaning” expressly concern linguistic inventories, structures, and behaviors. In other words, they are metalinguistic—they use lan- guage to discuss language (cf. Wilson, 2013). They may concern a particular instance of language use, or properties of a language or speaker in general; either way, they are metalinguistic in making lin- guistic phenomena the subject matter of a linguistic utterance. For the rest of this paper, the term meta- language is used for natural language text in which natural language is also the subject matter. While NLP models have become powerful at pre- dicting text in many settings, it remains to be seen whether such capability extends to metalanguage— where linguistic strings are not being deployed to contribute to the discourse with their normal deno- tations, but rather, are treated as entities with lin- guistic properties (e.g., grammar, meaning). One way this can be explored is in a question answering framework, which requires suitable datasets, ide- ally based on questions that are realistic and paired with high-quality answers. In this paper, we present a corpus of metalinguis- tic questions and answers about English. The cor- pus is collected and carefully processed from two Stack Exchange forum sites: English Language & Usage (ENG) and English Language Learners (ELL). It covers more than 70k questions on numer- ous topics about English such as grammar, mean- ing, fluency, and etymology along with answers. Our corpus, ELQA (English Language Questions and Answers), can serve as a tool to facilitate met- alinguistic studies. Moreover, since questions in ELQA cover a variety of topics in English, it can be used …

Content truncated to 500 words