GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6761–6768 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6761 GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors Masato Hagiwara1 and Masato Mita2, 3 1Octanove Labs, Seattle, WA, USA 2RIKEN AIP, Tokyo, Japan 3Tohoku University, Miyagi, Japan 1masato@octanove.com, 2masato.mita@riken.jp Abstract The lack of large-scale datasets has been a major hindrance to the development of NLP tasks such as spelling correction and grammatical error correction (GEC). As a complementary new resource for these tasks, we present the GitHub Typo Corpus, a large-scale, multilingual dataset of misspellings and grammatical errors along with their corrections harvested from GitHub, a large and popular platform for hosting and sharing git repositories. The dataset, which we have made publicly available, contains more than 350k edits and 65M characters in more than 15 languages, making it the largest dataset of misspellings to date. We also describe our process for filtering true typo edits based on learned classifiers on a small annotated subset, and demonstrate that typo edits can be identified with F1 ∼ 0.9 using a very simple classifier with only three features. The detailed analyses of the dataset show that existing spelling correctors merely achieve an F-measure of approx. 0.5, suggesting that the dataset serves as a new, rich source of spelling errors that complement existing datasets. Keywords: spelling correction, grammatical error correction, GitHub, misspellings, atomic edits, language modeling 1. Introduction Spelling correction (Islam and Inkpen, 2009; Zhou et al., 2017; Etoori et al., 2018) and grammatical error correction (GEC) (Leacock et al., 2010) are two fundamental tasks that have important implications for downstream NLP tasks and for education in general. In recent years, the use of statistical machine translation (SMT) and neural sequence- to-sequence (seq2seq) models has been becoming increas- ingly popular for solving these tasks. Such modern NLP models are usually data hungry and require a large amount of parallel training data consisting of sentences before and after the correction. However, only relatively small datasets are available for these tasks, compared to other NLP tasks such as machine translation. This is especially the case for spelling correction, for which only a small number of datasets consisting of individual misspelled words are avail- able, including the Birkbeck spelling error corpus1 and a list of typos collected from Twitter2. Due to this lack of large-scale datasets, many research stud- ies (Foster and Andersen, 2009; Etoori et al., 2018; Li et al., 2018) resort to automatic generation of artificial errors (also called pseudo-errors). Although such methods are ef- ficient and have seen some success, they do not guarantee that generated errors reflect the range and the distribution of true errors made by humans (Zesch, 2012). As one way to complement this lack of resources, Wikipedia has been utilized as a rich source of textual edits, including typos (Grundkiewicz and Junczys-Dowmunt, 2014; Boyd, 2018; Faruqui et al., 2018). However, the edits harvested from Wikipedia are often …

Content truncated to 500 words