Spelling convention sensitivity in neural language models

Findings of the Association for Computational Linguistics: EACL 2023, pages 1334–1346 May 2-6, 2023 ©2023 Association for Computational Linguistics Spelling convention sensitivity in neural language models Elizabeth Nielsen† Christo Kirov◦ Brian Roark◦ †School of Informatics, University of Edinburgh, UK ◦Google e.nielsen@ed.ac.uk {ckirov,roark}@google.com Abstract We examine whether large neural language models, trained on very large collections of varied English text, learn the potentially long- distance dependency of British versus Ameri- can spelling conventions, i.e., whether spelling is consistently one or the other within model- generated strings. In contrast to long-distance dependencies in non-surface underlying struc- ture (e.g., syntax), spelling consistency is eas- ier to measure both in LMs and the text cor- pora used to train them, which can provide ad- ditional insight into certain observed model be- haviors. Using a set of probe words unique to either British or American English, we first establish that training corpora exhibit substan- tial (though not total) consistency. A large T5 language model does appear to internalize this consistency, though only with respect to observed lexical items (not nonce words with British/American spelling patterns). We fur- ther experiment with correcting for biases in the training data by fine-tuning T5 on syn- thetic data that has been debiased, and find that finetuned T5 remains only somewhat sensitive to spelling consistency. Further experiments show GPT2 to be similarly limited. 1 Introduction The probabilities that neural language models (LMs) assign to strings can be used to assess how effectively they capture linguistic dependencies found in their training data. Much as in psycholin- guistic experiments on human language speak- ers, we can present LMs with strings both with and without agreement in key dependencies and measure the assigned probabilities to determine whether the model has learned these linguistic gen- eralizations or not (see e.g., Futrell et al. 2018). For example, sentences both with and without sub- ject/verb number agreement (but otherwise identi- cal) can be used to assess whether the model ac- counts for that particular dependency, even over long distances. Various long-distance dependen- cies have been investigated in this manner, from purely linguistic phenomena such as syntactic de- pendencies (e.g., Gulordava et al. 2018) to extra- linguistic phenomena such as socio-cultural biases (e.g., Rudinger et al. 2018). In this paper, we examine dependencies based on orthographic cues to language variety. Many LMs are trained on large corpora scraped from the web, and data from different language varieties are often combined. For example, LMs trained on web- scraped English (e.g., the WebText Corpus of Rad- ford et al. 2019) encounter British English, North American English, and multiple World Englishes. Likewise, Spanish web corpora may include sev- eral distinct varieties of Latin American Spanish, as well as Iberian Spanish (e.g., Kilgarriff and Renau 2013). Here we use differences between British and American English spelling conventions to ask whether LMs trained on large and diverse collec- tions of English learn to apply these conventions consistently within the same span of text. For ex- ample, if the British spelling of the …

Content truncated to 500 words