How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1101–1114 November 20–23, 2022. ©2022 Association for Computational Linguistics 1101 How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection Xiaomeng Ma The Graduate Center, CUNY xma3@gradcenter.cuny.edu Lingyu Gao Toyota Technological Institute at Chicago lygao@ttic.edu Abstract There is an ongoing debate on whether neu- ral networks can grasp the quasi-regularities in languages like humans. In a typical quasi- regularity task, English past tense inflections, the neural network model has long been criti- cized that it learns only to generalize the most frequent pattern, but not the regular pattern, thus can not learn the abstract categories of reg- ular and irregular and is dissimilar to human performance. In this work, we train a set of transformer models with different settings to examine their behavior on this task. The mod- els achieved high accuracy on unseen regular verbs and some accuracy on unseen irregular verbs. The models’ performance on the regu- lars is heavily affected by type frequency and ratio but not token frequency and ratio, and vice versa for the irregulars. The different behaviors on the regulars and irregulars suggest that the models have some degree of symbolic learn- ing on the regularity of the verbs. In addition, the models are weakly correlated with human behavior on nonce verbs. Although the trans- former model exhibits some level of learning on the abstract category of verb regularity, its performance does not fit human data well, sug- gesting that it might not be a good cognitive model. 1 1 Introduction Many aspects of language can be characterized as quasi-regular: the relationship between inputs and outputs is systematic but allow many excep- tions. English past tense inflection exhibits such quasi-regularity that the regular verbs follow the ‘-ed’ rule (help - helped) and the irregular forms consist of a variety of changes such as changing vowel (sing - sang). There has been heated de- bate about how people represent regular and irreg- ular for the past 40 years. For the single-route 1The code and data for this paper can be found at: https://github.com/xiaomeng-ma/English-Past-Tense approach, Rumelhart and McClelland (1986) de- scribed a feed-forward connectionist neural model that learned both regular and irregular forms of the English verbs’ past tense without explicit symbolic rules. However, this model received fierce criti- cisms from the proponents of the dual-route model (e.g. Pinker and Prince, 1988; Marcus et al., 1992), who argue that the speakers first reason over the abstract categories (regular - irregular), and pro- cess the regulars through rule-applying mechanism (adding -ed) and process the irregulars via gradi- ent analogical processes. In addition, Pinker and Prince (1988) highlighted many empirical inade- quacies of the model and argued that these failures stemmed from ‘central features of connectionist ideology’ and would persist in any neural network model. With the advancement of deep learning …

Content truncated to 500 words