Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12722–12736 July 9-14, 2023 ©2023 Association for Computational Linguistics Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model Jakob Prange Hong Kong Polytechnic University jakob.prange@polyu.edu.hk Man Ho Ivy Wong Hong Kong Shue Yan University mhwong@hksyu.edu Abstract We use both Bayesian and neural models to dissect a data set of Chinese learners’ pre- and post-interventional responses to two tests mea- suring their understanding of English prepo- sitions. The results mostly replicate previous findings from frequentist analyses and reveal new and crucial interactions between student ability, task type, and stimulus sentence. Given the sparsity of the data as well as high diversity among learners, the Bayesian method proves most useful; but we also see potential in using language model probabilities as predictors of grammaticality and learnability.1 1 Introduction Learning a second or third language is hard—not only for NLP models but also for humans! Which linguistic properties and external factors make it so difficult? And how can we improve instruction and testing to help learners accomplish their goals? Here we also ask a third question: How can we best apply different computational models to such be- havioral experimental data in order to get intuitive and detailed answers to the first two questions in a practical and efficient way? For example, we are in- terested in whether language model (LM) probabil- ities might give a rough estimate of grammaticality and learning difficulty (table 1, right columns). This work is in part a replication study of Wong (2022), who, in addressing these questions about native Chinese speakers’ learning of English prepo- sitions in context (see examples in table 1), mainly focused on instructional intervention and found generally positive effects as well as differences between instruction types, in particular favoring conceptual over rule-based teaching. We pick up where Wong (2022) left off and search for more fine-grained patterns among students’ individual differences, linguistic items, stimulus sentences, 1Our experimental code is available at https://github. com/jakpra/L2-Prepositions. and their grammaticality. Our main hypothesis is that the full story of the complex interactions among these factors can only be revealed by mod- eling them holistically. Such a fine-grained holistic analysis is well-aligned with Item Response Theory (IRT; Fischer, 1973; Lord, 1980). IRT allows us to formulate models in terms of predicting whether students provide the intended response to each test item. We consider sparse Bayesian and dense neu- ral versions of this framework. We can then inspect how strongly each model input (the linguistic, ex- perimental, and student-specific factors mentioned above, which are realized as random and fixed ef- fects for the Bayesian model and feature vectors for the neural model) affects the outcome. As a repre- sentative of yet another modeling strategy, and also as an additional input to the IRT models, we ob- tain probability estimates for the target prepositions in each stimulus sentence from a pretrained trans- former LM. These probabilities serve as a proxy …

Content truncated to 500 words