Word Discriminations for Vocabulary Inventory Prediction

Proceedings of Recent Advances in Natural Language Processing, pages 1188–1195 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_134 1188 Word Discriminations for Vocabulary Inventory Prediction Frankie Robertson University of Jyväskylä frankie@robertson.name Abstract The aim of vocabulary inventory prediction is to predict a learner’s whole vocabulary based on a limited sample of query words. This paper approaches the problem starting from the 2- parameter Item Response Theory (IRT) model, giving each word in the vocabulary a difficulty and discrimination parameter. The discrimina- tion parameter is evaluated on the sub-problem of question item selection, familiar from the fields of Computerised Adaptive Testing (CAT) and active learning. Next, the effect of the discrimination parameter on prediction perfor- mance is examined, both in a binary classifi- cation setting, and in an information retrieval setting. Performance is compared with base- lines based on word frequency. A number of different generalisation scenarios are examined, including generalising word difficulty and dis- crimination using word embeddings with a pre- dictor network and testing on out-of-dataset data. 1 Introduction Given a small sample of words, how well can we predict whether a learner knows some out-of- sample word? This is the task of vocabulary inven- tory prediction. A clear motivation for the topic is to enable quicker and more precise placement testing. For example, a 40 word self-assessed word knowledge quiz used as a benchmark in this paper is quick enough that an L2 learner returning to a language learning app after a long break, in which they may have either forgotten a lot or had a lot of extra exposure to their target language, can be placed again quickly without excessive disruption. This paper addresses the following research questions: 1. What are the empirical differences in per- formances between difficulty parameters pro- duced by estimation of Item Response The- ory (IRT) models and those based on word frequency in terms of their application to vo- cabulary inventory prediction? 2. How well can the IRT parameters of diffi- culty and discrimination be regressed based on word embeddings? 3. Which approaches from the field of Comput- erised Adaptive Testing (CAT) help to select good items to query? Does the addition of a discrimination parameter help with question selection? 4. Does the addition of a discrimination parame- ter help with the final prediction step? 2 Related Work Milton (2009) refers to the common assump- tion when quantifying vocabulary acquisition that words are learnt in approximately descending or- der of frequency as the frequency assumption. It has been used in the field of reading research, for example in estimating vocabulary size, but can also provide a simple baseline for the task of vocabulary inventory prediction. Avdiu et al. (2019) approached the problem through feature engineering, taking frequency pro- files of different genres and associating learners with them according to their responses. They used a large section of the data for training, without testing a scenario in which learned data is to be generalised to new learners with less data available, as in this paper Item Response …

Content truncated to 500 words