TallVocabL2Fi: A Tall Dataset of 15 Finnish L2 Learners’ Vocabulary

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6377–6386 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6377 TallVocabL2Fi: A Tall Dataset of 15 Finnish L2 Learners’ Vocabulary Frankie Robertson†, Li-Hsin Chang♣, Sini S¨oyrinki♠ †Faculty of Information Technology, University of Jyv¨askyl¨a, Finland ♣TurkuNLP group, Department of Computing, Faculty of Technology, University of Turku, Finland ♠Centre for Applied Language Studies, University of Jyv¨askyl¨a, Finland frankie.r.robertson@jyu.fi, lhchan@utu.fi, sini.t.soyrinki@jyu.fi Abstract Previous work concerning measurement of second language learners has tended to focus on the knowledge of small numbers of words, often geared towards measuring vocabulary size. This paper presents a “tall” dataset containing information about a few learners’ knowledge of many words, suitable for evaluating Vocabulary Inventory Prediction (VIP) techniques, including those based on Computerised Adaptive Testing (CAT). In comparison to previous comparable datasets, the learners are from varied backgrounds, so as to reduce the risk of overfitting when used for machine learning based VIP. The dataset contains both a self-rating test and a translation test, used to derive a measure of reliability for learner responses. The dataset creation process is documented, and the relationship between variables concerning the participants, such as their completion time, their language ability level, and the triangulated reliability of their self-assessment responses, are analysed. The word list is constructed by taking into account the extensive derivation morphology of Finnish, and infrequent words are included in order to account for explanatory variables beyond word frequency. Keywords: word knowledge, word response data, mental lexicon, Finnish, learner data 1. Introduction This paper presents TallVocabL2Fi (Robertson, 2022), a word knowledge response dataset collected from 15 paid L2 Finnish participants. The dataset triangu- lates word knowledge between a “tall” self-rating test of 12 000 words and a smaller translation test of 100 words, chosen based on the results of the self-rating test. The self-rating test consists of participants giving their own assessment on a 5-point scale of how well they know a given prompt word. The full raw response data is deposited in the Language Bank of Finland and made available under the CC0 combination public do- main dedication and license. There is emerging interest in the field of Computational Linguistics in the task of Vocabulary Inventory Predic- tion (VIP). In the VIP task, the input is a sample of a learner’s vocabulary typically as known/unknown re- sponses to a relatively small set of words, and the out- put is a function which can predict known/unknown for any input word. However, thus far, there is very little data which can be used for evaluation of the task. To evaluate VIP, we need as much information about the vocabulary of each participant as possible. The first reason for this is the validity of the evaluation metrics. Having many words is not only likely to increase sta- tistical power, but also to ensure inclusion of a wide va- riety of words, rather than selecting only a few bench- mark words, which could bias the evaluation. …

Content truncated to 500 words