Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1196–1206 August 1–6, 2021. ©2021 Association for Computational Linguistics 1196 Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability Venelin Kovatchev1, Phillip Smith2, Mark Lee2, and Rory Devine1 1 School of Psychology, University of Birmingham 2 School of Computer Science, University of Birmingham 52 Prichatts Road, Edgbaston, B15 2SB, United Kingdom {v.o.kovatchev , P.Smith.7 , m.g.lee , R.T.Devine} @bham.ac.uk Abstract In this paper we implement and compare 7 dif- ferent data augmentation strategies for the task of automatic scoring of children’s ability to un- derstand others’ thoughts, feelings, and desires (or “mindreading”). We recruit in-domain experts to re-annotate augmented samples and determine to what ex- tent each strategy preserves the original rating. We also carry out multiple experiments to mea- sure how much each augmentation strategy im- proves the performance of automatic scoring systems. To determine the capabilities of auto- matic systems to generalize to unseen data, we create UK-MIND-20 - a new corpus of chil- dren’s performance on tests of mindreading, consisting of 10,320 question-answer pairs. We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro- F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The task- specific augmentations generally outperform task-agnostic augmentations. Automatic aug- mentations based on vectors (GloVe, FastText) perform the worst. We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demon- strate that data augmentation strategies also improve the performance on unseen data. 1 Introduction Many state-of-the-art NLP models are limited by the availability of high quality human-annotated training data. The process of gathering and annotat- ing additional data is often expensive and time con- suming. It is especially difficult to gather data for tasks within psychology and psycholinguistics, as test administration typically requires highly trained in-domain experts, controlled environments, and large numbers of human participants. Data augmentation is a popular technique for artificially enlarging datasets. Typically, data aug- mentation uses one or more predefined strategies to modify existing gold-standard examples while retaining the original label. The objectives of data augmentation are: 1) to increase the size of the dataset; 2) to introduce more variety; 3) to reduce overfitting; and 4) to improve generalizability. Data augmentation has been used successfully in computer vision (Shorten and Khoshgoftaar, 2019) and has recently become more popular in the field of NLP (Wei and Zou, 2019; Min et al., 2020; Dai and Adel, 2020; Marivate and Sefara, 2020). We use data augmentation to improve the per- formance of systems for automatic scoring of chil- dren’s performance on tests of mindreading (i.e., the ability to reason about others’ thoughts, feel- ings and desires) (Hughes and Devine, 2015). Au- tomatic scoring of mindreading was recently intro- duced by Kovatchev et al. (2020). Their corpus, MIND-CA contains hand-scored …

Content truncated to 500 words