Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5276 - 5290 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns Daniel Wiechmann University of Amsterdam d.wiechmann@uva.nl Yu Qiao RWTH-Aachen University yu.qiao@rwth-aachen.de Elma Kerz RWTH-Aachen University elma.kerz@ifaar.rwth-aachen.de Justus Mattern RWTH-Aachen University justus.mattern@rwth-aachen.de Abstract There is a growing interest in the combined use of NLP and machine learning methods to predict gaze patterns during naturalistic reading. While promising results have been obtained through the use of transformer-based language models, little work has been un- dertaken to relate the performance of such models to general text characteristics. In this paper we report on experiments with two eye-tracking corpora of naturalistic reading and two language models (BERT and GPT- 2). In all experiments, we test effects of a broad spectrum of features for predicting hu- man reading behavior that fall into five cat- egories (syntactic complexity, lexical rich- ness, register-based multiword combinations, readability and psycholinguistic word prop- erties). Our experiments show that both the features included and the architecture of the transformer-based language models play a role in predicting multiple eye-tracking mea- sures during naturalistic reading. We also report the results of experiments aimed at de- termining the relative importance of features from different groups using SP-LIME. 1 Introduction Extensive studies using eye-trackers to observe gaze patterns have shown that humans read sen- tences efficiently by performing a series of fixa- tions and saccades (for comprehensive overviews, see, e.g. Rayner et al. (2012), Seidenberg (2017), and Brysbaert (2019)). During a fixation, the eyes stay fixed on a word and remain fairly static for 200-250 milliseconds. Saccades are rapid jumps between fixations that typically last 20-40 ms and span 7-9 characters. In addition, when reading, humans do not fixate one word at a time, i.e. some saccades run in the opposite direction, and some words or word combinations are fixed more than once or skipped altogether. Much of the early work in this area was concerned with the care- ful construction of sentences to model human reading behavior and understand predictive lan- guage processing (Staub, 2015; Kuperberg and Jaeger, 2016). The use of isolated, decontextu- alized sentences in human language processing research has been questioned on ecological va- lidity grounds. With the growing awareness of the importance of capturing naturalistic reading, new corpora of eye movement data over contigu- ous text segments have emerged. Such corpora serve as a valuable source of data for establish- ing the basic benchmarks of eye movements in reading and provide an essential testing ground for models of eye movements in reading, such as the E-Z Reader model (Reichle et al., 1998) and the SWIFT model (Engbert et al., 2005). They are also used to evaluate theories of human lan- guage processing in psycholinguistics: For ex- ample, the predictions of two theories of syntac- tic processing complexity (dependency locality theory and surprisal) …

Content truncated to 500 words