Modeling Human Reading with Neural Attention

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 85–95, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Modeling Human Reading with Neural Attention Michael Hahn Frank Keller Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK s1582047@inf.ed.ac.uk keller@inf.ed.ac.uk Abstract When humans read text, they fixate some words and skip others. However, there have been few attempts to explain skipping behav- ior with computational models, as most ex- isting work has focused on predicting read- ing times (e.g., using surprisal). In this pa- per, we propose a novel approach that models both skipping and reading, using an unsuper- vised architecture that combines a neural at- tention with autoencoding, trained on raw text using reinforcement learning. Our model ex- plains human reading behavior as a tradeoff between precision of language understanding (encoding the input accurately) and economy of attention (fixating as few words as possi- ble). We evaluate the model on the Dundee eye-tracking corpus, showing that it accurately predicts skipping behavior and reading times, is competitive with surprisal, and captures known qualitative features of human reading. 1 Introduction Humans read text by making a sequence of fixations and saccades. During a fixation, the eyes land on a word and remain fairly static for 200–250 ms. Sac- cades are the rapid jumps that occur between fixa- tions, typically lasting 20–40 ms and spanning 7– 9 characters (Rayner, 1998). Readers, however, do not simply fixate one word after another; some sac- cades go in reverse direction, and some words are fixated more than once or skipped altogether. A range of computational models have been de- veloped to account for human eye-movements in reading (Rayner and Reichle, 2010), including mod- els of saccade generation in cognitive psychology, such as EZ-Reader (Reichle et al., 1998, 2003, 2009), SWIFT (Engbert et al., 2002, 2005), or the Bayesian Model of Bicknell and Levy (2010). More recent approaches use machine learning mod- els trained on eye-tracking data to predict human reading patterns (Nilsson and Nivre, 2009, 2010; Hara et al., 2012; Matthies and Søgaard, 2013). Both types of models involve theoretical assump- tions about human eye-movements, or at least re- quire the selection of relevant eye-movement fea- tures. Model parameters have to be estimated in a supervised way from eye-tracking corpora. Unsupervised approaches, that do not involve training the model on eye-tracking data, have also been proposed. A key example is surprisal, which measures the predictability of a word in context, de- fined as the negative logarithm of the conditional probability of the current word given the preced- ing words (Hale, 2001; Levy, 2008). Surprisal is computed by a language model, which can take the form of a probabilistic grammar, an n-gram model, or a recurrent neural network. While surprisal has been shown to correlate with word-by-word reading times (McDonald and Shillcock, 2003a,b; Demberg and Keller, 2008; Frank and Bod, 2011; Smith and Levy, 2013), it cannot explain other aspects of …

Content truncated to 500 words