Sequence Classification with Human Attention

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 302–312 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 302 Sequence classification with human attention Maria Barrett1 Joachim Bingel2 Nora Hollenstein3 Marek Rei4 Anders Søgaard2 1Department of Nordic Studies and Linguistics, University of Copenhagen, Denmark 2Department of Computer Science, University of Copenhagen, Denmark 3Department of Computer Science, ETH Zurich, Switzerland 4Department of Computer Science and Technology, University of Cambridge, United Kingdom barrett@hum.ku.dk noraho@ethz.ch{bingel, soegaard}@di.ku.dk marek.rei@cl.cam.ac.uk Abstract Learning attention functions requires large volumes of data, but many NLP tasks sim- ulate human behavior, and in this paper, we show that human attention really does pro- vide a good inductive bias on many atten- tion functions in NLP. Specifically, we use estimated human attention derived from eye- tracking corpora to regularize attention func- tions in recurrent neural networks. We show substantial improvements across a range of tasks, including sentiment analysis, grammat- ical error detection, and detection of abusive language. 1 Introduction When humans read a text, they do not attend to all its words (Carpenter and Just, 1983; Rayner and Duffy, 1988). For example, humans are likely to omit many function words and other words that are predictable in context and focus on less pre- dictable content words. Moreover, when they fix- ate on a word, the duration of that fixation depends on a number of linguistic factors (Clifton et al., 2007; Demberg and Keller, 2008). Since learning good attention functions for re- current neural networks requires large volumes of data (Zoph et al., 2016; Britz et al., 2017), and er- rors in attention are known to propagate to classi- fication decisions (Alkhouli et al., 2016), we ex- plore the idea of using human attention, as esti- mated from eye-tracking corpora, as an inductive bias on such attention functions. Penalizing atten- tion functions for departing from human attention may enable us to learn better attention functions when data is limited. Eye-trackers provide millisecond-accurate records on where humans look when they are reading, and they are becoming cheaper and more easily available by the day (San Agustin et al., 2009). In this paper, we use publicly available eye-tracking corpora, i.e., texts augmented with eye-tracking measures such as fixation duration times, and large eye-tracking corpora have ap- peared increasingly over the past years. Some studies suggest that the relevance of text can be inferred from the gaze pattern of the reader (Salojärvi et al., 2003) – even on word-level (Loboda et al., 2011). Contributions We present a recurrent neural architecture with attention for sequence classi- fication tasks. The architecture jointly learns its parameters and an attention function, but can alternate between supervision signals from labeled sequences (with no explicit supervision of the attention function) and from attention trajectories. This enables us to use per-word fixation durations from eye-tracking corpora to regularize attention functions for sequence classification tasks. We show such regularization leads to significant improvements across a range of tasks, including sentiment analysis, detection of abusive …

Content truncated to 500 words