Bridging Information-Seeking Human Gaze and Machine Reading Comprehension

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 142–152 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 142 Bridging Information-Seeking Human Gaze and Machine Reading Comprehension Jonathan Malmaud MIT BCS malmaud@mit.edu Roger Levy MIT BCS rplevy@mit.edu Yevgeni Berzak MIT BCS berzak@mit.edu Abstract In this work, we analyze how human gaze during reading comprehension is conditioned on the given reading comprehension question, and whether this signal can be beneficial for machine reading comprehension. To this end, we collect a new eye-tracking dataset with a large number of participants engaging in a mul- tiple choice reading comprehension task. Our analysis of this data reveals increased fixation times over parts of the text that are most rel- evant for answering the question. Motivated by this finding, we propose making automated reading comprehension more human-like by mimicking human information-seeking read- ing behavior during reading comprehension. We demonstrate that this approach leads to per- formance gains on multiple choice question an- swering in English for a state-of-the-art read- ing comprehension model. 1 Introduction Much of the work in NLP strives to develop sys- tems that are able to perform linguistic tasks sim- ilarly to humans. To achieve this goal, one typi- cally provides NLP systems with human knowledge about the task at hand. This knowledge can come in the form of linguistic annotations, hand-crafted rules and access to linguistic databases, as well as various model design choices. In this work, we study the possibility of pro- viding the model with an inductive bias by using human behavioral signals based on eye movements in reading as an additional source of information which can guide NLP models to adequately pro- cess linguistic input and solve linguistic tasks. As a case study, we examine reading comprehension, a task of central importance for probing both human and machine understanding of text. To enable this study, we collect eye movement data from 269 par- ticipants who engage in a reading comprehension task using the materials of OneStopQA (Berzak et al., 2020). We argue that reading comprehension is a par- ticularly well-suited task for linking human eye movement information to NLP modelling due to the substantial correspondence between reading times and the relevance of the text segment for answering the question. Hahn and Keller (2018) have shown this correspondence by establishing increased reading times on the correct answer in a question answering task where answers are named entities. Our study generalizes this result to an arbitrary QA setting, and demonstrates longer read- ing times for portions of the text which are most pertinent for answering the question correctly. Building on this observation, we develop a new approach to machine reading comprehension in which the model is directed to mimic human fixa- tion times over the text, given the question. The idea behind this approach is to encourage the model to focus on question-relevant information. Specifi- cally, we introduce a multi-task reading comprehen- sion architecture in which a state-of-the-art trans- former model jointly performs question-answering and prediction …

Content truncated to 500 words