Eye4Ref: A Multimodal Eye Movement Dataset of Referentially Complex Situations

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2396–2404 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2396 Eye4Ref: A Multimodal Eye Movement Dataset of Referentially Complex Situations ¨Ozge Alac¸am, Eugen Ruppert, Amr Rekaby Salama, Tobias Staron, Wolfgang Menzel University of Hamburg, Department of Informatics Vogt-Koelln Strasse 30, 22527, Hamburg, Germany alacam,ruppert,salama,staron,menzel@informatik.uni-hamburg.de Abstract Eye4Ref is a rich multimodal dataset of eye-movement recordings collected from referentially complex situated settings where the linguistic utterances and their visual referential world were available to the listener. It consists of not only fixation parameters but also saccadic movement parameters that are time-locked to accompanying German utterances (with English translations). Additionally, it also contains symbolic knowledge — contextual — representations of the images to map the referring expressions onto the objects in corresponding images. Overall, the data was collected from 59 participants in three different experimental setups (86 systematically controlled sentence–image pairs and 2024 eye-movement recordings). Referential complexity was controlled by visual manipulations (e.g. number of objects in the scene, visibility of the target items, etc.), and by linguistic manipulations (e.g. the position of the disambiguating word in a sentence, of the relative clause attachment). This multimodal dataset – in which the three different sources of information, namely eye-tracking, language, and visual environment, are aligned – provides testing possibilities for various research questions not only from a linguistic but also from a computer vision perspective. Keywords: eye-movement parameters, multimodality, situated language understanding, reference resolution 1. Reference Resolution and Meaning Extraction Limiting communication to a single mode requires com- pleteness of that mode, and spontaneously spoken words typically lack completeness, since we use language in com- bination with other modalities to communicate what we want to convey (Tversky, 2014). Therefore, incorporat- ing other cues in relation with the spoken utterance gets us closer to its meaning. A growing body of literature demon- strates that the human language processing system inte- grates information from various modalities to extract the meaning of the linguistic input accurately, e.g. by resolving references early while the sentence is unfolding and even by re-constructing the meaning from noisy / missing input (MacDonald and Seidenberg, 2006; Altmann and Mirkovi´c, 2009; Knoeferle et al., 2005; Tanenhaus et al., 1995). The facilitating effects of non-linguistic cues like prosodic or contextual cues on the performance of Natural Language Processing (NLP) solutions for reference resolution, disam- biguation or meaning recovery have already been well es- tablished in the literature (Coco and Keller, 2015; Snedeker and Trueswell, 2003; Ferreira et al., 2013; Spivey and Huette, 2013; Louwerse, 2008), also see Alac¸am et al. (2020) for a review. In recent years, with the advancements in eye-tracking technology, incorporating eye movements of a speaker or a listener starts to be seen as another benefi- cial tool in predicting / resolving which entity in a complex visual environment is referred to in the utterances (Mitev et al., 2018; Koleva et al., 2015). Uni-modal approaches like language models (low-level N- gram approaches or …

Content truncated to 500 words