Do You See What I Mean? Visual Resolution of Linguistic Ambiguities

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1477–1487, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Do You See What I Mean? Visual Resolution of Linguistic Ambiguities Yevgeni Berzak CSAIL MIT berzak@mit.edu Andrei Barbu CSAIL MIT andrei@0xab.com Daniel Harari CSAIL MIT hararid@mit.edu Boris Katz CSAIL MIT boris@mit.edu Shimon Ullman Weizmann Institute of Science shimon.ullman@weizmann.ac.il Abstract Understanding language goes hand in hand with the ability to integrate complex contextual information obtained via per- ception. In this work, we present a novel task for grounded language understanding: disambiguating a sentence given a visual scene which depicts one of the possible interpretations of that sentence. To this end, we introduce a new multimodal cor- pus containing ambiguous sentences, rep- resenting a wide range of syntactic, se- mantic and discourse ambiguities, coupled with videos that visualize the different in- terpretations for each sentence. We ad- dress this task by extending a vision model which determines if a sentence is depicted by a video. We demonstrate how such a model can be adjusted to recognize dif- ferent interpretations of the same under- lying sentence, allowing to disambiguate sentences in a unified fashion across the different ambiguity types. 1 Introduction Ambiguity is one of the defining characteristics of human languages, and language understand- ing crucially relies on the ability to obtain un- ambiguous representations of linguistic content. While some ambiguities can be resolved using intra-linguistic contextual cues, the disambigua- tion of many linguistic constructions requires in- tegration of world knowledge and perceptual in- formation obtained from other modalities. In this work, we focus on the problem of grounding language in the visual modality, and in- troduce a novel task for language understanding which requires resolving linguistic ambiguities by utilizing the visual context in which the linguistic content is expressed. This type of inference is fre- quently called for in human communication that occurs in a visual environment, and is crucial for language acquisition, when much of the linguis- tic content refers to the visual surroundings of the child (Snow, 1972). Our task is also fundamental to the problem of grounding vision in language, by focusing on phe- nomena of linguistic ambiguity, which are preva- lent in language, but typically overlooked when using language as a medium for expressing un- derstanding of visual content. Due to such ambi- guities, a superficially appropriate description of a visual scene may in fact not be sufficient for demonstrating a correct understanding of the rel- evant visual content. Our task addresses this issue by introducing a deep validation protocol for vi- sual understanding, requiring not only providing a surface description of a visual activity but also demonstrating structural understanding at the lev- els of syntax, semantics and discourse. To enable the systematic study of visually grounded processing of ambiguous language, we create a new corpus, LAVA (Language and Vision Ambiguities). This corpus contains sentences with linguistic ambiguities that can only be resolved us- ing external information. The sentences are paired with short videos …

Content truncated to 500 words