Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1493–1503, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes Gordon Christie1,∗, Ankit Laddha2,∗, Aishwarya Agrawal1, Stanislaw Antol1 Yash Goyal1, Kevin Kochersberger1, Dhruv Batra3,1 1Virginia Tech 2Carnegie Mellon University 3Georgia Institute of Technology ankit1991laddha@gmail.com {gordonac,aish,santol,ygoyal,kbk,dbatra}@vt.edu Abstract We present an approach to simultaneously per- form semantic segmentation and prepositional phrase attachment resolution for captioned images. Some ambiguities in language can- not be resolved without simultaneously rea- soning about an associated image. If we con- sider the sentence “I shot an elephant in my pajamas”, looking at language alone (and not using common sense), it is unclear if it is the person or the elephant wearing the pajamas or both. Our approach produces a diverse set of plausible hypotheses for both semantic segmentation and prepositional phrase attach- ment resolution that are then jointly reranked to select the most consistent pair. We show that our semantic segmentation and preposi- tional phrase attachment resolution modules have complementary strengths, and that joint reasoning produces more accurate results than any module operating in isolation. Multiple hypotheses are also shown to be crucial to im- proved multiple-module reasoning. Our vi- sion and language approach significantly out- performs the Stanford Parser (De Marneffe et al., 2006) by 17.91% (28.69% relative) and 12.83% (25.28% relative) in two different ex- periments. We also make small improvements over DeepLab-CRF (Chen et al., 2015). 1 Introduction Perception and intelligence problems are hard. Whether we are interested in understanding an im- * Denotes equal contribution PASCAL Sentence Dataset Consistent NLP: Sentence Parsing Ambiguity: (woman on couch) vs (dog on couch) Output: Parse Tree “A dog is standing next to a woman on a couch” Vision: Semantic Segmentation Labels: Chairs, desks, etc. Person Couch Couch Person Dog Solution #1 Solution #M Ambiguity: (dog next to woman) on couch vs dog next to (woman on couch) Ambiguity: (dog next to woman) on couch vs dog next to (woman on couch) Figure 1: Overview of our approach. We propose a model for simultaneous 2D semantic segmentation and preposi- tional phrase attachment resolution by reasoning about sentence parses. The language and vision modules each produce M diverse hypotheses, and the goal is to select a pair of consistent hypotheses. In this example the am- biguity to be resolved from the image caption is whether the dog is standing on or next to the couch. Both modules benefit by selecting a pair of compatible hypotheses. age or a sentence, our algorithms must operate un- der tremendous levels of ambiguity. When a hu- man reads the sentence “I eat sushi with tuna”, it is clear that the preposition phrase “with tuna” mod- ifies “sushi” and not the act of eating, but this may be ambiguous to a machine. This problem of deter- mining whether a prepositional phrase (“with tuna”) modifies a noun phrase (“sushi”) or verb phrase (“eating”) is formally …

Content truncated to 500 words