Discovering coreference using image-grounded verb models

Proceedings of Recent Advances in Natural Language Processing, pages 610–615, Hissar, Bulgaria, 12-14 September 2011. Discovering Coreference Using Image-Grounded Verb Models Amitabha Mukerjee Dept. of CS, IITK, India amit@cse.iitk.ac.in Kruti Neema Dept. of CS, IITK, India krutineema@gmail.com Sushobhan Nayak Dept. of EE, IITK, India snayak@iitk.ac.in Abstract Breaking away from traditional attempts at coreference resolution from discourse- only inputs, we try to do the same by constructing rich verb semantics from per- ceptual data, viz. a 2-D video. Using a bottom-up dynamic attention model and relative-motion-features between agents in the video, transitive verbs, their argument ordering etc. are learned through asso- ciation with co-occurring adult commen- tary. This leads to learning of synony- mous NP phrases as well as anaphora such as “it”,“each other” etc. This preliminary demonstration argues for a new approach to developmental NLP, with multi-modal semantics as the basis for computational language learning. 1 Introduction It is common in discourse to refer to the same object using many phrases. For example, in a shared scene with two square shapes (Figure 1), the larger square may be called “the big box”, “the square” or by anaphoric references such as “it”, “itself”, etc. Resolving the many types of co-reference remains a challenging problem in NLP (Stoyanov et al.(2009)). There are increas- ing calls for mechanisms with direct semantic in- terpretation, learned from multimodal input (Roy and Reiter(2005)). This work is posited along such lines; it does not attempt to resolve corefer- ences, but merely to illustrate how knowledge re- lating verb argument structure to the visual action schemas may be learned from multi-modal input. The possibility hinted at is that such grounds-up learning driven NLP systems may eventually have a rich enough library of syntacto-semantic struc- ture to handle coreference more fully. Present at- tempts at analyzing multimodal interfaces (Fang Figure 1: Multimodal input: 2D video “Chase”: Three shapes, [big-square], [small-square] and [circle] interact playfully (velocities shown with aroows). et al.(2009); Steels(2003)) aim to identify the ref- erents in interaction discourse, whereas our ob- jectives are to build a system that can learn the principles of coreference, particularly anaphora. Furthermore, models that consider actions often use prior knowledge for visual parsing of actions (Dominey and Boucher(2005)). With reference to the work on resolving coreference problems, such models typically encode considerable struc- tural knowledge of the linguistic and visual do- mains. Our work proposes mechanisms whereby these structures may be learned. (a) (b) (c) Figure 2: Computed bottom-up attention dur- ing the part of the action [chase(big-square,small- square)]. 2 Learning Action Models and Argument Structure Here we consider how an unsupervised process may acquire action structures from simple videos by clustering frequently observed sequences of motions. The perceptual database in the present 610 model is a single 2-D video (from Heider and Simmel(1944)) (Figure 1). Here, the referent ob- jects (a big square, a small square and a circle) are moving around, interacting with each other, and are easily segmented, as opposed to static referents in game-like contexts used in other …

Content truncated to 500 words