Learning Prototypical Event Structure from Photo Albums

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1769–1779, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Learning Prototypical Event Structure from Photo Albums Antoine Bosselut†, Jianfu Chen‡, David Warren‡, Hannaneh Hajishirzi† and Yejin Choi† †Computer Science & Engineering, University of Washington, Seattle, WA {antoineb, hannaneh, yejin}@cs.washington.edu ‡Department of Computer Science, Stony Brook University, Stony Brook, NY {jianchen, warren}@cs.stonybrook.edu Abstract Activities and events in our lives are struc- tural, be it a vacation, a camping trip, or a wedding. While individual details vary, there are characteristic patterns that are specific to each of these scenarios. For ex- ample, a wedding typically consists of a sequence of events such as walking down the aisle, exchanging vows, and dancing. In this paper, we present a data-driven ap- proach to learning event knowledge from a large collection of photo albums. We for- mulate the task as constrained optimiza- tion to induce the prototypical temporal structure of an event, integrating both vi- sual and textual cues. Comprehensive evaluation demonstrates that it is possible to learn multimodal knowledge of event structure from noisy web content. 1 Introduction Many common scenarios in our lives, such as a wedding or a camping trip, show characteristic structural patterns. As illustrated in Figure 1, these patterns can be sequential, such as in a wedding, where exchanging vows generally happens before cutting the cake. In other scenarios, there may be a set of composing events, but no prominent tempo- ral relations. A camping trip, for example, might include events such as hiking, which can happen either before or after setting up a tent. This observation on the prototypical patterns in everyday scenarios goes back to early artificial in- telligence research. Scripts (Schank and Abelson, 1975), an early formalism, were developed to en- code the necessary background knowledge to sup- port an inference engine for common sense rea- soning in limited domains. However, early ap- -Ring&'me.& -Exchanging&our&rings.& -Rings&and&promises.& & Kiss$ -Our&first&ever&kiss.& -You&may&kiss&the&bride.& -Sealed&with&a&kiss.& & Cut$the$cake$ -Cake&cuBng.& -The&cake&was&so&solid.& & -Dancing&excitement.& -First&dance.& -Ballroom&dancing.& reading&vows& presen'ng&rings& best&cake&ever& Photo%albums% down&the&aisle& Prototypical%Cap2ons:%% Exchange$rings$ Dance$ Learned%Events:% -Reading&our&vows.& -Our&vows.& & Vows$ Temporal%Knowledge:% Dance$ …& …& Kiss$ Cake$ Vows$ Rings$ Figure 1: We collect photo albums of common scenar- ios (e.g., weddings) and cluster their images and captions to learn the hierarchical events that make up these scenarios. We use constrained optimization to decode the temporal order of these events, and we extract the prototypical descriptions that define them. proaches based on hand-coded symbolic represen- tations proved to be brittle and difficult to scale. An alternative direction in recent years has been statistical knowledge induction, i.e., learn- ing script or common sense knowledge bottom-up from large-scale data. While most prior work is based on text (Pichotta and Mooney, 2014; Jans et al., 2012; Chambers and Jurafsky, 2008; Cham- bers, 2013), recent work begins exploring the use of images as well (Bagherinezhad et al., 2016; Vedantam et al., 2015). In this paper, we present the first study for learn- ing knowledge about common …

Content truncated to 500 words