CITE: A Corpus of Image-Text Discourse Relations

Proceedings of NAACL-HLT 2019, pages 570–575 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 570 CITE: A Corpus of Image–Text Discourse Relations Malihe Alikhani Computer Science Rutgers University malihe.alikhani@rutgers.edu Sreyasi Nag Chowdhury Max Planck Institute for Informatics sreyasi@mpi-inf.mpg.de Gerard de Melo Computer Science Rutgers University gerard.demelo@rutgers.edu Matthew Stone Computer Science Rutgers University matthew.stone@rutgers.edu Abstract This paper presents a novel crowd-sourced resource for multimodal discourse: our re- source characterizes inferences in image–text contexts in the domain of cooking recipes in the form of coherence relations. Like previ- ous corpora annotating discourse structure be- tween text arguments, such as the Penn Dis- course Treebank, our new corpus aids in es- tablishing a better understanding of natural communication and common-sense reasoning, while our findings have implications for a wide range of applications, such as understanding and generation of multimodal documents. 1 Introduction “Sometimes a picture is worth the proverbial thou- sand words; sometimes a few well-chosen words are far more effective than a picture” – Feiner and McKeown (1991). Modeling how visual and lin- guistic information can jointly contribute to coher- ent and effective communication is a longstanding open problem with implications across cognitive science. As Feiner and McKeown (1991) already observe, it is particularly important for automating the understanding and generation of text–image presentations. Theoretical models have suggested that images and text fit together into integrated presentations via coherence relations that are anal- ogous to those that connect text spans in discourse; see Alikhani and Stone (2018a) and Section 2. This paper follows up this theoretical perspective through systematic corpus investigation. We are inspired by research on text discourse, which has led to large-scale corpora with infor- mation about discourse structure and discourse se- mantics. The Penn Discourse Treebank (PDTB) is one of the most well-known examples (Milt- sakaki et al., 2004; Prasad et al., 2008). However, although multimodal corpora increasingly include discourse relations between linguistic and non- linguistic contributions, particularly for utterances and other events in dialogue (Cuayáhuitl et al., 2015; Hunter et al., 2015), to date there has existed no dataset describing the coherence of text–image presentations. In this paper, we describe the con- struction of an annotated corpus that fills this gap, and report initial analyses of the communicative inferences that connect text and accompanying im- ages in this corpus. As we describe in Section 2, our approach asks annotators to identify the presence of specific in- ferences linking text and images, rather than to use a taxonomy of coherence relations. This enables us to deal with the distinctive discourse contribu- tions of photographic imagery. We describe our data collection process in Section 3, showing that our annotation scheme allows us to get reliable la- bels by crowdsourcing. We present analyses in Section 4 that show that our annotation highlights a range of cases where text and images work to- gether in distinctive and theoretically challenging ways, and discuss the implications of our work for the understanding and generation of multimodal documents. We conclude …

Content truncated to 500 words