Enhancing the AI2 Diagrams Dataset Using Rhetorical Structure Theory

Enhancing the AI2 Diagrams Dataset Using Rhetorical Structure Theory Tuomo Hiippala and Serafina Orekhova Department of Languages, Digital Geography Lab University of Helsinki, Finland tuomo.hiippala, serafina.orekhova @helsinki.fi Abstract This paper describes ongoing work on a multimodal resource based on the Allen Institute AI2 Diagrams (AI2D) dataset, which contains nearly 5000 grade-school level science diagrams that have been annotated for their elements and the semantic relations that hold between them. This emerging resource, named AI2D-RST, aims to provide a drop-in replacement for the annotation of semantic relations between diagram elements, whose description is informed by recent theories of multimodality and text-image relations. As the name of the resource suggests, the revised annotation schema is based on Rhetorical Structure Theory (RST), which has been previously used to describe the multimodal structure of diagrams and entire documents. The paper documents the proposed annotation schema, describes challenges in applying RST to diagrams, and reports on inter-annotator agreement for this task. Finally, the paper discusses the use of AI2D-RST for research on multimodality and artificial intelligence. Keywords: multimodality, diagrams, rhetorical structure theory, artificial intelligence 1. Introduction The Allen Institute AI2 Diagrams dataset (hereafter AI2D)1 contains nearly 5 000 grade-school level science diagrams, which have been annotated for their elements and the se- mantic relations that hold between them (Kembhavi et al., 2016). The AI2D dataset was initially developed with two emerging computer vision tasks in mind: diagram under- standing and visual question answering. Both tasks are challenging, as diagrams are inherently multimodal: they frequently combine various diagrammatic elements, such as arrows and lines with natural language, two-dimensional graphic elements, illustrations and photographs, and draw a multitude of semantic relations between them. AI2D describes the semantic relations between diagram el- ements using a set of relations drawn from the framework for describing diagrams proposed in Engelhardt (2002). This paper proposes an alternative scheme for describing the semantic relations that hold between diagram elements, building on Rhetorical Structure Theory, or RST for short (Mann and Thompson, 1988; Taboada and Mann, 2006). To distinguish the original AI2D dataset from the emerging language resource described in this paper, we will adopt the name AI2D-RST to refer to the RST-enhanced dataset in the following discussion. Although RST was originally developed for describing the organization of entire texts, the framework has also been applied to the generation of diagrammatic representations (Andr´e and Rist, 1995; Bateman et al., 2001). More re- cently, RST has been used to describe discourse struc- tures in various multimodal artefacts ranging from newspa- pers and journals (Kong, 2013; Taboada and Habel, 2013) and product packaging (Thomas, 2014) to tourist brochures (Hiippala, 2015) and health care posters (Zhang, 2018). When combined with additional layers of description to capture the logical structure of content, its visual appear- ance and layout, even relatively small RST-annotated cor- 1Available at http://allenai.org/plato/diagram-understanding/ (accessed 13 February 2018). pora have been able to reveal patterns characteristic to the structure of the aforementioned multimodal artefacts. This suggests that RST can provide descriptions that are sufficiently fine-grained …

Content truncated to 500 words