Arrows are the Verbs of Diagrams

Proceedings of the 27th International Conference on Computational Linguistics, pages 3552–3563 Santa Fe, New Mexico, USA, August 20-26, 2018. 3552 Arrows are the Verbs of Diagrams Malihe Alikhani and Matthew Stone Computer Science, Rutgers University malihe.alikhani, matthew.stone@rutgers.edu Abstract Arrows are a key ingredient of schematic pictorial communication. This paper investigates the interpretation of arrows through linguistic, crowdsourcing and machine-learning methodology. Our work establishes a novel analogy between arrows and verbs: we advocate representing ar- rows in terms of qualitatively different structural and semantic frames, and resolving frames to specific interpretations using shallow world knowledge. 1 Introduction Natural communication is multimodal—people get their ideas across not just through words but through gestures, diagrams, illustrations, and even practical activity. Research in discourse has found suggestive evidence for interpretative parallels across modalities; the challenge now is to substantiate these parallels at large scale, by developing broad-coverage cognitive models of multimodal communication. This paper illustrates the promise of new data sets to further such a general understanding of visual communication. In particular, Kembhavi et al. (2016) have curated, annotated and released a sizeable corpus of scientific diagrams in diverse domains. We draw on formal analysis, crowdsourcing, and machine-learning experiments to carry out a systematic study of the interpretation of arrows in these diagrams. The central insight in our work is to give a novel account of arrows using existing linguistic concepts and methodology—specifically, ideas developed for explaining verb meaning in context. We begin in Section 2 with a tour of work on multimodal communication, which has made a strong case for parsing diagrams to organize basic conventional elements into recursive structural relationships, and for interpreting such parses by combining compositional meaning with discourse-based inference. We use this framework in Section 3 to motivate a linguistically-inspired approach to the interpretation of arrows: we characterize arrows into four qualitatively different structures, analogous to verb subcatego- rization frames; we associate each structure with a meaning, analogous to verb frame semantics (Baker et al., 1998); and we describe the contextual supplementation some structures require, by analogy to the co-compositionality of generative lexicon theory (Pustejovsky, 1998, GLT). The remainder of the paper offers empirical evidence in support of our approach. Our first experiments, presented in Section 4, assess our semantic frames for arrows in light of crowd workers’ judgments about the Kembhavi et al. (2016) data set. People label our categories with high agreement, and the categories account for the overwhelming majority of items in the corpus. Moreover, we find that arrows are normally used in one sense per diagram, much as lexical items exhibit one sense per discourse (Gale et al., 1992). Our next experiments, presented in Section 5, assess our approach to contextual interpretation: certain frames must be supplemented with a salient relationship. Assuming one sense per diagram, we train a machine-learning model that predicts this relationship from the textual content of the diagram. The performance of the method corroborates our hypothesis (analogous to GLT) that these relationships can and should be resolved based on relatively shallow …

Content truncated to 500 words