Semantic Frames and Visual Scenes: Learning Semantic Role Inventories from Image and Video Descriptions

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 149–154, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Semantic Frames and Visual Scenes: Learning Semantic Role Inventories from Image and Video Descriptions Ekaterina Shutova Computer Laboratory University of Cambridge, UK es407@cam.ac.uk Andreas Wundsam Big Switch Networks Santa Clara, CA andi@wundsam.net Helen Yannakoudakis ALTA Institute University of Cambridge, UK hy260@cl.cam.ac.uk Abstract Frame-semantic parsing and semantic role labelling, that aim to automatically assign semantic roles to arguments of verbs in a sentence, have recently become an ac- tive strand of research in NLP. However, to date these methods have relied on a pre- defined inventory of semantic roles. In this paper, we present a method to auto- matically learn argument role inventories for verbs from large corpora of text, im- ages and videos. We evaluate the method against manually constructed role inven- tories in FrameNet and show that the vi- sual model outperforms the language-only model and operates with a high precision. 1 Introduction The theory of frame semantics (Fillmore, 1976) postulates that our interpretation of word mean- ings is not limited to isolated concepts, but rather instantiates complex knowledge structures about events and their participants, known as semantic frames. For instance, the COMMERCIAL TRANS- ACTION frame includes elements such as a seller, a buyer, goods and money which can be mapped to higher-level semantic roles such as agent, patient, instrument etc. The verbs linked to this frame are buy, sell, pay, cost and charge, each evoking dif- ferent aspects of the frame. This theory has been implemented in a lexical- semantic resource called FrameNet (Fillmore et al., 2003). Each semantic frame is encoded in FrameNet as a list of lexical units that evoke this frame (typically verbs) and the roles that their se- mantic arguments may take given the scenario rep- resented by the frame. FrameNet has inspired a di- rection in NLP research known as semantic role la- belling (Gildea and Jurafsky, 2002; M`arquez et al., 2008) and frame-semantic parsing (Das et al., 2014), whose goal is to assign semantic roles to arguments of the verbs in a sentence. However, these works point out the coverage limitations of the hand-constructed FrameNet database, suggest- ing that a data-driven frame acquisition method is needed to enable the integration of frame se- mantics into real-world NLP applications. In this paper, we propose such a method, experimenting with semantic frame induction from linguistic and visual data. Our system first performs clustering of verb arguments to identify their possible seman- tic roles and then computes the level of associa- tion between a given argument role and the verb, thus deriving the structure of the semantic frame in which the verb participates. Frame semantics emphasizes the relation be- tween our lexical semantic knowledge and our ex- perience in the world, suggesting that semantic frames are not merely a linguistic construct but also a result of our sensory-motor and perceptual experience. However, frame semantic approaches in NLP typically rely on …

Content truncated to 500 words