Multimodal Frame Identification with Multilingual Evaluation

Proceedings of NAACL-HLT 2018, pages 1481–1491 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Multimodal Frame Identification with Multilingual Evaluation Teresa Botschen§†, Iryna Gurevych∗§†, Jan-Christoph Klie∗†, Hatem Mousselly-Sergieh∗†, Stefan Roth∗§‡ § Research Training Group AIPHES † Ubiquitous Knowledge Processing (UKP) Lab ‡ Visual Inference Lab Department of Computer Science, Technische Universit¨at Darmstadt www.{aiphes, ukp, visinf}.tu-darmstadt.de Abstract An essential step in FrameNet Semantic Role Labeling is the Frame Identification (FrameId) task, which aims at disambiguating a situation around a predicate. Whilst current FrameId methods rely on textual representations only, we hypothesize that FrameId can profit from a richer understanding of the situational con- text. Such contextual information can be ob- tained from common sense knowledge, which is more present in images than in text. In this paper, we extend a state-of-the-art FrameId system in order to effectively leverage multi- modal representations. We conduct a compre- hensive evaluation on the English FrameNet and its German counterpart SALSA. Our anal- ysis shows that for the German data, tex- tual representations are still competitive with multimodal ones. However on the English data, our multimodal FrameId approach out- performs its unimodal counterpart, setting a new state of the art. Its benefits are particularly apparent in dealing with ambiguous and rare instances, the main source of errors of current systems. For research purposes, we release (a) the implementation of our system, (b) our eval- uation splits for SALSA 2.0, and (c) the em- beddings for synsets and IMAGINED words.1 1 Introduction FrameNet Semantic Role Labeling analyzes sen- tences with respect to frame-semantic structures based on FrameNet (Fillmore et al., 2003). Typ- ically, this involves two steps: First, Frame Iden- tification (FrameId), capturing the context around a predicate (frame evoking element) and assigning a frame, basically a word sense label for a pro- totypical situation, to it. Second, Role Labeling, i.e. identifying the participants (fillers) of the pred- icate and connecting them with predefined frame- ∗named alphabetically 1https://github.com/UKPLab/ naacl18-multimodal-frame-identification specific role labels. FrameId is crucial to the suc- cess of Semantic Role Labeling as FrameId errors account for most wrong predictions in current sys- tems (Hartmann et al., 2017). Consequently, im- proving FrameId is of major interest. The main challenge and source of prediction er- rors of FrameId systems are ambiguous predicates, which can evoke several frames, e.g., the verb sit evokes the frame Change posture in a context like ‘a person is sitting back on a bench’, while it evokes Being located when ‘a company is sit- ting in a city’. Understanding the predicate con- text, and thereby the context of the situation (here, ‘Who / what is sitting where?’), is crucial to iden- tifying the correct frame for ambiguous cases. State-of-the-art FrameId systems model the sit- uational context using pretrained distributed word embeddings (see Hermann et al., 2014). Hence, it is assumed that the context of the situation is explicitly expressed in words. However, lan- guage understanding involves implicit knowledge, which is not mentioned but still seems obvious to humans, …

Content truncated to 500 words