Multimodal Grounding for Language Processing

Proceedings of the 27th International Conference on Computational Linguistics, pages 2325–2339 Santa Fe, New Mexico, USA, August 20-26, 2018. 2325 Multimodal Grounding for Language Processing Lisa Beinborn◦∗3 Teresa Botschen∗△ Iryna Gurevych △ 3 Language Technology Lab, University of Duisburg-Essen △ Ubiquitous Knowledge Processing Lab (UKP) and Research Training Group AIPHES Department of Computer Science, Technische Universit¨at Darmstadt www.ukp.tu-darmstadt.de Abstract This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with re- spect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language. Title and Abstract in German Multimodale konzeptuelle Verankerung f¨ur die automatische Sprachverarbeitung Dieser ¨Uberblick er¨ortert, wie aktuelle Entwicklungen in der automatischen Verarbeitung mul- timodaler Inhalte die konzeptuelle Verankerung sprachlicher Inhalte erleichtern k¨onnen. Die automatischen Methoden zur Verarbeitung multimodaler Inhalte werden zun¨achst hinsichtlich der zugrundeliegenden kognitiven Modelle menschlicher Informationsverarbeitung kategorisiert. Daraus ergeben sich verschiedene Methoden um Repr¨asentationen unterschiedlicher Modalit¨aten miteinander zu kombinieren. Ausgehend von diesen methodischen Grundlagen wird disku- tiert, wie verschiedene Forschungsprobleme in der automatischen Sprachverarbeitung von multi- modaler Verankerung profitieren k¨onnen und welche Herausforderungen sich dabei ergeben. Ein besonderer Schwerpunkt wird dabei auf die multimodale konzeptuelle Verankerung von Verben gelegt, da diese eine wichtige kompositorische Funktion erf¨ullen. 1 Introduction Natural languages are continually developing constructs that include numerous variations and irregulari- ties. Modeling the subtleties of language in a formal, processable way has driven computational linguis- tics for decades. In recent years, distributional approaches have become the most widely accepted solu- tion to model the associative character of word meaning (Harris, 1954; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014). These approaches learn word representations in a high-dimensional vector space based on context patterns in large text collections. Machine learning researchers aim at reducing external knowledge to an absolute minimum and simply interpret language as a continuous stream of characters. From an engineering perspective, these data-driven approaches are highly attrac- tive because they reduce the need of domain experts. From a cognitive perspective, processing language in isolation without information on situational con- text seems to be an overly artificial setup. Human acquisition of semantic representations does not occur based on pure language input. The term conceptual grounding refers to the idea that language is ◦ The work by Lisa Beinborn has been carried out during her affiliation with Ubiquitous Knowledge Processing Lab (UKP) and Research Training Group AIPHES, Technische Universit¨at Darmstadt. ∗ The first and the second authors contributed equally to this work. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 2326 (a) Cross-modal transfer. Information from modality A is aligned to compa- rable information in B. (b) Cross-modal interpretation. Rel- evant information in modality …

Content truncated to 500 words