Multi3Generation: Multitask, Multilingual, Multimodal Language Generation

Multi3Generation: Multitask, Multilingual, Multimodal Language Generation Anabela Barreiro1 José GC de Souza2 Albert Gatt3,4 Mehul Bhatt5 Elena Lloret6 Aykut Erdem7 Dimitra Gkatzia8 Helena Moniz9,1 Irene Russo10 Fabio Kepler2 Iacer Calixto11 Marcin Paprzycki12 François Portet 13 Isabelle Augenstein14 Mirela Alhasani15 1INESC-ID, Portugal 2Unbabel, Portugal 3University of Malta, Malta 4Utrecht University, The Netherlands 5Örebro University, Sweden 6University of Alicante, Spain 7Koç University, Turkey 8Edinburgh Napier University, United Kingdom 9University of Lisbon, Portugal 10National Research Council, Italy 11Amsterdam University Medical Centers, The Netherlands 12Polish Academy of Sciences, Poland 14University of Copenhagen, Denmark13Grenoble Alpes University, France 15Epoka University, Albania anabela.barreiro@inesc-id.pt Abstract This paper presents the Multitask, Mul- tilingual, Multimodal Language Genera- tion COST Action – Multi3Generation (CA18231), an interdisciplinary network of research groups working on different as- pects of language generation. This "meta- paper" will serve as reference for citations of the Action in future publications. presents the objectives, challenges and aIt the links for the achieved outcomes. 1 Introduction Multi3Generation1 fosters the development of a network of researchers and technologists across in- terdisciplinary fields working on topics related to language generation (LG). We frame LG broadly as the set of tasks where the ultimate goal in- volves generating language. In contrast to the more classical definition of natural language gen- eration (NLG), this also includes tasks not con- cerned with LG in an immediate sense, but that can inform or improve LG models. The action focuses on four core challenges: (a) data and information representation challenges, such as those involving inputs of different sources: images, videos, knowl- edge bases (KBs) and graphs; (b) machine learn- ing (ML) challenges of modern approaches, such as mapping of inputs to different correct outputs, e.g. structured prediction and representation learn- ing; (c) interaction in applications of LG, such as © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1https://www.cost.eu/actions/CA18231/. The Action is funded by the European Commission and is running from June 2019 till September 2023. dialogue systems, conversational search interfaces and human-robot interaction due to the uncertainty derived from the changing environment and the non-deterministic fashion of interaction; (d) KB exploitation: structured knowledge is key to nat- ural language processing (NLP) tasks, including NLG, supporting ML methods that require expan- sion, filtering, disambiguation or user adaptation of generated content. The Action addresses these challenges by answering the following questions: 1. How can we efficiently exploit common- sense, world knowledge and multimodal in- formation from various inputs such as KBs, images and videos to address LG tasks such as multimodal machine translation (MT), video description and summarisation? 2. How can ML methods such as multi–task learning (MTL), representation learning and structured prediction be leveraged for LG? 3. How can the models from (1) and (2) be ex- ploited to develop dialogue-based, conversa- tional human-computer and human-robot in- teraction methods? 2 Objectives Multi3Generation created an interdisciplinary Eu- ropean LG research network targeting scientific advances and societal benefits in the following four focus themes: (T1) grounded multimodal rea- …

Content truncated to 500 words