The 2023 WebNLG Shared Task on Low Resource Languages. Overview and Evaluation Results (WebNLG 2023)

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 55–66 September 12, 2023 ©2023 Association for Computational Linguistics The 2023 WebNLG Shared Task on Low Resource Languages Overview and Evaluation Results (WebNLG 2023) Liam Cripwell3,4, Anya Belz1,6, Claire Gardent3, Albert Gatt5, Claudia Borg2, Marthese Borg2, John Judge1, Michela Lorandi1, Anna Nikiforovskaya3,4, William Soto-Martinez3,4, Craig Thomson6 1ADAPT/DCU, Ireland; 2University of Malta, Malta; 3CNRS/LORIA, France; 4Université de Lorraine, France; 5Utrecht University, the Netherlands; 6University of Aberdeen, UK Abstract The WebNLG task consists of mapping a knowledge graph to a text verbalising the con- tent of that graph. The 2017 WebNLG edi- tion required participating systems to gener- ate English text from a set of DBpedia triples, while the 2020 WebNLG+ challenge addition- ally included generation into Russian and se- mantic parsing of English and Russian texts. In contrast, WebNLG 2023 focuses on four under-resourced languages which are severely under-represented in research on text genera- tion, namely Breton, Irish, Maltese and Welsh. In addition, WebNLG 2023 once again includes Russian. In this paper, we present the organi- sation of the shared task (data, timeline, eval- uation), briefly describe the participating sys- tems and summarise results for participating systems. 1 Introduction The WebNLG challenges seeks to add to the re- search on Knowledge Graph verbalisation i.e., how to convert a knowledge graph into a text verbalising its content. This is illustrated in Figure 1 where the text shown conveys the content of the input graph. Alan Bean graduated from UT Austin in 1955 with a Bachelor of Science degree. He was hired by NASA in 1963 and served as a test pilot. Apollo 12’s backup pilot was Alfred Worden and was com- manded by David Scott. Figure 1: WebNLG Input/Output Example: the gener- ated text should convey the content of the input graph. The first edition of the challenge, the 2017 WebNLG shared task, required participating sys- tems to generate English text from a set of DB- pedia triples (Gardent et al., 2017). The 2020 WebNLG+ challenge encompassed four tasks: RDF-to-English, RDF-to-Russian, English-to-RDF and Russian-to-RDF. With the development of large-scale pretrained models, research in automatic text generation has acquired new impetus. Yet, the current state-of- the-art is dominated by a handful of languages, for which training data is relatively easy to ac- quire. At the same time, the field has recently witnessed some encouraging developments which focus on generation for under-resourced and under- represented languages (Abhishek et al., 2022; Team et al., 2022). This trend is paralleled by a growing interest in multilingual models and applications in NLP more broadly. The WebNLG 2023 Challenge was organised in response to these trends and specifically ad- dresses generation for four under-resourced lan- guages: Breton, Irish, Maltese and Welsh. In addi- tion, WebNLG 2023 once again includes Russian. Timeline. Noisy training data, gold development data and evaluation scripts were released on Febru- ary 24, 2023. The test data was made available on June 8th and the deadline for submitting …

Content truncated to 500 words