Constructing Distributions of Variation in Referring Expression Type from Corpora for Model Evaluation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2989–2997 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2989 Constructing Distributions of Variation in Referring Expression Type from Corpora for Model Evaluation T. Mark Ellison, Fahime Same University of Cologne Cologne, Germany {t.m.ellison, f.same}@uni-koeln.de Abstract The generation of referring expressions (REs) is a non-deterministic task. However, the algorithms for the generation of REs are standardly evaluated against corpora of written texts which include only one RE per each reference. Our goal in this work is firstly to reproduce one of the few studies taking the distributional nature of the RE generation into account. We add to this work, by introducing a method for exploring variation in human RE choice on the basis of longitudinal corpora - substantial corpora with a single human judgement (in the process of composition) per RE. We focus on the prediction of RE types, proper name, description and pronoun. We compare evaluations made against distributions over these types with evaluations made against parallel human judgements. Our results show agreement in the evaluation of learning algorithms against distributions constructed from parallel human evaluations and from longitudinal data. Keywords: Referring Expression, Prediction, Evaluation, Variation 1. Introduction Referring is a fundamental part of communication and has been extensively investigated in theoretical, em- pirical and computational linguistic studies. Refer- ring Expression Generation (REG), as one of the main components of a Natural Language Generation (NLG) pipeline, tackles two different tasks: the first task, or one-shot REG, is concerned with finding a set of at- tributes to single out a particular referent from a set of distractors. The second task, which is the focus of this article, involves generating referring expressions (REs) in a discourse context. This task is hereafter referred to as REG-in-context. Belz and Varges (2007) defined REG-in-context as follows: “given an intended refer- ent and a discourse context, how do we generate ap- propriate referential [referring] expressions to refer to the referent at different points in the discourse?” (p. 9). Classic REG-in-context studies often approach REG in two steps. The first step is to decide which form to use in a given context; for example, whether to use a pronoun (e.g. he), a proper name (Barack Obama), or a description (the former president of the United States). The second step determines the content and linguistic realisation of the expressions (Reiter and Dale, 2000; Krahmer and van Deemter, 2012).1 The focus of this paper is on the first step of the classic REG approach, namely the choice of RE form. Algorithms for the generation of REs in context are standardly evaluated against corpora of written texts, offering a single ‘correct’ response in the given con- text. However, the generation of REs is a probabilistic 1With the rapid advancement of neural approaches, end- to-end REG models are also gaining popularity where the de- cision about form and content is made simultaneously (Cas- tro Ferreira et al., 2018; Cao and Cheung, 2019; …

Content truncated to 500 words