Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2106–2122 May 2-6, 2023 ©2023 Association for Computational Linguistics Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples Philipp Sadler1 and David Schlangen1,2 1CoLabPotsdam / Computational Linguistics Department of Linguistics, University of Potsdam, Germany 2German Research Center for Artificial Intelligence (DFKI), Berlin, Germany firstname.lastname@uni-potsdam.de Abstract NLP tasks are typically defined extensionally through datasets containing example instanti- ations (e.g., pairs of image i and text t), but motivated intensionally through capabilities in- voked in verbal descriptions of the task (e.g., “t is a description of i, for which the content of i needs to be recognised and understood”). We present Pento-DIARef, a diagnostic dataset in a visual domain of puzzle pieces where re- ferring expressions are generated by a well- known symbolic algorithm (the “Incremental Algorithm”), which itself is motivated by ap- peal to a hypothesised capability (eliminating distractors through application of Gricean max- ims). Our question then is whether the exten- sional description (the dataset) is sufficient for a neural model to pick up the underlying regu- larity and exhibit this capability given the sim- ple task definition of producing expressions from visual inputs. We find that a model sup- ported by a vision detection step and a targeted data generation scheme achieves an almost per- fect BLEU@1 score and sentence accuracy, whereas simpler baselines do not. 1 Introduction Being able to effectively and efficiently refer to ob- jects is a central component of human language competence (van Deemter, 2016). The compu- tational task of referring expression generation (REG) goes beyond the production of image de- scriptions (as in image captioning), in that it is a uniquely identifying description that needs to be produced, given a specific situation. In the formula- tion of Krahmer and van Deemter (2012), the REG task involves reasoning over all relevant objects in a scene, in order to determine what would make a description uniquely identifying. Additionally, maxims of efficiency (Grice, 1967) predict that it is a minimal natural language expression that should be preferred. The Incremental Algorithm (IA) (Dale and Reiter, 1995) is a well-known classic symbolic Figure 1: An example board with a referring expres- sion (b) as produced by the Incremental Algorithm (IA) (minimal wrt. a preference order); and an unnecessarily verbose reference (a) (that still is uniquely referring). The reference target is highlighted with bounding box; images regions separated by addition of lines. algorithm that tries to realise these desiderata. For example given a reference target and various dis- tractors as in Figure 1 (an example of the domain chosen in this paper (Pentomino, Golomb (1996); Zarrieß et al. (2016); Kennington and Schlangen (2017))), then the Incremental Algorithm (IA) pro- duces “Take the X”, achieving the desired uniquely identifying reference by mentioning only the shape and not also color and position. Can such a reference strategy be learned by neu- ral generation models from visual inputs alone? This is …

Content truncated to 500 words