Non-neural Models Matter: a Re-evaluation of Neural Referring Expression Generation Systems

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5554 - 5567 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Non-neural Models Matter: A Re-evaluation of Neural Referring Expression Generation Systems Fahime Same♥∗, Guanyi Chen♠∗, and Kees van Deemter♠ ♥Department of Linguistics, University of Cologne ♠Department of Information and Computing Sciences, Utrecht University f.same@uni-koeln.de, g.chen@uu.nl, c.j.vandeemter@uu.nl Abstract In recent years, neural models have often outperformed rule-based and classic Machine Learning approaches in NLG. These classic approaches are now often disregarded, for ex- ample when new neural models are evaluated. We argue that they should not be overlooked, since for some tasks, well-designed non-neural approaches achieve better performance than neural ones. In this paper, the task of gen- erating referring expressions in linguistic con- text is used as an example. We examined two very different English datasets (WEBNLG and WSJ), and evaluated each algorithm us- ing both automatic and human evaluations. Overall, the results of these evaluations sug- gest that rule-based systems with simple rule sets achieve on-par or better performance on both datasets compared to state-of-the-art neu- ral REG systems. In the case of the more re- alistic dataset, WSJ, a machine learning-based system with well-designed linguistic features performed best. We hope that our work can encourage researchers to consider non-neural models in future. 1 Introduction Natural Language Generation (NLG) is concerned with the generation of natural language text from non-linguistic input (Gatt and Krahmer, 2018). One step in a classic generation pipeline (Reiter and Dale, 2000) is Referring Expression Genera- tion (REG, Krahmer and van Deemter (2019) for an overview). REG has important practical value for commercial natural language generation (Re- iter, 2017), computer vision (Mao et al., 2016), and robotics (Fang et al., 2015), for example. It has also been used as a tool to understand human language use (van Deemter, 2016). REG contains two different problems. One is to find a set of at- tributes to single out a referent from a set (also ∗ Equal contribution. Order determined by swapping the order in Chen et al. (2021). called one-shot REG). The other is to generate re- ferring expressions (REs) to refer to a referent at different points in a discourse (Belz and Varges, 2007). We will focus on the latter task. We call this the REG-in-context task. In earlier works, REG is often tackled in two steps (Henschel et al., 2000; Krahmer and Theune, 2002). The first step decides the form of an RE. For example, whether a reference should be a proper name (“Marie Skłodowska-Curie”), a description (“the physicist”), or a pronoun (“she”) at a given point in the context. The second step is concerned with content selection, i.e., the different ways in which a referential form can be realised. For exam- ple, to generate a description of Marie Curie, the REG system decides whether it is sufficient to men- tion her profession (i.e., “the physicist”) or whether it is better to mention her nationality as well …

Content truncated to 500 words