Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer

Proceedings of the 27th International Conference on Computational Linguistics, pages 962–972 Santa Fe, New Mexico, USA, August 20-26, 2018. 962 Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer Chris van der Lee Tilburg University c.vdrlee@tilburguniversity.edu Bart Verduijn Tilburg University b.w.verduijn@tilburguniversity.edu Emiel Krahmer Tilburg University e.j.krahmer@tilburguniversity.edu Sander Wubben Tilburg University s.wubben@tilburguniversity.edu Abstract We present an evaluation of PASS, a data-to-text system that generates Dutch soccer reports from match statistics which are automatically tailored towards fans of one club or the other. The eval- uation in this paper consists of two studies. An intrinsic human-based evaluation of the system’s output is described in the first study. In this study it was found that compared to human-written texts, computer-generated texts were rated slightly lower on style-related text components (flu- ency and clarity) and slightly higher in terms of the correctness of given information. Further- more, results from the first study showed that tailoring was accurately recognized in most cases, and that participants struggled with correctly identifying whether a text was written by a human or computer. The second study investigated if tailoring affects perceived text quality, for which no results were garnered. This lack of results might be due to negative preconceptions about computer-generated texts which were found in the first study. 1 Introduction Evaluation of end-to-end Natural Language Generation (NLG) systems is important to assess whether the system has properly expressed certain properties (e.g. quality, speed), or whether the designed properties work as intended (Gkatzia and Mahamood, 2015). Traditional NLG evaluation approaches can typically be assigned to one of two categories: intrinsic or extrinsic (Belz and Reiter, 2006). Intrinsic approaches seek to evaluate properties of the system itself. This can be done using automatic measures such as BLEU, NIST, ROUGE, etc. or by asking human participants to rate the systems output with e.g. Likert or rating scales. Extrinsic approaches aim to assess the impact of the system, by measuring if the system can fulfill its purpose or what the user gains from the systems output. While scholars have been positive about the effort the NLG community has put into their evaluations (Gatt and Belz, 2010, for instance), it is often the case that an extensive evaluation does not take much priority after a system is built. The usage of automatic measures is gaining traction due to its quickness and low costs, but they are still considered controversial by many (Reiter and Belz, 2009; Novikova et al., 2017, for instance). Furthermore, the intrinsic approaches with human ratings are often limited in scope, using a relatively small sample of participants, using relatively short questionnaires that only shine light on a small aspect of text quality, and/or comparing the computer-generated texts against non- representative human texts. Similarly, the amount of extrinsic evaluations that have been performed up until now is low. While they are considered the most useful type of evaluation by some (Reiter and Belz, 2009), they are not carried out as often as other …

Content truncated to 500 words