Invisible to People but not to Machines: Evaluation of Style-aware HeadlineGeneration in Absence of Reliable Human Judgment

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6709–6717 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6709 Invisible to People but not to Machines: Evaluation of Style-aware Headline Generation in Absence of Reliable Human Judgment Lorenzo De Mattei123, Michele Cafagna23, Felice Dell’Orletta1 and Malvina Nissim3 ItaliaNLP Lab, ILC-CNR, Pisa, Italy1, Department of Computer Science, University of Pisa, Italy2, University of Groningen, The Netherlands 3 {l.de.mattei,m.cafagna,m.nissim}@rug.nl felice.dellorletta@ilc.cnr.it Abstract We automatically generate headlines that are expected to comply with the specific styles of two different Italian newspapers. Through a data alignment strategy and different training/testing settings, we aim at decoupling content from style and preserve the latter in generation. In order to evaluate the generated headlines’ quality in terms of their specific newspaper-compliance, we devise a fine-grained evaluation strategy based on automatic classification. We observe that our models do indeed learn newspaper-specific style. Importantly, we also observe that humans aren’t reliable judges for this task, since although familiar with the newspapers, they are not able to discern their specific styles even in the original human-written headlines. The utility of automatic evaluation goes therefore beyond saving the costs and hurdles of manual annotation, and deserves particular care in its design. Keywords: Natural Language Generation, Stylistic variations, Evaluation 1. Introduction Automatic headline generation is conceptually a simple task which can be conceived as a form of extreme sum- marisation (Rush et al., 2015): given an article, or a portion of it, generate its headline. Generation of headlines though is not just a matter of sum- marising the content. Different newspapers report the news in different ways, depending on their policies and strate- gies. For example, they might exhibit some topic-biases, such as writing more about gossip vs more about politics. But even when reporting on the same topics, they might exhibit specific stylistic features related to word choices, word order, punctuation usage, etc. This might be even more evident when newspapers are positioned at opposite ends of the political spectrum (Cafagna et al., 2019b). Such newspaper-specific style is likely to be exhibited not only in the articles’ body, but also in the headlines, which are a prime tool to capture attention and make clear statements about the newspaper’s position over a certain event. Can this newspaper-specific style be distinguished? And is it preserved in automatically generated headlines? To answer such questions, we train newspaper-specific head- line generation models, and evaluate how style-compliant the generated headline is for a given newspaper. How such evaluation can be performed though is yet another research question of its own. Evaluating generated text just using standard metrics based on lexical overlap is normally not accurate enough (Liu et al., 2016). In machine translation, for example, the deci- sive, final system evaluation is typically human-based, as the lexically-based BLEU score is not exhaustive. Auto- matic evaluation strategies are still used because human evaluation is expensive, not always available, and complex to include in highly iterative developments. However, hu- man …

Content truncated to 500 words