GLEU: Automatic Evaluation of Sentence-Level Fluency

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344–351, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics GLEU: Automatic Evaluation of Sentence-Level Fluency Andrew Mutton∗ Mark Dras∗ Stephen Wan∗,† Robert Dale∗ ∗Centre for Language Technology †Information and Communication Technologies Macquarie University CSIRO NSW 2109 Australia NSW 2109 Australia madras@ics.mq.edu.au Abstract In evaluating the output of language tech- nology applications—MT, natural language generation, summarisation—automatic eval- uation techniques generally conflate mea- surement of faithfulness to source content with fluency of the resulting text. In this paper we develop an automatic evaluation metric to estimate fluency alone, by examin- ing the use of parser outputs as metrics, and show that they correlate with human judge- ments of generated text fluency. We then de- velop a machine learner based on these, and show that this performs better than the indi- vidual parser metrics, approaching a lower bound on human performance. We finally look at different language models for gener- ating sentences, and show that while individ- ual parser metrics can be ‘fooled’ depending on generation method, the machine learner provides a consistent estimator of fluency. 1 Introduction Intrinsic evaluation of the output of many language technologies can be characterised as having at least two aspects: how well the generated text reflects the source data, whether it be text in another lan- guage for machine translation (MT), a natural lan- guage generation (NLG) input representation, a doc- ument to be summarised, and so on; and how well it conforms to normal human language usage. These two aspects are often made explicit in approaches to creating the text. For example, in statistical MT the translation model and the language model are treated separately, characterised as faithfulness and fluency respectively (as in the treatment in Jurafsky and Martin (2000)). Similarly, the ultrasummarisa- tion model of Witbrock and Mittal (1999) consists of a content model, modelling the probability that a word in the source text will be in the summary, and a language model. Evaluation methods can be said to fall into two cate- gories: a comparison to gold reference, or an appeal to human judgements. Automatic evaluation meth- ods carrying out a comparison to gold reference tend to conflate the two aspects of faithfulness and flu- ency in giving a goodness score for generated out- put. BLEU (Papineni et al., 2002) is a canonical ex- ample: in matching n-grams in a candidate transla- tion text with those in a reference text, the metric measures faithfulness by counting the matches, and fluency by implicitly using the reference n-grams as a language model. Often we are interested in know- ing the quality of the two aspects separately; many human judgement frameworks ask specifically for separate judgements on elements of the task that cor- respond to faithfulness and to fluency. In addition, the need for reference texts for an evaluation metric can be problematic, and intuitively seems unneces- sary for characterising an aspect of text quality that is not related to its content …

Content truncated to 500 words