Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 313–323 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 313 Sentence-Level Fluency Evaluation: References Help, But Can Be Spared! Katharina Kann∗ Center for Data Science New York University New York, USA kann@nyu.edu Sascha Rothe Google Research Zurich, Switzerland rothe@google.com Katja Filippova Google Research Berlin, Germany katjaf@google.com Abstract Motivated by recent findings on the proba- bilistic modeling of acceptability judgments, we propose syntactic log-odds ratio (SLOR), a normalized language model score, as a met- ric for referenceless fluency evaluation of nat- ural language generation output at the sentence level. We further introduce WPSLOR, a novel WordPiece-based version, which harnesses a more compact language model. Even though word-overlap metrics like ROUGE are com- puted with the help of hand-written references, our referenceless methods obtain a signifi- cantly higher correlation with human fluency scores on a benchmark dataset of compressed sentences. Finally, we present ROUGE-LM, a reference-based metric which is a natural ex- tension of WPSLOR to the case of available references. We show that ROUGE-LM yields a significantly higher correlation with human judgments than all baseline metrics, including WPSLOR on its own. 1 Introduction Producing sentences which are perceived as natu- ral by a human addressee—a property which we will denote as fluency1 throughout this paper —is a crucial goal of all natural language generation (NLG) systems: it makes interactions more natu- ral, avoids misunderstandings and, overall, leads to higher user satisfaction and user trust (Martin- dale and Carpuat, 2018). Thus, fluency evaluation is important, e.g., during system development, or ∗*This research was carried out while the first author was interning at Google. 1Alternative names include naturalness, grammaticality or readability. Note that the exact definitions of all those terms vary slightly throughout the literature. If access to a synonym dictionary is likely to be of use, then this package may 3 be of service. Participants are invited to submit a set pair do domain name that is already 1.6 taken along with alternative. Even $15 was The HSUS. 1 Table 1: Example compressions from our dataset with their fluency scores; scores in [1, 3], higher is better. for filtering unacceptable generations at applica- tion time. However, fluency evaluation of NLG systems constitutes a hard challenge: systems are often not limited to reusing words from the input, but can generate in an abstractive way. Hence, it is not guaranteed that a correct output will match any of a finite number of given references. This results in difficulties for current reference-based evaluation, especially of fluency, causing word- overlap metrics like ROUGE (Lin and Och, 2004) to correlate only weakly with human judgments (Toutanova et al., 2016). As a result, fluency eval- uation of NLG is often done manually, which is costly and time-consuming. Evaluating sentences on their fluency, on the other hand, is a linguistic ability of humans which has been the subject of a decade-long debate in cognitive science. In particular, the question has been …

Content truncated to 500 words