Predicting the Fluency of Text with Shallow Structural Features: Case Studies of Machine Translation and Human-Written Text

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 139–147, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text Jieun Chae University of Pennsylvania chaeji@seas.upenn.edu Ani Nenkova University of Pennsylvania nenkova@seas.upenn.edu Abstract Sentence fluency is an important compo- nent of overall text readability but few studies in natural language processing have sought to understand the factors that define it. We report the results of an ini- tial study into the predictive power of sur- face syntactic statistics for the task; we use fluency assessments done for the purpose of evaluating machine translation. We find that these features are weakly but sig- nificantly correlated with fluency. Ma- chine and human translations can be dis- tinguished with accuracy over 80%. The performance of pairwise comparison of fluency is also very high—over 90% for a multi-layer perceptron classifier. We also test the hypothesis that the learned models capture general fluency properties applica- ble to human-written text. The results do not support this hypothesis: prediction ac- curacy on the new data is only 57%. This finding suggests that developing a dedi- cated, task-independent corpus of fluency judgments will be beneficial for further in- vestigations of the problem. 1 Introduction Numerous natural language applications involve the task of producing fluent text. This is a core problem for surface realization in natural language generation (Langkilde and Knight, 1998; Banga- lore and Rambow, 2000), as well as an impor- tant step in machine translation. Considerations of sentence fluency are also key in sentence sim- plification (Siddharthan, 2003), sentence compres- sion (Jing, 2000; Knight and Marcu, 2002; Clarke and Lapata, 2006; McDonald, 2006; Turner and Charniak, 2005; Galley and McKeown, 2007), text re-generation for summarization (Daum´e III and Marcu, 2004; Barzilay and McKeown, 2005; Wan et al., 2005) and headline generation (Banko et al., 2000; Zajic et al., 2007; Soricut and Marcu, 2007). Despite its importance for these popular appli- cations, the factors contributing to sentence level fluency have not been researched indepth. Much more attention has been devoted to discourse-level constraints on adjacent sentences indicative of co- herence and good text flow (Lapata, 2003; Barzi- lay and Lapata, 2008; Karamanis et al., to appear). In many applications fluency is assessed in combination with other qualities. For example, in machine translation evaluation, approaches such as BLEU (Papineni et al., 2002) use n-gram over- lap comparisons with a model to judge overall “goodness”, with higher n-grams meant to capture fluency considerations. More sophisticated ways to compare a system production and a model in- volve the use of syntax, but even in these cases flu- ency is only indirectly assessed and the main ad- vantage of the use of syntax is better estimation of the semantic overlap between a model and an out- put. Similarly, the metrics proposed for text gener- ation by (Bangalore et al., 2000) (simple accuracy, generation accuracy) are based on string-edit …

Content truncated to 500 words