What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain

What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain Annie Louis University of Pennsylvania Philadelphia, PA 19104 lannie@seas.upenn.edu Ani Nenkova University of Pennsylvania Philadelphia, PA 19104 nenkova@seas.upenn.edu Abstract Great writing is rare and highly admired. Readers seek out articles that are beautifully written, informative and entertaining. Yet information-access technologies lack capabil- ities for predicting article quality at this level. In this paper we present first experiments on article quality prediction in the science jour- nalism domain. We introduce a corpus of great pieces of science journalism, along with typical articles from the genre. We imple- ment features to capture aspects of great writ- ing, including surprising, visual and emotional content, as well as general features related to discourse organization and sentence structure. We show that the distinction between great and typical articles can be detected fairly ac- curately, and that the entire spectrum of our features contribute to the distinction. 1 Introduction Measures of article quality would be hugely bene- ficial for information retrieval and recommendation systems. In this paper, we describe a dataset of New York Times science journalism articles which we have categorized for quality differences and present a system that can automatically make the distinction. Science journalism conveys complex scientific ideas, entertaining and educating at the same time. Consider the following opening of a 2005 article by David Quammen from Harper’s magazine: One morning early last winter a small item appeared in my local newspaper announcing the birth of an extraordi- nary animal. A team of researchers at Texas A&M Uni- versity had succeeded in cloning a whitetail deer. Never done before. The fawn, known as Dewey, was developing normally and seemed to be healthy. He had no mother, just a surrogate who had carried his fetus to term. He had no father, just a “donor” of all his chromosomes. He was the genetic duplicate of a certain trophy buck out of south Texas whose skin cells had been cultured in a laboratory. One of those cells furnished a nucleus that, transplanted and rejiggered, became the DNA core of an egg cell, which became an embryo, which in time be- came Dewey. So he was wildlife, in a sense, and in an- other sense elaborately synthetic. This is the sort of news, quirky but epochal, that can cause a person with a mouth- ful of toast to pause and marvel. What a dumb idea, I marveled. The writing is clear and well-organized but the text also contains creative use of language and a clever story-like explanation of the scientific con- tribution. Such properties make science journalism an attractive genre for studying writing quality. Sci- ence journalism is also a highly relevant domain for information retrieval in the context of educational as well as entertaining applications. Article quality measures can hugely benefit such systems. Prior work indicates that three aspects of article quality can be successfully predicted: a) whether a text meets the acceptable standards for spelling (Brill and Moore, 2000), grammar …

Content truncated to 500 words