Improving the Annotation of Sentence Specificity

Improving the Annotation of Sentence Specificity Junyi Jessy Li1, Bridget O’Daniel2, Yi Wu1, Wenli Zhao1, Ani Nenkova1 1 University of Pennsylvania, Philadelphia, PA 19104 {ljunyi,wuyd,wenliz,nenkova}@seas.upenn.edu 2 Berea College, Berea, KY 40404 Bridget Odaniel@berea.edu Abstract We introduce improved guidelines for annotation of sentence specificity, addressing the issues encountered in prior work. Our annotation provides judgements of sentences in context. Rather than binary judgements, we introduce a specificity scale which accommodates nuanced judgements. Our augmented annotation procedure also allows us to define where in the discourse context the lack of specificity can be resolved. In addition, the cause of the underspecification is annotated in the form of free text questions. We present results from a pilot annotation with this new scheme and demonstrate good inter-annotator agreement. We found that the lack of specificity distributes evenly among immediate prior context, long distance prior context and no prior context. We find that missing details that are not resolved in the the prior context are more likely to trigger questions about the reason behind events, “why” and “how”. Our data is accessible at http://www.cis.upenn.edu/%7Enlp/corpora/lrec16spec.html Keywords: specificity rating, underspecification, discourse 1. Introduction Louis and Nenkova (2012) introduced a corpus of sentences annotated as general or specific. Their definition of sen- tence specificity relied mostly on examples and intuition, related to the amount of detail contained by the sentence. They used the corpus of general and specific sentences to evaluate a classifier for the binary task (Louis and Nenkova, 2011a) and showed that changes in sentence and overall text specificity are strongly associated with perceptions of text quality. Science writing of the best quality in the New York Times is overall more general than regular science pieces in NYT and contain fewer stretches of specific con- tent (Louis and Nenkova, 2013). Automatic summaries, which are often judged to be incoherent, are significantly more specific than same length human-written summaries for the same events (Louis and Nenkova, 2011b). Sentence specificity is also more robust than sentence length as in- dicator of which sentences may pose comprehension prob- lems and need to be simplified for given audiences (Li and Nenkova, 2015). It is also a stable predictor in identifying high-quality arguments in online discussions (Swanson et al., 2015). Given the demonstrated importance of sentence and text specificity in practical applications and the known short- comings of the existing annotation, we set out to develop a more detailed framework for annotation of sentence speci- ficity. In the brief annotation guidelines of Louis and Nenkova (2012), the general vs. specific distinction was de- fined in the following way: “General sentences are broad statements about a topic. Specific sentences contain details and can be used to support or explain the general sen- tences further. In other words, general sentences create expectations in the minds of a reader who would definitely need evidence or examples from the author. Specific sentences can stand by them- selves.” Their analysis of annotator disagreement led to the conclu- sion that a scale of specificity would be more …

Content truncated to 500 words