Extending the Entity-based Coherence Model with Multiple Ranks

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 315–324, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Extending the Entity-based Coherence Model with Multiple Ranks Vanessa Wei Feng Department of Computer Science University of Toronto Toronto, ON, M5S 3G4, Canada weifeng@cs.toronto.edu Graeme Hirst Department of Computer Science University of Toronto Toronto, ON, M5S 3G4, Canada gh@cs.toronto.edu Abstract We extend the original entity-based coher- ence model (Barzilay and Lapata, 2008) by learning from more fine-grained coher- ence preferences in training data. We asso- ciate multiple ranks with the set of permuta- tions originating from the same source doc- ument, as opposed to the original pairwise rankings. We also study the effect of the permutations used in training, and the effect of the coreference component used in en- tity extraction. With no additional manual annotations required, our extended model is able to outperform the original model on two tasks: sentence ordering and summary coherence rating. 1 Introduction Coherence is important in a well-written docu- ment; it helps make the text semantically mean- ingful and interpretable. Automatic evaluation of coherence is an essential component of vari- ous natural language applications. Therefore, the study of coherence models has recently become an active research area. A particularly popular coherence model is the entity-based local coher- ence model of Barzilay and Lapata (B&L) (2005; 2008). This model represents local coherence by transitions, from one sentence to the next, in the grammatical role of references to entities. It learns a pairwise ranking preference between al- ternative renderings of a document based on the probability distribution of those transitions. In particular, B&L associated a lower rank with au- tomatically created permutations of a source doc- ument, and learned a model to discriminate an original text from its permutations (see Section 3.1 below). However, coherence is matter of de- gree rather than a binary distinction, so a model based only on such pairwise rankings is insuffi- ciently fine-grained and cannot capture the sub- tle differences in coherence between the permuted documents. Since the first appearance of B&L’s model, several extensions have been proposed (see Sec- tion 2.3 below), primarily focusing on modify- ing or enriching the original feature set by incor- porating other document information. By con- trast, we wish to refine the learning procedure in a way such that the resulting model will be able to evaluate coherence on a more fine-grained level. Specifically, we propose a concise exten- sion to the standard entity-based coherence model by learning not only from the original docu- ment and its corresponding permutations but also from ranking preferences among the permutations themselves. We show that this can be done by assigning a suitable objective score for each permutation indi- cating its dissimilarity from the original one. We call this a multiple-rank model since we train our model on a multiple-rank basis, rather than tak- ing the original pairwise ranking approach. This extension can also be easily combined with other extensions …

Content truncated to 500 words