Cluster-based Prediction of User Ratings for Stylistic Surface Realisation

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 702–711, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Cluster-based Prediction of User Ratings for Stylistic Surface Realisation Nina Dethlefs, Heriberto Cuay´ahuitl, Helen Hastie, Verena Rieser and Oliver Lemon Heriot-Watt University, Mathematical and Computer Sciences, Edinburgh n.s.dethlefs@hw.ac.uk Abstract Surface realisations typically depend on their target style and audience. A challenge in estimating a stylistic realiser from data is that humans vary significantly in their sub- jective perceptions of linguistic forms and styles, leading to almost no correlation be- tween ratings of the same utterance. We ad- dress this problem in two steps. First, we estimate a mapping function between the linguistic features of a corpus of utterances and their human style ratings. Users are partitioned into clusters based on the sim- ilarity of their ratings, so that ratings for new utterances can be estimated, even for new, unknown users. In a second step, the estimated model is used to re-rank the out- puts of a number of surface realisers to pro- duce stylistically adaptive output. Results confirm that the generated styles are recog- nisable to human judges and that predictive models based on clusters of users lead to better rating predictions than models based on an average population of users. 1 Introduction Stylistic surface realisation aims not only to find the best realisation candidate for a semantic input based on some underlying trained model, but also aims to adapt its output to properties of the user, such as their age, social group, or location, among others. One of the first systems to address stylis- tic variation in generation was Hovy (1988)’s PAULINE, which generated texts that reflect dif- ferent speaker attitudes towards events based on multiple, adjustable features. Stylistic variation in such contexts can often be modelled systemat- ically as a multidimensional variation space with several continuous dimensions, so that varying stylistic scores indicate the strength of each di- mension in a realisation candidate. Here, we fo- cus on the dimensions of colloquialism, politeness and naturalness. Assuming a target score on one or more dimensions, candidate outputs of a data- driven realiser can then be ranked according to their predicted affinity with the target scores. In this paper, we aim for an approach to stylis- tic surface realisation which is on the one hand based on natural human data so as to reflect stylis- tic variation that is as natural as possible. On the other hand, we aim to minimise the amount of annotation and human engineering that informs the design of the system. To this end, we esti- mate a mapping function between automatically identifiable shallow linguistic features character- istic of an utterance and its human-assigned style ratings. In addition, we aim to address the high degree of variability that is often encountered in subjective rating studies, such as assessments of recommender systems (O’Mahony et al., 2006; Amatriain et al., 2009), sentiment analysis (Pang and Lee, 2005), or surface realisations, where …

Content truncated to 500 words