The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–25, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task Roy Schwartz1,2, Maarten Sap1, Ioannis Konstas1, Li Zilles1, Yejin Choi1 and Noah A. Smith1 1Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA 2Allen Institute for Artificial Intelligence, Seattle, WA, USA {roysch,msap,ikonstas,lzilles,yejin,nasmith}@cs.washington.edu Abstract A writer’s style depends not just on per- sonal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different con- straints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent end- ing to a story. We show that a simple linear classifier informed by stylistic fea- tures is able to successfully distinguish among the three cases, without even look- ing at the story context. In addition, com- bining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.1 1 Introduction Writing style is expressed through a range of lin- guistic elements such as words, sentence structure, and rhetorical devices. It is influenced by per- sonal factors such as age and gender (Schler et al., 2006), by personality traits such as agreeableness and openness (Ireland and Mehl, 2014), as well as by mental states such as sentiment (Davidov et al., 2010), sarcasm (Tsur et al., 2010), and deception (Feng et al., 2012). In this paper, we study the ex- tent to which writing style is affected by the nature of the writing task the writer was asked to perform, 1This paper extends our LSDSem 2017 shared task sub- mission (Schwartz et al., 2017). Story Prefix Ending John liked a girl at his work. He tried to get her attention by acting silly. She told him to grow up. John con- fesses he was trying to make her like him more. She feels flattered and asks John on a date. The girl found this charming, and gave him a second chance. John was happy about being rejected. Table 1: Examples of stories from the story cloze task. The table shows a story prefix with three con- trastive endings: The original ending, a coherent ending and a incoherent one. since different tasks likely engage different cogni- tive processes (Campbell and Pennebaker, 2003; Banerjee et al., 2014).2 We show that similar writing tasks with dif- ferent constraints on the author can lead to mea- surable differences in her writing style. As a case study, we present experiments based on the recently introduced ROC story …

Content truncated to 500 words