Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1752–1765 May 2-6, 2023 ©2023 Association for Computational Linguistics Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing Wenjie Zhong1,3, Jason Naradowsky1, Hiroya Takamura3 Ichiro Kobayashi2,3, Yusuke Miyao1,3 1The University of Tokyo, 2Ochanomizu University, 3National Institute of Advanced Industrial Science and Technology {zvengin, narad, yusuke}@is.s.u-tokyo.ac.jp takamura.hiroya@aist.go.jp, koba@is.ocha.ac.jp Abstract We explore the idea of incorporating con- cepts from writing skills curricula into human- machine collaborative writing scenarios, focus- ing on adding writing modes as a control for text generation models. Using crowd-sourced workers, we annotate a corpus of narrative text paragraphs with writing mode labels. Classi- fiers trained on this data achieve an average accuracy of ∼ 87% on held-out data. We fine- tune a set of large language models to condition on writing mode labels, and show that the gen- erated text is recognized as belonging to the specified mode with high accuracy. To study the ability of writing modes to pro- vide fine-grained control over generated text, we devise a novel turn-based text reconstruc- tion game to evaluate the difference between the generated text and the author’s intention. We show that authors prefer text suggestions made by writing mode-controlled models on average 61.1% of the time, with satisfaction scores 0.5 higher on a 5-point ordinal scale. When evaluated by humans, stories generated via collaboration with writing mode-controlled models achieve high similarity with the profes- sionally written target story. We conclude by identifying the most common mistakes found in the generated stories. The datasets and codes are available at the Github1. 1 Introduction Large-scale pre-trained language models (PLMs) have demonstrated a remarkable aptitude for gen- erating text with an exceptional degree of fluency and structure (Guan et al., 2021; Tan et al., 2021), sparking renewed efforts to utilize them for the pur- pose of generating narrative fiction. Recent work has explored various ways of controlling PLMs, us- ing sentiment (Luo et al., 2019), style (Kong et al., 2021a), and even character information (Liu et al., 2020a), in an attempt to cater the generated text to an author’s intentions. 1https://github.com/ZVengin/fride Summary: A boy was walking along a path in the forest, when he came across a heap of leaves. Mode Story Dialogue “Here are some leaves,” he whispered. “They were wet when we came, and are wet now. I'll lie them down and wait.” “What is it?” I exclaimed. “If you will stand still,” said my boy, “I will show you … “ Action He stood for a moment looking at me, then quietly he picked up the leaves, and carrying them in his hand, climbed to the top of the heap, and examined them … Description This heap consisted of dead leaves, many of them rotten, and still wet, with one or two lying flat on the ground, others lying up against the branches. The first to fall was the one I had thought dead. It had been crushed by the wind. … Figure …

Content truncated to 500 words