MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10324 - 10334 December 7-11, 2022 ©2022 Association for Computational Linguistics MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective Zhe Hu Baidu Inc huzhe01@baidu.com Hou Pong Chan University of Macau hpchan@um.edu.mo Lifu Huang Virginia Tech lifuh@vt.edu Abstract Teaching neural models to generate narrative coherent texts is a critical problem. Recent pre- trained language models have achieved promis- ing results, but there is still a gap between hu- man written texts and machine-generated out- puts. In this work, we propose a novel multi- task training strategy for coherent text genera- tion grounded on the cognitive theory of writ- ing, which empowers the model to learn es- sential subskills needed for writing including planning and reviewing besides end-to-end gen- eration. We extensively evaluate our model on three open-ended generation tasks including story generation, news article writing and argu- ment generation. Experiments show that our model achieves better results on both few-shot and fully-supervised settings than strong base- lines, and human evaluations confirm that our model can generate more coherent outputs. 1 Introduction With the recent development of pretraining tech- niques, large neural language models have achieved impressive results on various text generation tasks and can generate fluent outputs. However, when generating long-form texts (i.e., paragraphs with multiple sentences), there is still a large gap be- tween machine-generated outputs and human writ- ten texts: the generated outputs usually suffer from incoherence issues and fail to maintain overall nar- rative coherence (See et al., 2019). One possible reason for the above defects is the lack of effective text planning as global guid- ance to control the generation process. Compared with the traditional generation systems which of- ten decompose the generation task into text plan- ning and surface realization (Reiter and Dale, 1997; Carenini and Moore, 2006), current autoregressive neural language models are typically trained to produce texts in a left-to-right token-level man- ner, which lacks anchored goal to constrain the generation process (Fan et al., 2019). Recent stud- ies incorporate text planning into neural models by leveraging structured representations (Goldfarb- Tarrant et al., 2020; Hua and Wang, 2020) or latent variables (Wang et al., 2022; Hu et al., 2022) as high-level plans, but they need manually designed domain-specific plans or complicated supervision signals to train the model. Another reason is the ineffective usage of the negative samples as contrasts to teach the model better distinguish between correct and incorrect targets. Negative samples are useful to enhance the model ability to generate better outputs (He and Glass, 2020). Recent work explores techin- ques such as contrastive learning (Lee et al., 2020; Su et al., 2022; An et al., 2022) and unlikelihood training (Welleck et al., 2020; Li et al., 2020) to leverage negative samples for model training. We draw our motivations from the cognitive process theory of writing (Flower and Hayes, 1981): “Writing is best understood as a set of distinct thinking processes which writers orches- trate or …

Content truncated to 500 words