RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1822 - 1835 July 10-15, 2022 ©2022 Association for Computational Linguistics RSTGen: Imbuing Fine-Grained Interpretable Control into Long-Form Text Generators Rilwan A. Adewoyin1,4, Ritabrata Dutta2, Yulan He1,3 1Department of Computer Science, University of Warwick, UK 2Department of Statistics, University of Warwick, UK 3The Alan Turing Institute, UK 4Department of Computer Science and Engineering, Southern University of Science and Technology, China {rilwan.adewoyin,ritabrata.dutta,yulan.he}@warwick.ac.uk Abstract In this paper, we study the task of improving the cohesion and coherence of long-form text generated by language models. To this end, we propose RSTGen, a framework that utilises Rhetorical Structure Theory (RST), a classical language theory, to control the discourse struc- ture, semantics and topics of generated text. Firstly, we demonstrate our model’s ability to control structural discourse and semantic fea- tures of generated text in open generation eval- uation. Then we experiment on the two chal- lenging long-form text tasks of argument gen- eration and story generation. Evaluation using automated metrics and a metric with high cor- relation to human evaluation, shows that our model performs competitively against existing models, while offering significantly more con- trols over generated text than alternative meth- ods. 1 Introduction Controllable text generation has attracted much at- tention in recent years. Generating coherent and cohesive long-form text with controllable attributes is particularly challenging due to the relatively com- plex syntactic and semantic structures involved compared to short-text generation. Long-form text generation can find applications in automatic argu- mentation, motivational speech, and opinionated writing, to name a few. A popular approach to controllable text genera- tion is to design prompts, which control high-level linguistic features (i.e., text style and sentiment) by prefixing simple context to the input of a language model. These non-granular methods are often too coarse to allow different parts of the text to have diverse or contrasting features. Furthermore, they tend to focus on a single linguistic feature of text, meaning that extra frameworks such as Plugand- Play (Dathathri et al., 2019) are required to control multiple linguistic features. To achieve improved content control, researchers have coupled a content planning module with a text generation module (Hua and Wang, 2020; Hua et al., 2021), in which the content planning mod- ule firstly generates a set of keyphrases and their corresponding positions in sentences and the gener- ation module fills in the gaps. To control syntactic structures of text, recent works have proposed a neuro-symbolic approach leveraging Dependency Structure Theory (DST) (Shen et al., 2019; Du et al., 2020). Enforcing specific structures over the generated text has been proven to be useful in increasing coherence and cohesion of short-form text. However, a relatively large amount of DST information is required to encode comparatively short texts. Thus, they are unsuitable for use in long-form text generation tasks. Similar to DST, Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) is a classi- cal tree-based interpretation of natural language. Whereas DST is …

Content truncated to 500 words