CaM-Gen: Causally Aware Metric-Guided Text Generation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2047 - 2060 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CaM-Gen: Causally Aware Metric-Guided Text Generation Navita Goyal University of Maryland∗ navita@umd.edu Roodram Paneri Microsoft∗ rpaneri@microsoft.com Ayush Agarwal Tournafest∗ aagarwal9782 @gmail.com Udit Kalani Adobe Systems∗ kalani@adobe.com Abhilasha Sancheti University of Maryland sancheti@umd.edu Niyati Chhaya Adobe Research nchhaya@adobe.com Abstract Content is created for a well-defined purpose, often described by a metric or signal repre- sented in the form of structured information. The relationship between the goal (metrics) of target content and the content itself is non- trivial. While large-scale language models show promising text generation capabilities, guiding the generated text with external met- rics is challenging. These metrics and con- tent tend to have inherent relationships and not all of them may be of consequence. We introduce CaM-Gen: Causally aware Gener- ative Networks guided by user-defined target metrics incorporating the causal relationships between the metric and content features. We leverage causal inference techniques to identify causally significant aspects of a text that lead to the target metric and then explicitly guide generative models towards these by a feedback mechanism. We propose this mechanism for variational autoencoder and Transformer-based generative models. The proposed models beat baselines in terms of the target metric control while maintaining fluency and language qual- ity of the generated text. To the best of our knowledge, this is one of the early attempts at controlled generation incorporating a metric guide using causal inference. 1 Introduction Most content is created for a well-defined goal. For example, a blog writer often publishes articles to gain popularity and trigger conversations, and a columnist may write an opinionated piece to gather feedback. In marketing applications, these goals are business objectives that need to be optimized using the content shared with the customers. The validation of whether the goal was met or not is done by tracking metrics that capture the reader behavior. In social media, metrics include number of comments, likes, or shares whereas for a pub- lishing house they are the number of views and ∗Work done while at Adobe Research readers. These engagement metrics (hereafter, met- rics) are proxy for target goals. Based on historical content, textual content characteristics that success- fully achieve the desired metrics can be assessed (Tan et al., 2019; Verma et al., 2020). Guiding text generation models by these signals is important for meeting the required goals. While recent neural language models have shown tremendous success towards fluent text gen- eration (Radford et al., 2018; Devlin et al., 2019), achieving controlled, goal-specific generation is challenging. There has been work on text gener- ation controlling for style, topic, or size (Keskar et al., 2019). These methods are able to leverage content characteristics that are common between the definition of goal (i.e., control) and the text. However, for metrics that are not explicit in the text, controlled generation is non-trivial to codify. The challenge is introduced due to the fact that for external metrics, there is a …

Content truncated to 500 words