Incorporating Stylistic Lexical Preferences in Generative Language Models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1074–1079 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1074 Incorporating Stylistic Lexical Preferences in Generative Language Models Hrituraj Singh Adobe Research hrisingh@adobe.com Gaurav Verma Adobe Research gaverma@adobe.com Balaji Vasan Srinivasan Adobe Research balsrini@adobe.com Abstract While recent advances in language modeling has resulted in powerful generation models, their generation style remains implicitly depen- dent on the training data and can not emulate a specific target style. Leveraging the generative capabilities of a transformer-based language models, we present an approach to induce certain target-author attributes by incorporat- ing continuous multi-dimensional lexical pref- erences of an author into generative language models. We introduce rewarding strategies in a reinforcement learning framework that encour- ages the use of words across multiple categori- cal dimensions, to varying extents. Our experi- ments demonstrate that the proposed approach can generate text that distinctively aligns with a given target author’s lexical style. We con- duct quantitative and qualitative comparisons with competitive and relevant baselines to il- lustrate the benefits of the proposed approach. 1 Introduction With recent advances in unconstrained language generation (Radford et al., a,b; Brown et al., 2020), an emerging direction is to adapt such pre-trained language models to follow certain stylistic con- straints (Wang et al., 2019; Syed et al., 2020). These approaches rely on the inherent properties of the training corpus to tailor generation to tar- get characteristics; for example, implicitly learn- ing author-stylized text generation by training on author-specific corpus (Syed et al., 2020) and learn- ing to generate formal text (Wang et al., 2019). However, it is desirable to have explicit control over certain stylistic aspects in such generation, for e.g., emulating lexical choices of an author in a generation, capturing syntactic constructs, induc- ing sentential preferences (active vs. passive) in generation. To this end, we propose an approach to adapt a pre-trained Transformer-based language model (Vaswani et al., 2017), specifically GPT-2 (Radford et al., b), to generate text that aligns with given lexical elements of style by providing explicit rewards in a reinforcement learning framework. Reinforcement learning (RL) has been success- fully applied to several natural language generation tasks like summarization (Paulus et al., 2018) and paraphrase generation (Li et al., 2018). RL over- comes the ‘exposure bias’ (Ranzato et al., 2015) in cross-entropy based training of language mod- els and allows for optimization with respect to non-differentiable objectives. However, existing explorations around the use of RL in generation tasks have been limited to RNN-based models due to issues surrounding stabilization of RL training on Transformer models (Parisotto et al., 2019). Parisotto et al. further conclude that a Transformer requires reordering of the normalization layer from output to the input streams along with a gated mech- anism instead of residual connections to stabilize its training. Building on this, we leverage the shifted position of the normalization layers in GPT-2 to train an RL framework with GPT-2 for aligning generated text to target lexical characteristics. Recent work by Ziegler et …

Content truncated to 500 words