SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation

Proceedings of NAACL-HLT 2018, pages 1528–1538 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation Ye Zhang∗ UT Austin yezhang@utexas.edu Nan Ding Radu Soricut Google AI {dingnan,rsoricut}@google.com Abstract Supervised training of abstractive language generation models results in learning condi- tional probabilities over language sequences based on the supervised training signal. When the training signal contains a variety of writ- ing styles, such models may end up learn- ing an ’average’ style that is directly influ- enced by the training data make-up and can- not be controlled by the needs of an applica- tion. We describe a family of model archi- tectures capable of capturing both generic lan- guage characteristics via shared model param- eters, as well as particular style characteristics via private model parameters. Such models are able to generate language according to a specific learned style, while still taking advan- tage of their power to model generic language phenomena. Furthermore, we describe an ex- tension that uses a mixture of output distribu- tions from all learned styles to perform on-the- fly style adaptation based on the textual input alone. Experimentally, we find that the pro- posed models consistently outperform models that encapsulate single-style or average-style language generation capabilities. 1 Introduction Encoder-decoder models have recently pushed forward the state-of-the-art performance on a va- riety of language generation tasks, including ma- chine translation (Bahdanau et al., 2015; Wu et al., 2016; Vaswani et al., 2017), text summariza- tion (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017), dialog systems (Li et al., 2016; As- ghar et al., 2017), and image captioning (Xu et al., 2015; Ranzato et al., 2015; Liu et al., 2017). This framework consists of an encoder that reads the input data and encodes it as a sequence of vectors, which is in turn used by a decoder to generate an- ∗Work done as an intern at Google AI. other sequence of vectors used to produce output symbols step by step. The prevalent approach to training such a model is to update all the model parameters using all the examples in the training data (over multiple epochs). This is a reasonable approach, under the assumption that we are modeling a single underly- ing distribution in the data. However, in many ap- plications and for many natural language datasets, there exist multiple underlying distributions, char- acterizing a variety of language styles. For in- stance, the widely-used Gigaword dataset (Graff and Cieri, 2003) consists of a collection of arti- cles written by various publishers (The New York Times, Agence France Presse, Xinhua News, etc.), each with its own style characteristics. Training a model’s parameters on all the training examples results in an averaging effect across style charac- teristics, which may lower the quality of the out- puts; additionally, this averaging effect may be completely undesirable for applications that re- quire a level of control over the output style. At the opposite end of the spectrum, one can …

Content truncated to 500 words