Few-Shot NLG with Pre-Trained Language Model

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 183–190 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 183 Few-Shot NLG with Pre-Trained Language Model Zhiyu Chen1, Harini Eavani2, Wenhu Chen1, Yinyin Liu2, and William Yang Wang1 1University of California, Santa Barbara 2Intel AI {zhiyuchen, wenhuchen, william}@cs.ucsb.edu, {harini.eavani, yinyin.liu}@intel.com Abstract Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications dif- ficult with limited data. In this work, we pro- pose the new task of few-shot natural language generation. Motivated by how humans tend to summarize tabular data, we propose a simple yet effective approach and show that it not only demonstrates strong performance but also pro- vides good generalization across domains. The design of the model architecture is based on two aspects: content selection from input data and language modeling to compose coherent sentences, which can be acquired from prior knowledge. With just 200 training examples, across multiple domains, we show that our ap- proach achieves very reasonable performances and outperforms the strongest baseline by an average of over 8.0 BLEU points improvement. Our code and data can be found at https: //github.com/czyssrs/Few-Shot-NLG 1 Introduction Natural language generation (NLG) from struc- tured data or knowledge (Gatt and Krahmer, 2018) is an important research problem for vari- ous NLP applications. Some examples are task- oriented dialog, question answering (He et al., 2017; Ghazvininejad et al., 2018; Su et al., 2016; Saha et al., 2018; Yin et al., 2016) and interdis- ciplinary applications such as medicine (Hasan and Farri, 2019; Cawsey et al., 1997) and health- care (Hasan and Farri, 2019; DiMarco et al., 2007). There is great potential to use automatic NLG sys- tems in a wide range of real-life applications. Re- cently, deep neural network based NLG systems have been developed, such as those seen in the E2E challenge (Novikova et al., 2017), WEATHER- GOV (Liang et al., 2009), as well as more complex ones such as WIKIBIO (Liu et al., 2018) and RO- TOWIRE (Wiseman et al., 2017). Compared to traditional slot-filling pipeline approaches, such neural-based systems greatly reduce feature engi- neering efforts and improve text diversity as well as fluency. Although they achieve good performance on benchmarks such as E2E challenge (Novikova et al., 2017) and WIKIBIO (Lebret et al., 2016), their performance depends on large training datasets, e.g., 500k table-text training pairs for WIKIBIO (Lebret et al., 2016) in a single domain. Such data-hungry nature makes neural-based NLG systems difficult to be widely adopted in real-world applications as they have significant manual data curation overhead. This leads us to formulate an interesting research question: 1. Can we significantly reduce human annotation effort to achieve reasonable performance using neural NLG models? 2. Can we make the best of generative pre-training, as prior knowledge, to gen- erate text from structured data? Motivated by this, we propose the new task of few- shot natural language generation: given only a handful of labeled …

Content truncated to 500 words