Sparse Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4252–4273, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4252 Sparse Text Generation Pedro Henrique Martins� Zita Marinho�� and Andr´e F. T. Martins��� �Instituto de Telecomunicac¸˜oes �Priberam Labs �Institute of Systems and Robotics �LUMLIS (Lisbon ELLIS Unit), Instituto Superior T´ecnico �Unbabel Lisbon, Portugal pedrohenriqueamartins@tecnico.ulisboa.pt, zita.marinho@priberam.pt, andre.t.martins@tecnico.ulisboa.pt. Abstract Current state-of-the-art text generators build on powerful language models such as GPT- 2, achieving impressive performance. How- ever, to avoid degenerate text, they require sampling from a modified softmax, via tem- perature parameters or ad-hoc truncation tech- niques, as in top-k or nucleus sampling. This creates a mismatch between training and test- ing conditions. In this paper, we use the recently introduced entmax transformation to train and sample from a natively sparse lan- guage model, avoiding this mismatch. The result is a text generator with favorable per- formance in terms of fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. In order to evaluate our model, we propose three new metrics for comparing sparse or truncated distributions: ϵ-perplexity, sparsemax score, and Jensen-Shannon diver- gence. Human-evaluated experiments in story completion and dialogue generation show that entmax sampling leads to more engaging and coherent stories and conversations. 1 Introduction The last few years have seen a tremendous progress in neural language modeling, with a strong im- pact on key NLP tasks such as machine trans- lation (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), speech recognition (Chorowski et al., 2015), summarization (Rush et al., 2015), and open-ended text generation (Rad- ford et al., 2019). Benefiting from large amounts of data, models such as Transformer-XL (Dai et al., 2019) and GPT-2 (Radford et al., 2019) have achieved impressive perplexity scores on language modeling. However, the generated text is still often repetitive and incoherent (Table 1). A downside of current approaches is the mis- match between training and testing conditions: Context:“Are they ready to go live?” Zuko asked. “Yup, in just a minute” Tomi replied as he leaned back in his reclining chair. This was a moment that the entire galaxy would see. After 10 years... Greedy: ... of study, the scientists had finally come to the conclusion that the universe was not a place for living. The scientists had been studying the uni- verse for over a century. They had been studying the universe for over a century. Top-k: ... of studying this phenomenon, the team had finally come up with a plan to study what the aliens were doing. The team would be looking for a way to create a space station. “And what would you call it?” “What would you call the space station?” Nucleus: ... of diligent testing, and some test run after that, the results were well received. “they’re waiting for you, because there’s no way they would.” Stowedly at a local wedding place, the two archaeologists spoke together. Entmax: ... of hibernation, Tomi was back. These last days had been a …

Content truncated to 500 words