Sparse Sequence-to-Sequence Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1504–1519 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1504 Sparse Sequence-to-Sequence Models Ben Peters† Vlad Niculae† and Andr´e F. T. Martins†‡ †Instituto de Telecomunicac¸˜oes, Lisbon, Portugal ‡Unbabel, Lisbon, Portugal benzurdopeters@gmail.com, vlad@vene.ro, andre.martins@unbabel.com Abstract Sequence-to-sequence models are a powerful workhorse of NLP. Most variants employ a softmax transformation in both their attention mechanism and output layer, leading to dense alignments and strictly positive output proba- bilities. This density is wasteful, making mod- els less interpretable and assigning probability mass to many implausible outputs. In this pa- per, we propose sparse sequence-to-sequence models, rooted in a new family of α-entmax transformations, which includes softmax and sparsemax as particular cases, and is sparse for any α > 1. We provide fast algorithms to evaluate these transformations and their gra- dients, which scale well for large vocabulary sizes. Our models are able to produce sparse alignments and to assign nonzero probability to a short list of plausible outputs, sometimes rendering beam search exact. Experiments on morphological inflection and machine transla- tion reveal consistent gains over dense models. 1 Introduction Attention-based sequence-to-sequence (seq2seq) models have proven useful for a variety of NLP applications, including machine translation (Bah- danau et al., 2015; Vaswani et al., 2017), speech recognition (Chorowski et al., 2015), abstractive summarization (Chopra et al., 2016), and morpho- logical inflection generation (Kann and Sch¨utze, 2016), among others. In part, their strength comes from their flexibility: many tasks can be formu- lated as transducing a source sequence into a target sequence of possibly different length. However, conventional seq2seq models are dense: they compute both attention weights and output probabilities with the softmax function (Bridle, 1990), which always returns positive val- ues. This results in dense attention alignments, in which each source position is attended to at each d r a w e d </s> n </s> </s> 66.4% 32.2% 1.4% Figure 1: The full beam search of our best perform- ing morphological inflection model when generating the past participle of the verb “draw”. The model gives nonzero probability to exactly three hypotheses, includ- ing the correct form (“drawn”) and the form that would be correct if “draw” were regular (“drawed”). target position, and in dense output probabilities, in which each vocabulary type always has nonzero probability of being generated. This contrasts with traditional statistical machine translation systems, which are based on sparse, hard alignments, and decode by navigating through a sparse lattice of phrase hypotheses. Can we transfer such notions of sparsity to modern neural architectures? And if so, do they improve performance? In this paper, we provide an affirmative an- swer to both questions by proposing neural sparse seq2seq models that replace the softmax trans- formations (both in the attention and output) by sparse transformations. Our innovations are rooted in the recently proposed sparsemax transfor- mation (Martins and Astudillo, 2016) and Fenchel- Young losses (Blondel et al., 2019). Concretely, we consider a family of transformations …

Content truncated to 500 words