Online Segment to Segment Neural Transduction

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1307–1316, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Online Segment to Segment Neural Transduction Lei Yu1, Jan Buys1 and Phil Blunsom1,2 1University of Oxford 2DeepMind {lei.yu, jan.buys, phil.blunsom}@cs.ox.ac.uk Abstract We introduce an online neural sequence to se- quence model that learns to alternate between encoding and decoding segments of the input as it is read. By independently tracking the en- coding and decoding representations our algo- rithm permits exact polynomial marginaliza- tion of the latent segmentation during train- ing, and during decoding beam search is em- ployed to find the best alignment path to- gether with the predicted output sequence. Our model tackles the bottleneck of vanilla encoder-decoders that have to read and mem- orize the entire input sequence in their fixed- length hidden states before producing any out- put. It is different from previous attentive models in that, instead of treating the at- tention weights as output of a deterministic function, our model assigns attention weights to a sequential latent variable which can be marginalized out and permits online gener- ation. Experiments on abstractive sentence summarization and morphological inflection show significant performance gains over the baseline encoder-decoders. 1 Introduction The problem of mapping from one sequence to an- other is an importance challenge of natural language processing. Common applications include machine translation and abstractive sentence summarisation. Traditionally this type of problem has been tackled by a combination of hand-crafted features, align- ment models, segmentation heuristics, and language models, all of which are tuned separately. The recently introduced encoder-decoder paradigm has proved very successful for machine translation, where an input sequence is encoded into a fixed-length vector and an output sequence is then decoded from said vector (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014). This architecture is appealing, as it makes it possible to tackle the problem of sequence- to-sequence mapping by training a large neural network in an end-to-end fashion. However it is difficult for a fixed-length vector to memorize all the necessary information of an input sequence, especially for long sequences. Often a very large encoding needs to be employed in order to capture the longest sequences, which invariably wastes capacity and computation for short sequences. While the attention mechanism of Bahdanau et al. (2015) goes some way to address this issue, it still requires the full input to be seen before any output can be produced. In this paper we propose an architecture to tackle the limitations of the vanilla encoder-decoder model, a segment to segment neural transduction model (SSNT) that learns to generate and align simul- taneously. Our model is inspired by the HMM word alignment model proposed for statistical ma- chine translation (Vogel et al., 1996; Tillmann et al., 1997); we impose a monotone restriction on the alignments but incorporate recurrent dependencies on the input which enable rich locally non-monotone alignments to be captured. This is similar to the se- quence transduction model of …

Content truncated to 500 words