Length bias in Encoder Decoder Models and a Case for Global Conditioning

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516–1525, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Length bias in Encoder Decoder Models and a Case for Global Conditioning Pavel Sountsov Google siege@google.com Sunita Sarawagi ∗ IIT Bombay sunita@iitb.ac.in Abstract Encoder-decoder networks are popular for modeling sequences probabilistically in many applications. These models use the power of the Long Short-Term Memory (LSTM) archi- tecture to capture the full dependence among variables, unlike earlier models like CRFs that typically assumed conditional independence among non-adjacent variables. However in practice encoder-decoder models exhibit a bias towards short sequences that surprisingly gets worse with increasing beam size. In this paper we show that such phenomenon is due to a discrepancy between the full sequence margin and the per-element margin enforced by the locally conditioned training objective of a encoder-decoder model. The discrepancy more adversely impacts long sequences, explaining the bias towards predicting short sequences. For the case where the predicted sequences come from a closed set, we show that a glob- ally conditioned model alleviates the above problems of encoder-decoder models. From a practical point of view, our proposed model also eliminates the need for a beam-search dur- ing inference, which reduces to an efficient dot-product based search in a vector-space. 1 Introduction In this paper we investigate the use of neural net- works for modeling the conditional distribution Pr(y|x) over sequences y of discrete tokens in re- sponse to a complex input x, which can be another ∗ Work done while visiting Google Research on a leave from IIT Bombay. sequence or an image. Such models have applica- tions in machine translation (Bahdanau et al., 2014; Sutskever et al., 2014), image captioning (Vinyals et al., 2015), response generation in emails (Kannan et al., 2016), and conversations (Khaitan, 2016; Vinyals and Le, 2015; Li et al., 2015). The most popular neural network for probabilis- tic modeling of sequences in the above applications is the encoder-decoder (ED) network (Sutskever et al., 2014). A ED network first encodes an input x into a vector which is then used to initialize a re- current neural network (RNN) for decoding the out- put y. The decoder RNN factorizes Pr(y|x) using the chain rule as � j Pr(yj|y1, . . . , yj−1, x) where y1, . . . , yn denote the tokens in y. This factoriza- tion does not entail any conditional independence assumption among the {yj} variables. This is un- like earlier sequence models like CRFs (Lafferty et al., 2001) and MeMMs (McCallum et al., 2000) that typically assume that a token is independent of all other tokens given its adjacent tokens. Modern-day RNNs like LSTMs promise to capture non-adjacent and long-term dependencies by summarizing the set of previous tokens in a continuous, high-dimensional state vector. Within the limits of parameter capacity allocated to the model, the ED, by virtue of exactly factorizing the token sequence, is consistent. However, when we created and deployed an ED model for a …

Content truncated to 500 words