Attending to Future Tokens for Bidirectional Sequence Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1–10, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1 Attending to Future Tokens for Bidirectional Sequence Generation Carolin Lawrence and Bhushan Kotnis and Mathias Niepert NEC Laboratories Europe {carolin.lawrence, bhushan.kotnis, mathias.niepert}@neclab.eu Abstract Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into con- sideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversa- tional tasks where the proposed bidirectional model outperforms competitive baselines by a large margin. 1 Introduction When generating an output sequence, neural net- work models typically produce one token at a time. At each generation step, only the already produced sequence is taken into account. How- ever, future and not-yet-produced tokens can also be highly relevant when choosing the current to- ken. The importance of attending to both past and future tokens is apparent in self-attention archi- tectures such as the Transformer (Vaswani et al., 2017). The self-attention module of a Transformer network treats a sequence bidrectionally as a fully connected graph of tokens – when a token is pro- duced all other tokens are taken into considera- tion. However, this requires the entire sequence to be known a priori and when a Transformer is used for sequence generation, the self-attention process only includes previously produced tokens (Vaswani et al. (2017); Radford et al. (2019); in- ter alia). But the bidirectional self-attention is a crucial property of the highly successful language model BERT (Devlin et al., 2018). During the pre- training procedure of BERT, a fraction of input to- kens is randomly masked out and the training ob- jective is to predict these masked tokens correctly. BERT can then be fine-tuned for various classifi- cation tasks. Unfortunately, BERT cannot be di- rectly used for sequence generation because the bidirectional nature of the approach requires the entire sequence to be known beforehand. Inspired by BERT’s masking-based objective, we propose to start out with a sequence of place- holder tokens which are iteratively replaced by to- kens from the output vocabulary to eventually gen- erate the full output sequence. For an example see Figure 1. With this novel model component, the self-attention of a Transformer can take both past and future tokens into consideration, leading to Bidirectional Sequence generation (BISON). Fur- thermore, it allows us to directly incorporate the pre-trained language model BERT and, to the best of our knowledge, for the first time directly fine- tune it for sequence generation. BISON makes …

Content truncated to 500 words