Accelerated Reinforcement Learning for Sentence Generation by Vocabulary Prediction

Proceedings of NAACL-HLT 2019, pages 3115–3125 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3115 Accelerated Reinforcement Learning for Sentence Generation by Vocabulary Prediction Kazuma Hashimoto∗ Salesforce Research k.hashimoto@salesforce.com Yoshimasa Tsuruoka The University of Tokyo tsuruoka@logos.t.u-tokyo.ac.jp Abstract A major obstacle in reinforcement learning- based sentence generation is the large action space whose size is equal to the vocabulary size of the target-side language. To improve the efficiency of reinforcement learning, we present a novel approach for reducing the ac- tion space based on dynamic vocabulary pre- diction. Our method first predicts a fixed- size small vocabulary for each input to gen- erate its target sentence. The input-specific vocabularies are then used at supervised and reinforcement learning steps, and also at test time. In our experiments on six machine trans- lation and two image captioning datasets, our method achieves faster reinforcement learning (∼2.7x faster) with less GPU memory (∼2.3x less) than the full-vocabulary counterpart. We also show that our method more effectively re- ceives rewards with fewer iterations of super- vised pre-training. 1 Introduction Sentence generation with neural networks plays a key role in many language processing tasks, including machine translation (Sutskever et al., 2014), image captioning (Lin et al., 2014), and abstractive summarization (Rush et al., 2015). The most common approach for learning the sen- tence generation models is maximizing the like- lihood of the model on the gold-standard target sentences. Recently, approaches based on rein- forcement learning have attracted increasing at- tention to reduce the gap between training and test situations and to directly incorporate task- specific and more flexible evaluation metrics such as BLEU scores (Papineni et al., 2002) into opti- mization (Ranzato et al., 2016). While reinforcement learning-based sentence generation is appealing, it is often too computa- ∗ Work was done while the first author was working at the University of Tokyo. tionally demanding to be used with large training data. In reinforcement learning for sentence gen- eration, selecting an action corresponds to select- ing a word in the vocabulary V . The number of possible actions at each time step is thus equal to the vocabulary size, which often exceeds tens of thousands. Among such a large set of possible ac- tions, at most N actions are selected if the length of the generated sentence is N, where we can as- sume N ≪ |V |. In other words, most of the pos- sible actions are not selected, and the large action space slows down reinforcement learning and con- sumes a large amount of GPU memory. In this paper, we propose to accelerate rein- forcement learning by reducing the large action space. The reduction of action space is achieved by predicting a small vocabulary for each source input. Our method first constructs the small input- specific vocabulary by selecting K (≤ 1000) rele- vant words, and then the small vocabulary is used at both training and test time. Our experiments on six machine translation and two image captioning datasets show that …

Content truncated to 500 words