Bandit Structured Prediction for Neural Sequence-to-Sequence Learning

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1503–1513 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1138 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1503–1513 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1138 Bandit Structured Prediction for Neural Sequence-to-Sequence Learning Julia Kreutzer∗ and Artem Sokolov∗ and Stefan Riezler†,∗ ∗Computational Linguistics & †IWR, Heidelberg University, Germany {kreutzer,sokolov,riezler}@cl.uni-heidelberg.de Abstract Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feed- back. This feedback is received in athe form of a task loss evaluation to predicted output structure, without having access to gold standard struc- tures. We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural net- works. Furthermore, we show how to in- corporate control variates into our learning algorithms for variance reduction and im- proved generalization. We present an eval- uation on a neural machine translation task that shows improvements of up to 5.89 BLEU points for domain adaptation from simulated bandit feedback. 1 Introduction Many NLP tasks involve learning to predict a structured output such as a sequence, a tree or a graph. Sequence-to-sequence learning with neu- ral networks has recently become a popular ap- proach that allows tackling structured prediction as a mapping problem between variable-length se- quences, e.g., from foreign language sentences into target-language sentences (Sutskever et al., 2014), or from natural language input sentences into linearized versions of syntactic (Vinyals et al., 2015) or semantic parses (Jia and Liang, 2016). A known bottleneck in structured prediction is the requirement of large amounts of gold-standard structures for supervised learning of model pa- rameters, especially for data-hungry neural net- work models. Sokolov et al. (2016a,b) presented a framework for stochastic structured prediction under bandit feedback that alleviates the need for labeled output structures in learning: Following an online learning protocol, on each iteration the learner receives an input, predicts an output struc- ture, and receives partial feedback in form of a task loss evaluation of the predicted structure.1 They “banditize” several objective functions for linear structured predictions, and evaluate the resulting algorithms with simulated bandit feedback on var- ious NLP tasks. We show how to lift linear structured predic- tion under bandit feedback to non-linear models for sequence-to-sequence learning with attention- based recurrent neural networks (Bahdanau et al., 2015). Our framework is applicable to sequence- to-sequence learning from various types of weak feedback. For example, extracting learning signals from the execution of structured outputs against databases has been established in the communi- ties of semantic parsing and grounded language learning since more than a decade (Zettlemoyer and Collins, 2005; Clarke et al., 2010; Liang et al., 2011). Our work can build the basis for neural se- mantic parsing from weak feedback. In this paper, we focus on the application of ma- chine translation via neural sequence-to-sequence learning. The standard procedure of training neu- ral …

Content truncated to 500 words