Learning Neural Sequence-to-Sequence Models from Weak Feedback with Bipolar Ramp Loss

Learning Neural Sequence-to-Sequence Models from Weak Feedback with Bipolar Ramp Loss Laura Jehl∗ Carolin Lawrence∗ Computational Linguistics Heidelberg University 69120 Heidelberg, Germany {jehl, lawrence}@cl.uni-heidelberg.de Stefan Riezler Computational Linguistics & IWR Heidelberg University 69120 Heidelberg, Germany riezler@cl.uni-heidelberg.de Abstract In many machine learning scenarios, supervi- sion by gold labels is not available and conse- quently neural models cannot be trained directly by maximum likelihood estimation. In a weak supervision scenario, metric-augmented objec- tives can be employed to assign feedback to model outputs, which can be used to extract a supervision signal for training. We present sev- eral objectives for two separate weakly super- vised tasks, machine translation and semantic parsing. We show that objectives should ac- tively discourage negative outputs in addition to promoting a surrogate gold structure. This notion of bipolarity is naturally present in ramp loss objectives, which we adapt to neu- ral models. We show that bipolar ramp loss objectives outperform other non-bipolar ramp loss objectives and minimum risk training on both weakly supervised tasks, as well as on a supervised machine translation task. Addition- ally, we introduce a novel token-level ramp loss objective, which is able to outperform even the best sequence-level ramp loss on both weakly supervised tasks. 1 Introduction Sequence-to-sequence neural models are stan- dardly trained using a maximum likelihood esti- mation (MLE) objective. However, MLE training requires full supervision by gold target structures, which in many scenarios are too difficult or expensive to obtain. For example, in semantic parsing for question-answering it is often easier to collect gold answers rather than gold parses ∗Both authors contributed equally to this publication. (Clarke et al., 2010; Berant et al., 2013; Pasupat and Liang, 2015; Rajpurkar et al., 2016, inter alia). In machine translation, there are many domains for which no gold references exist, although cross- lingual document-level links are present for many multilingual data collections. In this paper we investigate methods where a supervision signal for output structures can be extracted from weak feedback. In the following, we use learning from weak feedback, or weakly supervised learning, to refer to a scenario where output structures generated by the model are judged according to an external metric, and this feedback is used to extract a supervision signal that guides the learning process. Metric-augmented sequence-level objectives from reinforcement learning (Williams, 1992; Ranzato et al., 2016), minimum risk training (MRT) (Smith and Eisner, 2006; Shen et al., 2016) or margin-based struc- tured prediction objectives (Taskar et al., 2005; Edunov et al., 2018) can be seen as instances of such algorithms. In natural language processing applications, such algorithms have mostly been used in com- bination with full supervision tasks, allowing to compute a feedback score from metrics such as BLEU or F-score that measure the similarity of output structures against gold structures. Our main interest is in weak supervision tasks where the calculation of a feedback score cannot fall back onto gold structures. For example, matching proposed answers to a gold answer can guide a semantic parser towards correct parses, and …

Content truncated to 500 words