Self-Regulated Interactive Sequence-to-Sequence Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 303–315 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 303 Self-Regulated Interactive Sequence-to-Sequence Learning Julia Kreutzer Computational Linguistics Heidelberg University Germany kreutzer@cl.uni-heidelberg.de Stefan Riezler Computational Linguistics & IWR Heidelberg University Germany riezler@cl.uni-heidelberg.de Abstract Not all types of supervision signals are cre- ated equal: Different types of feedback have different costs and effects on learning. We show how self-regulation strategies that de- cide when to ask for which kind of feed- back from a teacher (or from oneself) can be cast as a learning-to-learn problem leading to improved cost-aware sequence-to-sequence learning. In experiments on interactive neu- ral machine translation, we find that the self- regulator discovers an ϵ-greedy strategy for the optimal cost-quality trade-off by mixing different feedback types including corrections, error markups, and self-supervision. Further- more, we demonstrate its robustness under do- main shift and identify it as a promising alter- native to active learning. 1 Introduction The concept of self-regulation has been studied in educational research (Hattie and Timperley, 2007; Hattie and Donoghue, 2016), psychology (Zim- merman and Schunk, 1989; Panadero, 2017), and psychiatry (Nigg, 2017), and was identified as cen- tral to successful learning. “Self-regulated stu- dents” can be characterized as “becoming like teachers”, in that they have a repertoire of strate- gies to self-assess and self-manage their learning process, and they know when to seek help and which kind of help to seek. While there is a vast literature on machine learning approaches to meta- learning (Schmidhuber et al., 1996), learning-to- learn (Thrun and Pratt, 1998), or never-ending learning (Mitchell et al., 2015), the aspect of learn- ing when to ask for which kind of feedback has so far been neglected in this field. We propose a machine learning algorithm that uses self-regulation in order to balance the cost and effect of learning from different types of feed- back. This is particularly relevant for human-in- Seq2Seq Figure 1: Human-in-the-loop self-regulated learning. the-loop machine learning, where human super- vision is costly. The self-regulation module au- tomatically learns which kind of feedback to ap- ply when in training—full supervision by teacher demonstration or correction, weak supervision in the form of positive or negative rewards for student predictions, or a self-supervision signal generated by the student. Figure 1 illustrates this learning scenario. The learner, in our case a sequence-to- sequence (Seq2Seq) learner, aims to solve a cer- tain task with the help of a human teacher. For every input it receives for training, it can ask the teacher for feedback to its own output, or super- vise itself by training on its own output, or skip learning on the input example altogether. The self-regulator’s policy for choosing feedback types is guided by their cost and by the performance gain achieved by learning from a particular type of feedback. We apply the self-regulation algorithm to inter- active machine translation where a neural machine translation (NMT) system functions as a student which receives feedback …

Content truncated to 500 words