Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2560–2571, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2560 Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation Dana Ruiter Saarland University DFKI GmbH Josef van Genabith Saarland University DFKI GmbH druiter@lsv.uni-saarland.de {josef.van genabith,cristinae}@dfki.de Cristina Espa˜na-Bonet DFKI GmbH Abstract Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each other in a virtuous circle. In this study, we provide an in-depth analysis of the sampling choices the SSNMT model makes during training. We show how, without it having been told to do so, the model self-selects samples of increasing (i) complexity and (ii) task-relevance in com- bination with (iii) performing a denoising cur- riculum. We observe that the dynamics of the mutual-supervision signals of both system in- ternal representation types are vital for the ex- traction and translation performance. We show that in terms of the Gunning-Fog Readability index, SSNMT starts extracting and learning from Wikipedia data suitable for high school students and quickly moves towards content suitable for first year undergraduate students. 1 Introduction Human learners, when faced with a new task, gen- erally focus on simple examples before applying what they learned to more complex instances. This approach to learning based on sampling from a cur- riculum of increasing complexity has also been shown to be beneficial for machines and is re- ferred to as curriculum learning (CL) (Bengio et al., 2009). Previous research on curriculum learning has focused on selecting the best distribution of data, i.e. order, difficulty and closeness to the final task, to train a system. In such a setting, data is externally prepared for the system to ease the learn- ing task. In our work, we follow a complementary approach: we design a system that selects by itself the data to train on, and we analyse the selected dis- tribution of data, order, difficulty and closeness to the final task, without imposing it beforehand. Our method resembles self-paced learning (SPL) (Ku- mar et al., 2010), in that it uses the emerging model hypothesis to select samples online that fit into its space as opposed to most curriculum learning approaches that rely on judgements by the target hypothesis, i.e. an external teacher (Hacohen and Weinshall, 2019) to design the curriculum. We focus on machine translation (MT), in partic- ular, self-supervised machine translation (SSNMT) (Ruiter et al., 2019), which exploits the internal rep- resentations of an emergent neural machine transla- tion (NMT) system to select useful data for training, where each selection decision is dependent on the current state of the model. Self-supervised learning (Raina et al., 2007; Bengio et al., 2013) involves a primary task, for which labelled data is not avail- able, and an auxiliary task that enables the primary task to be learned by exploiting supervisory sig- nals within the data. In SSNMT, both tasks, data extraction and …

Content truncated to 500 words