Self-Supervised Learning for Pairwise Data Refinement

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 435–446 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 435 Self-Supervised Learning for Pairwise Data Refinement Gustavo Hern´andez ´Abrego, Bowen Liang, Wei Wang, Zarana Parekh, Yinfei Yang, Yunhsuan Sung Google Research, Mountain View, CA. USA {gustavoha,bowenl,wangwe,zarana,yinfeiy,yhsung}@google.com Abstract Pairwise data automatically constructed from weakly supervised signals has been widely used for training deep learning models. Pair- wise datasets such as parallel texts can have uneven quality levels overall, but usually con- tain data subsets that are more useful as learn- ing examples. We present two methods to refine data that are aimed at obtaining that kind of subsets in a self-supervised way. Our methods are based on iteratively training dual- encoder models to compute similarity scores. We evaluate our methods on de-noising paral- lel texts and training neural machine transla- tion models. We find that: (i) The self-super- vised refinement achieves most machine trans- lation gains in the first iteration, but following iterations further improve its intrinsic evalua- tion. (ii) Machine translations can improve the de-noising performance when combined with selection steps. (iii) Our methods are able to reach the performance of a supervised method. Being entirely self-supervised, our methods are well-suited to handle pairwise data without the need of prior knowledge or human annota- tions. 1 Introduction Deep learning models are widely adopted and have demonstrated their usefulness in many areas and applications. Despite their diversity, one common characteristic of these models is the large number of parameters that need to be adjusted during train- ing (some recent models that have billions of pa- rameters include T5 (Raffel et al., 2019) and GPT- 2 (Radford et al., 2019)). This leads to the need of collecting large amounts of training examples. Pairwise data, that captures the relationship in two modalities, is used to train deep learning models such as Neural Machine Translation (NMT) (Wu et al., 2016), Question Answering (Wang et al., 2007), Image Captioning (Sharma et al., 2018), etc. To train this kind of models, large-scale data can often be obtained from weak signals like text co- occurrence (Yang et al., 2018) or dictionary n-gram matching (Uszkoreit et al., 2010). For example, in the machine translation community, the large amount of multilingual text available on the inter- net has naturally led to the idea of using internet data to train NMT models (Resnik, 1999). This approach has proven advantageous but it has the drawback that data mined this way is intrinsically noisy (Resnik and Smith, 2003). Despite the poor quality, usually this kind of data contains a helpful subset that can be recovered through a process of data cleaning or refinement. Data cleaning could be implemented with linguistic knowledge such as its script, vocabulary, syntax, etc. Alternatively, a model can be trained on “clean” or “trusted” pairs that are verified through manual annotation. Both options can be highly effective, but the former …

Content truncated to 500 words