Data Augmentation with Adversarial Training for Cross-Lingual NLI

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5158–5167 August 1–6, 2021. ©2021 Association for Computational Linguistics 5158 Data Augmentation with Adversarial Training for Cross-Lingual NLI Xin Dong1, Yaxin Zhu1, Zuohui Fu1, Dongkuan Xu2, Gerard de Melo3 1 Rutgers University 2 The Pennsylvania State University 3 Hasso Plattner Institute / University of Potsdam {xd48, yz956, zuohui.fu}@rutgers.edu dux19@psu.edu, gdm@demelo.org Abstract Due to recent pretrained multilingual represen- tation models, it has become feasible to exploit labeled data from one language to train a cross- lingual model that can then be applied to mul- tiple new languages. In practice, however, we still face the problem of scarce labeled data, leading to subpar results. In this paper, we propose a novel data augmentation strategy for better cross-lingual natural language inference by enriching the data to reflect more diversity in a semantically faithful way. To this end, we propose two methods of training a generative model to induce synthesized examples, and then leverage the resulting data using an ad- versarial training regimen for more robustness. In a series of detailed experiments, we show that this fruitful combination leads to substan- tial gains in cross-lingual inference. 1 Introduction There is a growing need for NLP systems that support low-resource languages, for which task- specific training data may be lacking, while domain-specific parallel corpora may be too scarce to train a reliable machine translation engine. To overcome this, zero-shot cross-lingual systems can be trained on a source language LS and subse- quently also be applied to other languages LT de- spite a complete lack of labelled training data for those target languages. In the past, such systems typically drew on translation dictionaries, lexical knowledge graphs, or parallel corpora, to build a cross-lingual model that exploits simple connec- tions between words and phrases across different languages (de Melo and Siersdorfer, 2007; Fu et al., 2020). Recently, pretrained language model archi- tectures such as BERT (Devlin et al., 2019) have been shown capable of learning joint multilingual representations with self-supervised objectives un- der a shared vocabulary, simply by combining the input from multiple languages (Devlin et al., 2019; Artetxe and Schwenk, 2019; Conneau and Lample, 2019; Conneau et al., 2019). Such representations greatly facilitate cross-lingual applications. Still, the success of such cross-lingual transfer hinges on how close the involved languages are, with substan- tial drops observed for some more distant language pairs (Lauscher et al., 2020). For our study, we focus on natural language infer- ence (NLI), i.e., classifying whether a premise sen- tence entails, contradicts, or is neutral with regard to a hypothesis sentence (Williams et al., 2017). This is a useful building block for applications in- volving semantic understanding (Zhu et al., 2018; Reimers and Gurevych, 2019). However, the task is also very challenging, as it not only requires accounting for very subtle differences in meaning but also inferring presuppositions and implications that are not explicitly stated. Due to these intricate subtleties, …

Content truncated to 500 words