Hierarchical Sketch Induction for Paraphrase Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 201 - 218 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation Nishant Kambhatla Logan Born Anoop Sarkar School of Computing Science, Simon Fraser University 8888 University Drive, Burnaby BC, Canada {nkambhat, loborn, anoop}@sfu.ca Abstract We propose a novel data-augmentation tech- nique for neural machine translation based on ROT-k ciphertexts. ROT-k is a simple letter substitution cipher that replaces a letter in the plaintext with the kth letter after it in the al- phabet. We first generate multiple ROT-k ci- phertexts using different values of k for the plaintext which is the source side of the parallel data. We then leverage this enciphered train- ing data along with the original parallel data via multi-source training to improve neural ma- chine translation. Our method, CipherDAug, uses a co-regularization-inspired training pro- cedure, requires no external data sources other than the original training data, and uses a stan- dard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin. This technique combines easily with existing approaches to data augmen- tation, and yields particularly strong results in low-resource settings.1 1 Introduction One naturally wonders if the problem of transla- tion could conceivably be treated as a problem in cryptography. [...] frequencies of letters, letter combinations, [...] etc., [...] are to some signif- icant degree independent of the language used (Weaver, 1949) Indeed, to a system which treats inputs as atomic identifiers, the alphabet behind these identifiers is irrelevant. Distributional properties are of sole importance, and changes in the underlying encod- ing should be transparent provided these properties are preserved. In light of this, a bijective cipher such as ROT-k (Figure 1) is in effect invisible to modern NLP techniques: distributional features are invariant under such a cipher, guaranteeing that the meaning of an enciphered text is the same as the un-enciphered text, given the key. This work ex- ploits this fact to develop a novel approach to data 1Our code is available at https://github.com/ protonish/cipherdaug-nmt PLAIN abcdefghijklmnopqrstuvwxyz ROT- 1 bcdefghijklmnopqrstuvwxyza ROT- 2 cdefghijklmnopqrstuvwxyzab ROT- 3 defghijklmnopqrstuvwxyzabc SRC : es ist diese pyramide. ROT-1(SRC): ft jtu ejftf qzsbnjef. ROT-2(SRC): gu kuv fkgug rßtcokfg. TGT : it's that pyramid. SRC ROT-2(SRC) ROT-1(SRC) lexically diverse (non-overlapping) syntactically similar semantically equivalent lexical overlap possible semantically similar lexical overlap unlikely semantically similar Figure 1: ROT-k encipherment. The plaintext SRC is enciphered to generate the ciphertexts ROT-1(SRC) and ROT-2(SRC), which share distributional features with the plaintext but use a new encoding. augmentation which is completely orthogonal to previous approaches. Data augmentation is a simple regularization- inspired technique to improve generalization in neural machine translation (NMT) models. These models (Bahdanau et al., 2015; Vaswani et al., 2017) learn powerful representational spaces (Ra- ganato and Tiedemann, 2018; Voita et al., 2019; Kudugunta et al., 2019) which scale to large num- bers of languages and massive datasets (Aharoni et al., 2019). However, in the …

Content truncated to 500 words