Improved Data Augmentation for Translation Suggestion

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 188 - 202 December 7-8, 2022 ©2022 Association for Computational Linguistics Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation Danni Liu and Jan Niehues Karlsruhe Institute of Technology {danni.liu, jan.niehues}@kit.edu Abstract The cornerstone of multilingual neural transla- tion is shared representations across languages. Given the theoretically infinite representation power of neural networks, semantically identi- cal sentences are likely represented differently. While representing sentences in the continuous latent space ensures expressiveness, it intro- duces the risk of capturing of irrelevant features which hinders the learning of a common rep- resentation. In this work, we discretize the en- coder output latent space of multilingual mod- els by assigning encoder states to entries in a codebook, which in effect represents source sentences in a new artificial language. This discretization process not only offers a new way to interpret the otherwise black-box model representations, but, more importantly, gives potential for increasing robustness in unseen testing conditions. We validate our approach on large-scale experiments with realistic data volumes and domains. When tested in zero- shot conditions, our approach is competitive with two strong alternatives from the literature. We also use the learned artificial language to an- alyze model behavior, and discover that using a similar bridge language increases knowledge- sharing among the remaining languages. 1 Introduction A promising potential of multilingual (Dong et al., 2015; Firat et al., 2016; Ha et al., 2016; Johnson et al., 2017) neural machine translation (NMT) is knowledge-sharing between languages. To enable knowledge-sharing, a prerequisite is the ability to capture common features of languages, especially between related ones. Constructed languages such as Interlingua and Esperanto are excellent exam- ples of human-designed structures based on the commonalities of a wide range of related languages. For data-driven models, however, it is difficult to leverage such resources due to data scarcity: There is little parallel data to these constructed languages, and creating new translation heavily depends on source sentence learning a new language (English) ↓ ↓ ↓ ↓ discrete codes 3 609 57 1042 source sentence belajar bahasa baru (Indonesian) ↓ ↓ ↓ discrete codes 3 57 258 Table 1: We aim to learn a sequence of discrete codes to represent source sentences in multilingual NMT models. Our goal is to 1) improve inference-time robustness, 2) have more interpretable intermediate representations. expert curation. Instead of relying on manually- created data, we aim to learn an artificial language in a more unsupervised fashion in parallel with training the NMT model. Specifically, our goal is to learn a sequence of tokens to represent the source sentences, which then serves as context for the NMT decoder. Table 1 illustrates this idea. A potential advantage of representing inputs in discrete tokens is robustness, a property especially relevant when NMT systems must cope with un- expected testing conditions. By discretization, we restrict the continuous latent space to a finite size, providing the possibility for model intermediate representations to fall back to a position seen in training. For …

Content truncated to 500 words