Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 291–301 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation Nishant Kambhatla Logan Born Anoop Sarkar School of Computing Science, Simon Fraser University 8888 University Drive, Burnaby BC, Canada {nkambhat, loborn, anoop}@sfu.ca Abstract Multilingual neural translation models exploit cross-lingual transfer to perform zero-shot translation between unseen language pairs. Past efforts to improve cross-lingual transfer have focused on aligning contextual sentence-level representations. This paper introduces three novel contributions to allow exploiting near- est neighbours at the token level during train- ing, including: (i) an efficient, gradient-friendly way to share representations between neighbor- ing tokens; (ii) an attentional semantic layer which extracts latent features from shared em- beddings; and (iii) an agreement loss to har- monize predictions across different sentence representations. Experiments on two multilin- gual datasets demonstrate consistent gains in zero shot translation over strong baselines. 1 Introduction Many-to-many multilingual neural translation mod- els (Firat et al., 2016; Johnson et al., 2017; Khan- delwal et al., 2020; Fan et al., 2022) share a single representation space across multiple language pairs, which enables them to perform zero-shot transla- tions between unseen pairs (Ha et al., 2017; Chen et al., 2022; Wu et al., 2022). Prior work on zero- shot translation has focused on aligning contextual, sentence-level representations from multiple lan- guages (Ji et al., 2020; Pan et al., 2021a), to make these more ‘universal’ or language-agnostic (Gu et al., 2018; Gu and Feng, 2022). Non-contextual, token-level representations offer another space in which this kind of alignment could be pursued, but this space has not been thoroughly explored in prior work. Even lexicon-based methods (Conneau et al., 2020; Reid and Artetxe, 2022), which exploit token- level anchors from multilingual dictionaries (Duan et al., 2020), still use these to align representations at the sentence level. In this work, we explore a novel technique for sharing information across languages at the token he is great at playing football amazing jugando fußball awesome treten soccer great amazing awesome avg k-NN word embedding Figure 1: NN-informed embeddings average representa- tions from nearby subwords in the embedding space. level, which exploits nearest neighbours (NNs) to aggregate information from subwords across multi- ple languages. When analysing embedding spaces, many authors speak in terms of “neighborhoods” or “subspaces” which group together tokens from a particular semantic field or other natural class. These neighborhoods form implicitly as a model learns similarities between embedded words or sub- words. We propose to make this neighborhood structure explicit by forcing a model to consider a token’s neighbors when learning its embedding. Specifically, we dynamically perturb a translation model’s token embeddings at training time by av- eraging them with their nearest neighbors; thus a token like soccer may end up mixed together with related tokens such as football, fußball, or futbol from potentially distinct languages (Figure 1). This encourages the model to organize its subword em- beddings in such a way …

Content truncated to 500 words