Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2690–2701 June 6–11, 2021. ©2021 Association for Computational Linguistics 2690 Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation Xutan Peng Chenghua Lin∗ Mark Stevenson Department of Computer Science, The University of Sheffield, UK {x.peng, c.lin, mark.stevenson}@shef.ac.uk Abstract Cross-Lingual Word Embeddings (CLWEs) encode words from two or more languages in a shared high-dimensional space in which vectors representing words with similar mean- ing (regardless of language) are closely lo- cated. Existing methods for building high- quality CLWEs learn mappings that minimise the ℓ2 norm loss function. However, this op- timisation objective has been demonstrated to be sensitive to outliers. Based on the more ro- bust Manhattan norm (aka. ℓ1 norm) goodness- of-fit criterion, this paper proposes a simple post-processing step to improve CLWEs. An advantage of this approach is that it is fully ag- nostic to the training process of the original CLWEs and can therefore be applied widely. Extensive experiments are performed involv- ing ten diverse languages and embeddings trained on different corpora. Evaluation results based on bilingual lexicon induction and cross- lingual transfer for natural language inference tasks show that the ℓ1 refinement substantially outperforms four state-of-the-art baselines in both supervised and unsupervised settings. It is therefore recommended that this strategy be adopted as a standard for CLWE methods. 1 Introduction Cross-Lingual Word Embedding (CLWE) tech- niques have recently received significant atten- tion as an effective means to support Natural Lan- guage Processing applications for low-resource lan- guages, e.g., machine translation (Artetxe et al., 2018b) and transfer learning (Peng et al., 2021). The most successful CLWE models are the so- called projection-based methods, which learn map- pings between monolingual word vectors with very little, or even zero, cross-lingual supervision (Lam- ple et al., 2018; Artetxe et al., 2018a; Glavaš et al., 2019). Mainstream projection-based CLWE models typically identify orthogonal mappings by ∗Chenghua Lin is the corresponding author. minimising the topological dissimilarity between source and target embeddings based on ℓ2 loss (aka. Frobenius loss or squared error) (Glavaš et al., 2019; Ruder et al., 2019). This learning strategy has two advantages. First, adding the orthogonal- ity constraint to the mapping function has been demonstrated to significantly enhance the quality of CLWEs (Xing et al., 2015). Second, the existence of a closed-form solution to the ℓ2 optima (Schöne- mann, 1966) greatly simplifies the computation required (Artetxe et al., 2016; Smith et al., 2017). Despite its popularity, work in various appli- cation domains has noted that ℓ2 loss is not ro- bust to noise and outliers. It is widely known in computer vision that ℓ2-loss-based solutions can severely exaggerate noise, leading to inaccurate estimates (Aanæs et al., 2002; De La Torre and Black, 2003). In data mining, Principal Compo- nent Analysis (PCA) using ℓ2 loss has been shown to be sensitive to the presence of outliers in the input data, degrading the quality of the feature space produced (Kwak, 2008). Previous studies …

Content truncated to 500 words