Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment

Proceedings of the 29th International Conference on Computational Linguistics, pages 1089–1097 October 12–17, 2022. 1089 Multi-Stage Framework with Refinement based Point Set Registration for Unsupervised Bi-Lingual Word Alignment Silviu Vlad Oprea∗ Amazon Alexa AI Cambridge, UK silviuvo@amazon.co.uk Sourav Dutta Huawei Research Centre Dublin, Ireland sourav.dutta2@huawei.com Haytham Assem† Amazon Alexa AI Cambridge, UK hithsala@amazon.co.uk Abstract Cross-lingual alignment of word embeddings are important in knowledge transfer across languages, for improving machine translation and other multi-lingual applications. Current unsupervised approaches relying on learning structure-preserving transformations, using ad- versarial networks and refinement strategies, suffer from instability and convergence issues. This paper proposes BioSpere, a novel multi- stage framework for unsupervised mapping of bi-lingual word embeddings onto a shared vec- tor space, by combining adversarial initializa- tion, refinement procedure and point set regis- tration. Experiments for parallel dictionary in- duction and word similarity demonstrate state- of-the-art unsupervised results for BioSpere on diverse languages – showcasing robustness against variable adversarial performance. 1 Introduction and Background Distributed word representations like Word2Vec (Mikolov et al., 2013), GloVe (Pen- nington et al., 2014) and FastText (Bojanowski et al., 2017) capture rich semantic meaning, and is used for a range of NLP tasks. Cross-lingual word embeddings (CLWE) entails mapping vocabularies of different languages onto a single vector space for capturing semantic similarity across languages (Upadhyay et al., 2016) – for machine translation (Artetxe et al., 2018a; Lample et al., 2018a,b), POS tagging (Zhang et al., 2016; Ahmad et al., 2019), & named entity recognition (Tsai and Roth, 2016; Xie et al., 2018; Chen et al., 2019). Linguistic Correlation. This work is based on the observation that, monolingual representa- tion spaces learnt independently for different lan- guages tend to exhibit similarity in terms of geo- metric properties and orientations (Mikolov and ∗Work done during internship at Huawei Research, Ireland. † Work done while the author was at Huawei Research, Ireland. Sutskever, 2013) 1. Further, the frequency of words across languages have also been shown to follow the Zipf’s distribution 2, with an overlap of nearly 70% for the most frequent words (Aldarmaki et al., 2018) and 60% for synonyms (Dinu et al., 2015) across language pairs. Existing techniques for ex- tracting cross-lingual word correspondences rely on above inter-dependencies to learn transforma- tions across monolingual embedding spaces. State-of-the-art & Challenges. Early ap- proaches for obtaining multi-lingual word embed- dings used parallel or comparable corpora (Gouws et al., 2015; Mogadala and Rettinger, 2016; Vuli´c and Moens, 2016). However, such methods are not scalable as parallel datasets, especially for low-resource languages, are scarce. Linear trans- formations between two monolingual embedding spaces (via optimization formulation (Schönemann, 1966)) using small manually created bi-lingual dictionaries were thus proposed (Mikolov and Sutskever, 2013; Artetxe et al., 2016). Words having similar surface forms across languages were used to induce seed dictionaries for semi- supervised approaches (Artetxe et al., 2017; Zhou et al., 2019; Doval et al., 2018). Rigid transfor- mation based point set registration Cao and Zhao (2018), supervised cross-lingual alignment, joint training (Joulin et al., 2018; Jawanpuria et …

Content truncated to 500 words