Vocabulary-informed Language Encoding

Proceedings of the 29th International Conference on Computational Linguistics, pages 4883–4891 October 12–17, 2022. 4883 Vocabulary-informed Language Encoding Xi Ai College of Computer Science University of Chonqing barid.x.ai@gmail.com Bin Fang College of Computer Science University of Chonqing fb@cqu.edu.cn Abstract A Multilingual model relies on language en- codings to identify input languages because it has to distinguish between the input and out- put languages or among all the languages for cross-lingual tasks. Furthermore, we find that language encodings potentially refine multiple morphologies of different languages to form a better isomorphic space for multilinguality. To leverage this observation, we present a method to compute a vocabulary-informed language encoding as the language representation, for a required language, considering a local vocabu- lary covering an acceptable amount of the most frequent word embeddings in this language. In our experiments, our method can consistently improve the performance of multilingual mod- els on unsupervised neural machine translation and cross-lingual embedding. 1 Introduction With tied weights across required languages, a mul- tilingual model is trained on non-parallel or/and parallel multilingual corpora. Essentially, language encodings are required for cross-lingual tasks be- cause the multilingual model has to distinguish between the input and output languages or among all the languages generally. We observe that, be- sides identifications of languages, language encod- ings can help the model build isomorphic space for multilinguality with the help of shared tokens. Specifically, our hypothesis derives from the de- composition of attention mechanisms (Vania and Lopez, 2017; Luong et al., 2015; Libovický and Helcl, 2017), and we observe explicit alignments and implicit alignments, where explicit alignments are key for language identifications, and implicit alignments promote multilinguality. Furthermore, the implicit alignments are a special case of unsu- pervised bilingual/multilingual lexical induction, helping multilingual models learn multilingual and cross-lingual knowledge. Our goal is to retain the explicit alignments for language identifications and improve the implicit alignments for multilinguality. In this work, we render an analysis of the at- tention mechanism in optimization, and then we find implicit alignments among a language encod- ing and other language encodings. Stemming from morphology adaptation and the observation, we present a method to compute VLE (Vocabulary- informed Language Encoding) as the required lan- guage encodings, for multilingual models. Each required language maintains a local vocabulary cov- ering a subset of the most frequent tokens in this language from the shared vocabulary of the mul- tilingual model. Given one language and its vo- cabulary, when adapting to the multilingual model, we apply transformation layers to the average of token embeddings in the vocabulary for VLE that can be used in either padding style or adding style. VLE provides language characteristics for language identification and leverages implicit alignments for multilinguility. Eventually, the multilingual model can identify languages and learn better multilin- guality with the help of VLE. 2 Related Work and Background 2.1 Language Encoding Let Ex denotes x’s embedding. Given an in- put sentence X = {x0, ..., xn} and the lan- guage encoding ELTl both in language Langl, we have: …

Content truncated to 500 words