Does Typological Blinding Impede Cross-Lingual Sharing?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 480–486 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 480 Does Typological Blinding Impede Cross-Lingual Sharing? Johannes Bjerva Department of Computer Science Aalborg University jbjerva@cs.aau.dk Isabelle Augenstein Department of Computer Science University of Copenhagen augenstein@di.ku.dk Abstract Bridging the performance gap between high- and low-resource languages has been the fo- cus of much previous work. Typological fea- tures from databases such as the World Atlas of Language Structures (WALS) are a prime candidate for this, as such data exists even for very low-resource languages. However, previ- ous work has only found minor benefits from using typological information. Our hypothesis is that a model trained in a cross-lingual set- ting will pick up on typological cues from the input data, thus overshadowing the utility of explicitly using such features. We verify this hypothesis by blinding a model to typological information, and investigate how cross-lingual sharing and performance is impacted. Our model is based on a cross-lingual architecture in which the latent weights governing the shar- ing between languages is learnt during training. We show that (i) preventing this model from exploiting typology severely reduces perfor- mance, while a control experiment reaffirms that (ii) encouraging sharing according to ty- pology somewhat improves performance. 1 Introduction Most languages in the world have little access to NLP technology due to data scarcity (Joshi et al., 2020). Nonetheless, high-quality multilingual rep- resentations can be obtained using only a raw text signal, e.g. via multilingual language modelling (Devlin et al., 2019). Furthermore, structural sim- ilarities of languages are to a large extent docu- mented in typological databases such as the World Atlas of Language Structures (WALS, Dryer and Haspelmath (2013)). Hence, developing models which can take use typological similarities of lan- guages is an important direction in order to alleviate language technology inequalities. While previous work has attempted to use ty- pological information to inform NLP models, our Figure 1: A PoS tagger is exposed (or blinded with gra- dient reversal, −λ) to typological features. Observing α values tells us how typology affects sharing. work differs significantly from such efforts in that we blind a model to this information. Most previ- ous work includes language information as features, by using language IDs, or language embeddings (e.g. Ammar et al. (2016); O’Horan et al. (2016); ¨Ostling and Tiedemann (2017); Ponti et al. (2019); Oncevay et al. (2020)). Notably, limited effects are usually observed from including typological features explicitly. For instance, de Lhoneux et al. (2018) observe positive cross-lingual sharing ef- fects only in a handful of their settings. We there- fore hypothesise that relevant typological informa- tion is learned as a by-product of cross-lingual train- ing. Hence, although models do benefit from this information, it is not necessary to provide it ex- plicitly in a high-resource scenario, where there is abundant training data. This is confirmed by Bjerva and Augenstein (2018a), who find that, e.g., lan- guage embeddings trained on …

Content truncated to 500 words