Towards Language Agnostic Universal Representations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4033–4041 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4033 Towards Language Agnostic Universal Representations Armen Aghajanyan ∗ Facebook / Seattle, WA armenag@fb.com Xia Song and Saurabh Tiwary Microsoft / Bellevue, WA {xiaso, satiwary}@microsoft.com Abstract When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both lan- guages the student is fluent in, even if the math lessons were only taught in one language. However, current representations in machine learning are language dependent. In this work, we present a method to decouple the language from the problem by learning language ag- nostic representations and therefore allowing training a model in one language and apply- ing to a different one in a zero shot fashion. We learn these representations by taking in- spiration from linguistics, specifically the Uni- versal Grammar hypothesis and learn univer- sal latent representations that are language ag- nostic (Chomsky, 2014; Montague, 1970). We demonstrate the capabilities of these represen- tations by showing that models trained on a single language using language agnostic rep- resentations achieve very similar accuracies in other languages. 1 Introduction Anecdotally speaking, fluent bilingual speakers rarely face trouble translating a task learned in one language to another. For example, a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages. Furthermore there is a large collection of evidence in linguistics arguing that although separate lexi- cons exist in multilingual speakers the core repre- sentations of concepts and theories are shared in memory (Altarriba, 1992; Mitchel, 2005; Bentin et al., 1985). The fundamental question we’re in- terested in answering is on the learnability of these shared representations within a statistical frame- work. We approached this problem from a linguis- tics perspective. Languages have vastly varying syntactic features and rules. Linguistic Relativ- ity studies the impact of these syntactic variations on the formations of concepts and theories (Au, 1983). Within this framework of study, the two schools of thoughts are linguistic determinism and weak linguistic influence. Linguistic determinism argues that language entirely forms the range of cognitive processes, including the creation of var- ious concepts, but is generally agreed to be false (Hoijer, 1954; Au, 1983). Although there exists some weak linguistic influence, it is by no means fundamental (Ahearn, 2016). The superfluous na- ture of syntactic variations across languages brings forward the argument of principles and param- eters (PnP) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters (e.g. head-first or head- final syntax), as well as common principles shared across all languages (Culicover, 1997). Univer- sal Grammar (UG) is the study of principles and the parameters that are universal across languages (Montague, 1970). The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the specific lan- …

Content truncated to 500 words