Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 763–777 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 763 The Impact of Positional Encodings on Multilingual Compression Vinit Ravishankar Language Technology Group University of Oslo Norway vinitr@ifi.uio.no Anders Søgaard Department of Computer Science University of Copenhagen Denmark soegaard@di.ku.dk Abstract In order to preserve word-order information in a non-autoregressive setting, transformer ar- chitectures tend to include positional knowl- edge, by (for instance) adding positional en- codings to token embeddings. Several mod- ifications have been proposed over the sinu- soidal positional encodings used in the origi- nal transformer architecture; these include, for instance, separating position encodings and to- ken embeddings, or directly modifying atten- tion weights based on the distance between word pairs. We first show that surprisingly, while these modifications tend to improve monolingual language models, none of them result in better multilingual language models. We then answer why that is: Sinusoidal encod- ings were explicitly designed to facilitate com- positionality by allowing linear projections over arbitrary time steps. Higher variances in multilingual training distributions requires higher compression, in which case, composi- tionality becomes indispensable. Learned ab- solute positional encodings (e.g., in mBERT) tend to approximate sinusoidal embeddings in multilingual settings, but more complex positional encoding architectures lack the in- ductive bias to effectively learn composition- ality and cross-lingual alignment. In other words, while sinusoidal positional encodings were originally designed for monolingual ap- plications, they are particularly useful in mul- tilingual language models. 1 Introduction Multiple recent papers have attempted to pinpoint precisely what components of multilingual lan- guage models enable cross-lingual transfer. Pires et al. (2019) show that although wordpiece over- lap tends to improve cross-lingual transfer perfor- mance, even languages with different scripts (and no shared subwords) may enable zero-shot trans- fer. Wu and Dredze (2019) report similar results on a wider range of tasks. Artetxe et al. (2020) show Sinusoidal See §2 Vaswani et al. (2017) Absolute ((wi + pi)W Q,1) Devlin et al. (2019) ((wj + pj)W K,1)⊤ TUPE (xl iW Q,l)(xl jW K,l)⊤+ Ke et al. (2020) (piU Q)(pjU K)⊤ TUPE(r) . . . +bj−i Relative(k) (xiW Q)(xjW K + aij)⊤ Shaw et al. (2018) Relative(k/q) (xiW Q + aij) Huang et al. (2020) (xjW K + aij)⊤ Table 1: We compare six positional encodings and their impact on cross-lingual generalization in multilingual language models that neither a shared vocabulary nor joint multilin- gual pre-training are necessary to train successful multilingual models. K et al. (2020) find that model depth is a contributor to transfer performance, but that reducing the number of self-attention heads does not have much of an effect. Our starting point is Dufter and Schütze (2021), who claim that a) multilingual compression is caused by forced parameter sharing across lan- guages, and that b) positional encodings play a significant role in the creation of a multilingual space, even in the absence of shared subwords and shared special tokens, like delimiters. Contributions We build on Dufter and Schütze (2021) and …

Content truncated to 500 words