Word Order Matters When You Increase Masking

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1808 - 1815 December 7-11, 2022 ©2022 Association for Computational Linguistics Word Order Matters When You Increase Masking Karim Lasri@,B Alessandro LenciB Thierry Poibeau@ @Lattice (École Normale Supérieure-PSL, CNRS, U. Sorbonne Nouvelle) BUniversity of Pisa karim.lasri@ens.psl.eu alessandro.lenci@unipi.it thierry.poibeau@ens.psl.eu Abstract Word order, an essential property of natural lan- guages, is injected in Transformer-based neu- ral language models using position encoding. However, recent experiments have shown that explicit position encoding is not always useful, since some models without such feature man- aged to achieve state-of-the art performance on some tasks. To understand better this phe- nomenon, we examine the effect of removing position encodings on the pre-training objec- tive itself (i.e., masked language modelling), to test whether models can reconstruct position information from co-occurrences alone. We do so by controlling the amount of masked to- kens in the input sentence, as a proxy to affect the importance of position information for the task. We find that the necessity of position information increases with the amount of mask- ing, and that masked language models without position encodings are not able to reconstruct this information on the task. These findings point towards a direct relationship between the amount of masking and the ability of Trans- formers to capture order-sensitive aspects of language using position encoding. 1 Introduction Transformer-based language models have become ubiquitous since they demonstrated improvements in most NLP downstream tasks (Devlin et al., 2019; Liu et al., 2019). A lot of ink has been spilled re- garding the amount of linguistic structure that such models captured (Jawahar et al., 2019), pointing to- wards the acquisition of diverse linguistic abilities. As neural language models need to process infor- mation about the position of their input tokens to capture structural generalizations, a plethora of pro- posals to encode such information have been made (Press et al., 2021; He et al., 2020; Su et al., 2021; Chang et al., 2021; Chen et al., 2021; Chen, 2021). Recent work, however, questioned whether word order information is really useful for pre-trained models to solve downstream tasks (Sinha et al., 2021), showing that models could perform well when using only higher-order co-occurrence statis- tics. Other authors (Haviv et al., 2022) have shown that some transformers could reconstruct partly po- sition information without it being explicitly in- jected. Examining performance on downstream tasks can show that the task simply does not require order information, or that the dataset used to test the model is too easy (Abdou et al., 2022), leading to indirect observations regarding a model’s ability to reconstruct position information. In turn, we choose to test the importance of position encodings for the pre-training task itself, masked language modeling, to get more direct evi- dence about whether and when position matters to language models. We do so under different amounts of masking, as intuitively, position in- formation should be increasingly important when more tokens are missing from the context. Our ex- periments show that …

Content truncated to 500 words