Effect of Post-processing on Contextualized Word Representations

Proceedings of the 29th International Conference on Computational Linguistics, pages 3127–3142 October 12–17, 2022. 3127 Effect of Post-processing on Contextualized Word Representations Hassan Sajjad♣∗ Firoj Alam♢ Fahim Dalvi♢ Nadir Durrani♢ ♣Faculty of Computer Science, Dalhousie University, Canada ♢Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar hsajjad@dal.ca, {fialam,faimaduddin, ndurrani}@hbku.edu.qa Abstract Post-processing of static embedding has been shown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, we question the usefulness of post-processing for contextualized embeddings obtained from dif- ferent layers of pre-trained language models. More specifically, we standardize individual neuron activations using z-score, min-max nor- malization, and by removing top principal com- ponents using the all-but-the-top method. Ad- ditionally, we apply unit length normalization to word representations. On a diverse set of pre- trained models, we show that post-processing unwraps vital information present in the repre- sentations for both lexical tasks (such as word similarity and analogy) and sequence classifica- tion tasks. Our findings raise interesting points in relation to the research studies that use con- textualized representations, and suggest z-score normalization as an essential step to consider when using them in an application. 1 Introduction Contextualized word embeddings have been cen- tral to the recent revolution in NLP, achieving state- of-the-art performance for many tasks. They are commonly used in the form of fine-tuning based transfer learning and feature extraction (Peters et al., 2018, 2019).1 Feature-based approach gen- erates contextualized embedding vectors and that are used as frozen features in a classifier towards a downstream task. A similar pipeline is used for static embedding except that here a word acquires different embeddings depending on the context it appears in. While fine-tuning based is the more commonly used method, feature-based approach has shown to be a viable alternative with many ap- plications (Peters et al., 2019). For example, it has ∗The work was done while the author was at QCRI 1We have used feature-based transfer learning for this ap- proach in the paper. Figure 1: Layer-wise variance been used as a tool to analyze the inner learning of pre-trained contextualized models (Dalvi et al., 2017; Liu et al., 2019a; Belinkov et al., 2020). The literature on static embedding has empha- sized the usefulness of post-processing of embed- dings to maximize their performance on the down- stream tasks. For example, Mu and Viswanath (2018) showed that making static embedding isotropic is beneficial to lexical and sentence-level tasks. Similarly, Levy et al. (2015); Wilson and Schakel (2015) showed that using normalized word vectors improve performance on word similarity and word relation tasks. While the efficacy of post-processing has been empirically demonstrated for static embedding, it has not been studied whether it can be beneficial when applied to the representations extracted from the contextualized models such as BERT (Devlin et al., 2019), GPT2 (Radford et al., 2018, 2019), etc. Ethayarajh (2019) found contextualized word repre- sentations to be anisotropic. Given that isotropy is a desirable property with proven benefits in …

Content truncated to 500 words