An Empirical Study of Memorization in NLP

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6265 - 6278 May 22-27, 2022 c⃝2022 Association for Computational Linguistics An Empirical Study of Memorization in NLP Xiaosen Zheng Singapore Management University xszheng.2020@phdcs.smu.edu.sg Jing Jiang Singapore Management University jingjiang@smu.edu.sg Abstract A recent study by Feldman (2020) proposed a long-tail theory to explain the memorization behavior of deep learning models. However, memorization has not been empirically veri- fied in the context of NLP, a gap addressed by this work. In this paper, we use three different NLP tasks to check if the long-tail theory holds. Our experiments demonstrate that top-ranked memorized training instances are likely atypi- cal, and removing the top-memorized training instances leads to a more serious drop in test accuracy compared with removing training in- stances randomly. Furthermore, we develop an attribution method to better understand why a training instance is memorized. We empirically show that our memorization attribution method is faithful and share our interesting finding that the top-memorized parts of a training instance tend to be features negatively correlated with the class label. 1 Introduction In recent years, there has been an increasing amount of interest in the machine learning commu- nity to understand the memorization behaviour of deep neural network models. Studies have shown that deep learning models often have sufficient ca- pacities to “memorize” training examples (Zhang et al., 2017; Arpit et al., 2017). A number of re- cent studies tried to understand how memorization helps generalization (Chatterjee, 2018; Feldman, 2020; Montanari and Zhong, 2020; Khandelwal et al., 2020, 2021) In NLP, memorization of training examples by deep learning models is also often observed (Li and Wisniewski, 2021; Lewis et al., 2021; Raunak et al., 2021), and existing studies usually see memoriza- tion as something that hinders generalization. For example, Elangovan et al. (2021) tried to measure the amount of “data leakage” in NLP datasets in order to assess a model’s ability to memorize vs. its ability to generalize. However, recently Feldman (2020) proposed a long-tail theory, which states that memorization is necessary for generalization if the data follows a long-tail distribution. This theory was later em- pirically validated by Feldman and Zhang (2020), but their validation was done in only the computer vision domain. It is therefore interesting and useful for us to study whether the long-tail theory also holds in NLP; such validation would help us bet- ter understand the utility of memorization in the context of NLP. The long-tail theory states that if the training data form a long-tail distribution, where there are many small “sub-populations” that are atypical in- stances, and if these small sub-populations are also present in the test data, then memorizing these atyp- ical instances helps the model generalize to the test data. In order to validate this long-tail theory in the context of NLP, we follow the experiments and analyses on image classification done by Feldman and Zhang (2020). Specifically, we aim to answer the following questions in this …

Content truncated to 500 words