Serial Recall Effects in Neural Language Modeling

Proceedings of NAACL-HLT 2019, pages 688–694 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 688 Serial Recall Effects in Neural Language Modeling Hassan Hajipoor1, Hadi Amiri2, Maseud Rahgozar1, Farhad Oroumchian3 1School of Electrical and Computer Engineering, University of Tehran, Iran 2Department of Biomedical Informatics, Harvard University, Boston, USA 3University of Wollongong in Dubai, Knowledge Village, Dubai, UAE h.hajipoor@ut.ac.ir, hadi@hms.harvard.edu, rahgozar@ut.ac.ir,farhadoroumchian@uowdubai.ac.ae Abstract Serial recall experiments study the ability of humans to recall words in the order in which they occurred. The following serial recall ef- fects are generally investigated in studies with humans: word length and frequency, primacy and recency, semantic confusion, repetition, and transposition effects. In this research, we investigate LSTM language models in the con- text of these serial recall effects. Our work pro- vides a framework to better understand and an- alyze neural language models and opens a new window to develop accurate language models. 1 Introduction The goal of language modeling is to estimate the probability of a sequence of words in natural lan- guage, typically allowing one to make probabilis- tic predictions of the next word given preced- ing ones (Bahl et al., 1983; Berger et al., 1996). For several years now, Long Short-Term Mem- ory (LSTM) (Graves, 2013) language models have demonstrated state-of-the-art performance (Melis et al., 2018; Merity et al., 2018a; Sundermeyer et al., 2012). Recent studies have begun to shed light on the information encoded by LSTM net- works. These models can effectively use distant history (about 200 tokens of context) and are sensi- tive to word order, replacement, or removal (Khan- delwal et al., 2018), can learn function words much better than content words (Ford et al., 2018), can re- member sentence lengths, word identity, and word order (Adi et al., 2017), can capture syntactic struc- tures (Kuncoro et al., 2018) such as subject-verb agreement (Linzen et al., 2016). These characteris- tics are often attributed to LSTM’s ability in over- coming the curse of dimensionality–by associating a distributed feature vector to each word (Hinton et al., 1986; Neubig, 2017)–and modeling long- range dependencies in faraway context (Khandel- wal et al., 2018). The goal of our research is to complement the prior work to provide a richer understanding about how LSTM language models use prior linguistic context. Inspired by investigations in cognitive psychology about serial recall in humans (Avons et al., 1994; Henson, 1998; Poliˇsensk´a et al., 2015)– where participants are asked to recall a sequence of items in order in which they were presented, we investigate how word length or frequency (word- frequency effect), word position (primacy, recency, and transposition effects), word similarity (seman- tic confusion effect), and word repetition (repeti- tion effect) influence learning in LSTM language models. Our investigation provides a framework to better understand and analyze language models at a considerably finer-grained level than previous studies, and opens a new window to develop more accurate language models. We find that LSTM language models (a) can learn frequent/shorter words considerably better than infrequent/longer ones, (b) …

Content truncated to 500 words