Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars

Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars Ryo Yoshida1 and Hiroshi Noji2 and Yohei Oseki1 1The University of Tokyo 2Artificial Intelligence Research Center, AIST {yoshiryo0617, oseki}@g.ecc.u-tokyo.ac.jp hiroshi.noji@aist.go.jp Abstract In computational linguistics, it has been shown that hierarchical structures make language models (LMs) more human-like. However, the previous literature has been agnostic about a parsing strategy of the hierarchical models. In this paper, we investigated whether hierar- chical structures make LMs more human-like, and if so, which parsing strategy is most cogni- tively plausible. In order to address this ques- tion, we evaluated three LMs against human reading times in Japanese with head-final left- branching structures: Long Short-Term Mem- ory (LSTM) as a sequential model and Re- current Neural Network Grammars (RNNGs) with top-down and left-corner parsing strate- gies as hierarchical models. Our computa- tional modeling demonstrated that left-corner RNNGs outperformed top-down RNNGs and LSTM, suggesting that hierarchical and left- corner architectures are more cognitively plau- sible than top-down or sequential architectures. In addition, the relationships between the cog- nitive plausibility and (i) perplexity, (ii) pars- ing, and (iii) beam size will also be discussed.1 1 Introduction It has been debated in computational linguistics whether language models (LMs) become more human-like by explicitly modeling hierarchical structures of natural languages. Previous work has revealed that while sequential models such as recurrent neural networks (RNNs) can success- fully predict human reading times (Frank and Bod, 2011), there is an advantage in explicitly modeling hierarchical structures (Fossum and Levy, 2012). More recently, RNNs that explicitly model hier- archical structures, namely Recurrent Neural Net- work Grammars (RNNGs, Dyer et al., 2016), have attracted considerable attention, effectively captur- ing grammatical dependencies (e.g., subject-verb 1Our codes are available at https://github.com/ osekilab/RNNG-EyeTrack. agreement) much better than RNNs in targeted syn- tactic evaluations (Kuncoro et al., 2018; Wilcox et al., 2019). In addition, Hale et al. (2018) showed that RNNGs can successfully predict human brain activities, and recommended RNNGs as “a mecha- nistic model of the syntactic processing that occurs during normal human sentence processing.” However, this debate has focused almost exclu- sively on the dichotomy between the hierarchical and sequential models, without reference to the parsing strategies among the hierarchical models. Specifically, although Dyer et al. (2016) and Hale et al. (2018) adopted the vanilla RNNG with a top-down parsing strategy for English with head- initial right-branching structures, Abney and John- son (1991) and Resnik (1992) suggested that the top-down parsing strategy is not optimal for head- final left-branching structures, and alternatively proposed the left-corner parsing strategy as more human-like parsing strategy. In this paper, we investigate whether hierarchi- cal structures make LMs more human-like, and if so, which parsing strategy is most cognitively plausible. In order to address this question, we evaluate three LMs against human reading times in Japanese with head-final left-branching structures: Long Short-Term Memory (LSTM) as a sequential model and Recurrent Neural Network Grammars (RNNGs) with top-down and left-corner parsing strategies as hierarchical models. 2 Linking hypothesis It is well established …

Content truncated to 500 words