Text Genre and Training Data Size in Human-like Parsing

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5846–5852, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5846 Text Genre and Training Data Size in Human-Like Parsing John T. Hale and Adhiguna Kuncoro DeepMind / London {jthale,akuncoro}@google.com Keith B. Hall Google Research / New York kbhall@google.com Chris Dyer DeepMind / London cdyer@google.com Jonathan R. Brennan University of Michigan / Ann Arbor jobrenn@umich.edu Abstract Domain-specific training typically makes NLP systems work better. We show that this ex- tends to cognitive modeling as well by re- lating the states of a neural phrase-structure parser to electrophysiological measures from human participants. These measures were recorded as participants listened to a spo- ken recitation of the same literary text that was supplied as input to the neural parser. Given more training data, the system derives a better cognitive model — but only when the training examples come from the same tex- tual genre. This finding is consistent with the idea that humans adapt syntactic expectations to particular genres during language compre- hension (Kaan and Chun, 2018; Branigan and Pickering, 2017). 1 Introduction Natural language processing (NLP) systems based on deep neural networks are sensitive to the amount and type of training data that they receive. A “data hungry” method may not work well until it is supplied with sufficient examples (e.g. Yo- gatama et al., 2019). Likewise, transfer to a differ- ent textual genre may be poor (Petrov and McDon- ald, 2012). This is the classic problem of domain adaptation1 which arises in many areas of NLP. This paper revisits domain adaptation in the context of human-like parsing. With this human- like aspect in mind, we consider models that use linguistically-plausible trees (see Frank, 2011 for a review) and operate incrementally from left to right (e.g. Steedman, 2000). We quantify the fit to human language performance using freely- available electrophysiological data (henceforth: 1 It remains quite difficult to reconcile human-like in- cremental parsing with high performance out-of-domain; many researchers take a nonincremental whole-sentence ap- proach (Gildea, 2001; Baucom et al., 2013; Joshi et al., 2018). EEG) that was elicited by a pre-existing liter- ary text (Brennan and Hale, 2019). These EEG data come from a naturalistic stim- ulus, and in virtue of their higher temporal resolu- tion, are more detailed than reading times or plau- sibility judgments. Hale et al. (2018) were the first to model them using a neural parser. In that study, textual training data came from the same book as did the human participants’ stimuli. While this yielded a model that was quite well-matched to the EEG modeling task, its training data was confined to just 1543 sentences (24K words). In contrast, recent studies suggest that neural language models require substantially more data to achieve human- like linguistic competence (Gulordava et al., 2018; Futrell et al., 2019; Frank and Hoeks, 2019). We investigate this question of data size to- gether with a contrast …

Content truncated to 500 words