How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context

How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context Jey Han Lau1,7 Carlos Armendariz2 Shalom Lappin2,3,4 Matthew Purver2,5 Chang Shu6,7 1The University of Melbourne 3University of Gothenburg2Queen Mary University of London 4King’s College London 5Joˇzef Stefan Institute 6University of Nottingham Ningbo China 7DeepBrain jeyhan.lau@gmail.com, c.santosarmendariz@qmul.ac.uk shalom.lappin@gu.se, m.purver@qmul.ac.uk, scxcs1@nottingham.edu.cn Abstract We study the influence of context on sentence acceptability. First we compare the acceptabil- ity ratings of sentences judged in isolation, with a relevant context, and with an irrelevant context. Our results show that context induces a cognitive load for humans, which com- presses the distribution of ratings. Moreover, in relevant contexts we observe a discourse coherence effect that uniformly raises ac- ceptability. Next, we test unidirectional and bidirectional language models in their ability to predict acceptability ratings. The bidirectional models show very promising results, with the best model achieving a new state-of-the-art for unsupervised acceptability prediction.The two sets of experiments provide insights into the cognitive aspects of sentence processing and central issues in the computational modeling of text and discourse. 1 Introduction Sentence acceptability is the extent to which a sentence appears natural to native speakers of a language. Linguists have often used this property to motivate grammatical theories. Computational language processing has traditionally been more concerned with likelihood—the probability of a sentence being produced or encountered. The question of whether and how these properties are related is a fundamental one. Lau et al. (2017b) experiment with unsupervised language models to predict acceptability, and they obtained an encouraging correlation with human ratings. This raises foundational questions about the nature of linguistic knowledge: If probabilistic models can acquire knowledge of sentence acceptability from raw texts, we have prima facie support for an alternative view of language acquisition that does not rely on a categorical grammaticality component. It is generally assumed that our perception of sentence acceptability is influenced by context. Sentences that may appear odd in isolation can become natural in some environments, and sen- tences that seem perfectly well formed in some contexts are odd in others. On the computational side, much recent progress in language modeling has been achieved through the ability to incor- porate more document context, using broader and deeper models (e.g., Devlin et al., 2019; Yang et al., 2019). While most language modeling is restricted to individual sentences, models can benefit fromusing additional context (Khandelwal et al., 2018). However, despite the importance of context, few psycholinguistic or computational studies systematically investigate how context affects acceptability, or the ability of language models to predict human acceptability judgments. Tworecentstudiesthat explorethe impact of doc- ument context on acceptability judgments both identify a compression effect (Bernardy et al., 2018; Bizzoni and Lappin, 2019). Sentences per- ceived to be low in acceptability when judged without context receive a boost in acceptability when judged within context. Conversely, those with high out-of-context acceptability see a reduc- tion in acceptability when context is presented. It is unclear what causes this compression effect. Is it a result of cognitive load, imposed by additional …

Content truncated to 500 words