Unsupervised Prediction of Acceptability Judgements

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1618–1628, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Unsupervised Prediction of Acceptability Judgements Jey Han Lau, Alexander Clark, and Shalom Lappin jeyhan.lau@gmail.com, alexsclark@gmail.com, shalom.lappin@kcl.ac.uk King’s College London Abstract In this paper we present the task of un- supervised prediction of speakers’ accept- ability judgements. We use a test set generated from the British National Cor- pus (BNC) containing both grammatical sentences and sentences containing a va- riety of syntactic infelicities introduced by round trip machine translation. This set was annotated for acceptability judge- ments through crowd sourcing. We trained a variety of unsupervised language mod- els on the original BNC, and tested them to see the extent to which they could pre- dict mean speakers’ judgements on the test set. To map probability to acceptability, we experimented with several normalisa- tion functions to neutralise the effects of sentence length and word frequencies. We found encouraging results with the unsu- pervised models predicting acceptability across two different datasets. Our method- ology is highly portable to other domains and languages, and the approach has po- tential implications for the representation and the acquisition of linguistic knowl- edge. 1 Introduction Language modelling involves predicting the prob- ability of a sentence. Given a trained model, we can infer the quantitative likelihood that a sentence occurs. Acceptability, on the other hand, indicates the extent to which a sentence is permissible or ac- ceptable to native speakers of the language. While acceptability is affected by frequency and exhibits gradience (Keller, 2001; Sprouse, 2007; Lau et al., 2014), there is limited research on the relationship between acceptability and probability. In this pa- per, we consider the the task of unsupervised pre- diction of acceptability. Speakers have robust intuitions about accept- ability, and acceptability has been consistently rated on various scales (Sprouse and Almeida, 2012). The acceptability of a sentence appears to be relatively unaffected by its length (within cer- tain bounds), or the frequency of its words, prop- erties that we have confirmed experimentally. By contrast sentence probability does depend on these factors. To filter the effects of sentence length and word frequency, we devise normalising functions to map the probability of a sentence (inferred by our unsupervised language models) to an accept- ability score. Keller (2001) and Lau et al. (2014) present ev- idence that acceptability exhibits gradience. Ac- cordingly, we treat acceptability as a continuous variable here. We train a variety of unsuper- vised models for the acceptability prediction task, and we assess the performance of these models by measuring the correlation between their nor- malised acceptability scores and the mean crowd- sourced acceptability judgements on a set of test sentences. There are a number of NLP tasks to which our work can be fruitfully applied. It can be used to evaluate the fluency of the output for machine translation and other language generation systems. It could also contribute …

Content truncated to 500 words