Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2453–2463 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2453 Variance of average surprisal: a better predictor for quality of grammar from unsupervised PCFG induction Lifeng Jin and William Schuler Department of Linguistics The Ohio State University, Columbus, OH, USA {jin, schuler}@ling.osu.edu Abstract In unsupervised grammar induction, data like- lihood is known to be only weakly cor- related with parsing accuracy, especially at convergence after multiple runs. In order to find a better indicator for quality of in- duced grammars, this paper correlates sev- eral linguistically- and psycholinguistically- motivated predictors to parsing accuracy on a large multilingual grammar induction eval- uation data set. Results show that variance of average surprisal (VAS) better correlates with parsing accuracy than data likelihood, and that using VAS instead of data likelihood for model selection provides a significant ac- curacy boost. Further evidence shows VAS to be a better candidate than data likelihood for predicting word order typology classifica- tion. Analyses show that VAS seems to sepa- rate content words from function words in nat- ural language grammars, and to better arrange words with different frequencies into separate classes that are more consistent with linguistic theory. 1 Introduction Unsupervised grammar induction models learn to produce hierarchical structures for strings of words. Previous work (Seginer, 2007; Ponvert et al., 2011; Shain et al., 2016; Jin et al., 2018b) show that using data likelihood as both the objec- tive for optimization and the criterion for model selection, either implicitly (in the case of Bayesian models) or explicitly (in the case of EM), gives good results on grammar induction. However, it is also known that data likelihood is only weakly cor- related with parsing accuracy, especially at con- vergence (Smith, 2006; Johnson et al., 2007; Jin et al., 2018a). This weak correlation points to the fact that the maximization of data likelihood at convergence may be non-optimal for model selec- tion, and this non-optimality indicates other con- straints on learning may be at work in human ac- quisition. In this work, several linguistically- and psycholinguistically-motivated constraints related to syntax are explored as predictors of parsing ac- curacy for grammars learned by unsupervised in- duction (Jin et al., 2018a). Results show that vari- ance of average surprisal (VAS) is better corre- lated with parsing accuracy of induced grammars than data likelihood. Using VAS for model se- lection at convergence also produces significantly higher parsing accuracy. Further evidence shows VAS to be a better candidate than data likelihood for predicting word order typology classification. Analyses show that VAS seems to separate content words from function words in natural language grammars, and seems to better arrange words with different frequencies into separate classes that are more consistent with linguistic theory. 2 Related work Induction of PCFGs has previously been consid- ered a difficult problem (Carroll and Charniak, 1992; Johnson et al., 2007; Liang et al., 2009; Tu, 2012). Earlier work attributed the lack …

Content truncated to 500 words