Unsupervised Learning of PCFGs with Normalizing Flow

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2442–2452 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2442 Unsupervised Learning of PCFGs with Normalizing Flow Lifeng Jin Department of Linguistics The Ohio State University jin.544@osu.edu Finale Doshi-Velez Harvard University finale@seas.harvard.edu Timothy Miller Boston Children’s Hospital & Harvard Medical School timothy.miller@childrens.harvard.edu William Schuler Department of Linguistics The Ohio State University schuler@ling.osu.edu Lane Schwartz Department of Linguistics University of Illinois at Urbana-Champaign lanes@illinois.edu Abstract Unsupervised PCFG inducers hypothesize sets of compact context-free rules as explanations for sentences. These models not only pro- vide tools for low-resource languages, but also play an important role in modeling language acquisition (Bannard et al., 2009; Abend et al., 2017). However, current PCFG induc- tion models, using word tokens as input, are unable to incorporate semantics and morphol- ogy into induction, and may encounter issues of sparse vocabulary when facing morpholog- ically rich languages. This paper describes a neural PCFG inducer which employs context embeddings (Peters et al., 2018) in a normal- izing flow model (Dinh et al., 2015) to ex- tend PCFG induction to use semantic and mor- phological information1. Linguistically moti- vated similarity penalty and categorical dis- tance constraints are imposed on the inducer as regularization. Experiments show that the PCFG induction model with normalizing flow produces grammars with state-of-the-art accu- racy on a variety of different languages. Abla- tion further shows a positive effect of normal- izing flow, context embeddings and proposed regularizers. 1 Introduction Unsupervised PCFG inducers (Jin et al., 2018b) automatically bracket sentences into nested spans, and label these spans with consistent, linguisti- cally relevant syntactic categories, which may be useful in downstream applications or linguistic re- search on under-resourced languages. Their suc- cess also provides evidence for learnability of grammar in absence of strong linguistic univer- sals (MacWhinney and Bates, 1993; Plunkett and Wood, 2004; Bannard et al., 2009). However, cur- rent PCFG induction models, using word tokens 1The code can be found at https://github.com/ lifengjin/acl_flow as input, are unable to incorporate semantics and morphology into induction, and may encounter is- sues of sparse vocabulary when facing morpholog- ically rich languages. This paper describes a PCFG induction model which exploits recent advances in deep generative models and context embeddings to generalize over rare, morphologically rich forms. We contextual- ize a PCFG’s terminal emission rules with con- text embeddings (Peters et al., 2018) as observa- tions, in order to bring context and subword in- formation into the model. Probabilities for these contextualized terminal emission rules are mod- eled by transforming distributions with normal- izing flow (Rezende and Mohamed, 2015; Dinh et al., 2015; He et al., 2018). Through invert- ible transformations, flow models transform sim- ple distributions (e.g. Gaussian) into complex and potentially multi-modal distributions over obser- vation vectors. These improvements help increase the expressivity of the induction model and give the model the ability to generalize over rare words, but still preserve the tractability of marginal like- lihood computation so that …

Content truncated to 500 words