Probing the Linguistic Strengths and Limitations of Unsupervised Grammar Induction

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1395–1404, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Probing the Linguistic Strengths and Limitations of Unsupervised Grammar Induction Yonatan Bisk and Julia Hockenmaier Department of Computer Science The University of Illinois at Urbana-Champaign 201 N Goodwin Ave Urbana, IL 61801 {bisk1,juliahmr@illinois.edu} Abstract Work in grammar induction should help shed light on the amount of syntactic struc- ture that is discoverable from raw word or tag sequences. But since most cur- rent grammar induction algorithms pro- duce unlabeled dependencies, it is diffi- cult to analyze what types of constructions these algorithms can or cannot capture, and, therefore, to identify where additional supervision may be necessary. This pa- per provides an in-depth analysis of the errors made by unsupervised CCG parsers by evaluating them against the labeled de- pendencies in CCGbank, hinting at new research directions necessary for progress in grammar induction. 1 Introduction Grammar induction aims to develop algorithms that can automatically discover the latent syntactic structure of language from raw or part-of-speech tagged text. While such algorithms would have the greatest utility for low-resource languages for which no treebank is available to train supervised parsers, most work in this area has focused on languages where existing treebanks can be used to measure and compare the performance of the resultant parsers. Despite significant progress in the last decade (Klein and Manning, 2004; Head- den III et al., 2009; Blunsom and Cohn, 2010; Spitkovsky et al., 2013; Mareˇcek and Straka, 2013), there has been little analysis performed on the types of errors these induction systems make, and our understanding of what kinds of construc- tions these parsers can or cannot recover is still rather limited. One likely reason for this lack of analysis is the fact that most of the work in this do- main has focused on parsers that return unlabeled dependencies, which cannot easily be assigned a linguistic interpretation. This paper shows that approaches that are based on categorial grammar (Steedman, 2000) are amenable to more stringent evaluation metrics, which enable detailed analyses of the construc- tions they capture, while the commonly used unlabeled directed attachment scores hide linguis- tically important errors. Any categorial grammar based system, whether deriving its grammar from seed knowledge distinguishing nouns and verbs (Bisk and Hockenmaier, 2013), from a lexicon constructed from a simple questionnaire for linguists (Boonkwan and Steedman, 2011), or from sections of a treebank (Garrette et al., 2015), will attach linguistically expressive categories to individual words, and can therefore produce labeled dependencies. We provide a simple proof of concept for how these labeled dependencies can be used to isolate problem areas in CCG induction algorithms. We illustrate how they make the linguistic assumptions and mistakes of the model transparent, and are easily comparable to a treebank where available. They also allow us to identify linguistic phenomena that require addi- tional supervision or training signal to master. …

Content truncated to 500 words