Surface Statistics of an Unknown Language Indicate How to Parse It

Surface Statistics of an Unknown Language Indicate How to Parse It Dingquan Wang and Jason Eisner Department of Computer Science, Johns Hopkins University {wdd,jason}@cs.jhu.edu Abstract We introduce a novel framework for delex- icalized dependency parsing in a new lan- guage. We show that useful features of the target language can be extracted automati- cally from an unparsed corpus, which con- sists only of gold part-of-speech (POS) se- quences. Providing these features to our neural parser enables it to parse sequences like those in the corpus. Strikingly, our sys- tem has no supervision in the target lan- guage. Rather, it is a multilingual sys- tem that is trained end-to-end on a vari- ety of other languages, so it learns a fea- ture extractor that works well. We show experimentally across multiple languages: (1) Features computed from the unparsed corpus improve parsing accuracy. (2) In- cluding thousands of synthetic languages in the training yields further improvement. (3) Despite being computed from unparsed corpora, our learned task-specific features beat previous work’s interpretable typolog- ical features that require parsed corpora or expert categorization of the language. Our best method improved attachment scores on held-out test languages by an average of 5.6 percentage points over past work that does not inspect the unparsed data (McDonald et al., 2011), and by 20.7 points over past “grammar induction” work that does not use training languages (Naseem et al., 2010). 1 Introduction Dependency parsing is one of the core natural language processing tasks. It aims to parse a given sentence into its dependency tree: a directed graph of labeled syntactic relations between words. Supervised dependency parsers—which are trained using a “treebank” of known parses in the target language—have been very successful (McDonald, 2006; Nivre, 2008; Kiperwasser and Goldberg, 2016). By contrast, the progress of unsupervised dependency parsers has been slow, and they have apparently not been used in any downstream NLP systems (Mareˇcek, 2016). An unsupervised parser does not have access to a treebank, but only to a corpus of unparsed sentences in the target language. Unsupervised parsing has been studied for decades. The most common approach is gram- mar induction (Lari and Young, 1990; Carroll and Charniak, 1992; Klein and Manning, 2004). Grammar induction induces an explicit grammar from the unparsed corpus, such as a probabilis- tic context-free grammar (PCFG), and uses that to parse sentences of the language. This approach has encountered two major difficulties: • Search error: Most formulations of gram- mar induction involve optimizing a highly non-convex objective function such as likeli- hood. The optimization is typically NP-hard (Cohen and Smith, 2012), and approximate local search methods tend to get stuck in lo- cal optima. • Model error: Likelihood does not correlate well with parsing accuracy anyway (Smith, 2006, Figure 3.2). Likelihood optimization seeks latent trees that help to predict the observed sentences, but these unsupervised trees may use a non-standard syntactic anal- ysis or even be optimized to predict non- syntactic properties such as topic. We seek a standard syntactic analysis—what Smith (2006) …

Content truncated to 500 words