Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment

Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment Yonatan Belinkov, Tao Lei, Regina Barzilay Massachusetts Institute of Technology {belinkov, taolei, regina}@csail.mit.edu Amir Globerson The Hebrew University gamir@cs.huji.ac.il Abstract Prepositional phrase (PP) attachment disam- biguation is a known challenge in syntactic parsing. The lexical sparsity associated with PP attachments motivates research in word representations that can capture pertinent syn- tactic and semantic features of the word. One promising solution is to use word vectors in- duced from large amounts of raw text. How- ever, state-of-the-art systems that employ such representations yield modest gains in PP at- tachment accuracy. In this paper, we show that word vector repre- sentations can yield significant PP attachment performance gains. This is achieved via a non-linear architecture that is discriminatively trained to maximize PP attachment accuracy. The architecture is initialized with word vec- tors trained from unlabeled data, and relearns those to maximize attachment accuracy. We obtain additional performance gains with al- ternative representations such as dependency- based word vectors. When tested on both En- glish and Arabic datasets, our method outper- forms both a strong SVM classifier and state- of-the-art parsers. For instance, we achieve 82.6% PP attachment accuracy on Arabic, while the Turbo and Charniak self-trained parsers obtain 76.7% and 80.8% respectively.1 1 Introduction The problem of prepositional phrase (PP) attach- ment disambiguation has been under investigation 1The code and data for this work are available at http: //groups.csail.mit.edu/rbg/code/pp. She ate spaghetti with butter She ate spaghetti with chopsticks Figure 1: Two sentences illustrating the importance of lexicalization in PP attachment decisions. In the top sentence, the PP with butter attaches to the noun spaghetti. In the bottom sentence, the PP with chop- sticks attaches to the verb ate. for a long time. However, despite at least two decades of research (Brill and Resnik, 1994; Rat- naparkhi et al., 1994; Collins and Brooks, 1995), it remains a major source of errors for state-of-the-art parsers. For instance, in a comparative evaluation of parser performance on the Wall Street Journal cor- pus, Kummerfeld et al. (2012) report that PP attach- ment is the largest source of errors across all parsers. Moreover, the extent of improvement over time has been rather limited, amounting to about 32% error reduction since the work of (Collins, 1997). PP attachments are inherently lexicalized and part-of-speech (POS) tags are not sufficient for their correct disambiguation. For example, the two sen- tences in Figure 1 vary by a single noun — butter vs chopsticks. However, this word determines the structure of the whole PP attachment. If the corre- 561 Transactions of the Association for Computational Linguistics, vol. 2, pp. 561–572, 2014. Action Editor: Ryan McDonald. Submission batch: 10/2014; Revision batch 11/2014; Published 12/2014. c⃝2014 Association for Computational Linguistics. sponding word is not observed in the training data, a standard lexicalized parser does not have sufficient information to distinguish between these two cases. In fact, 72% of head-child pairs (e.g. spaghetti- butter) from the Wall Street Journal test set are un- …

Content truncated to 500 words