Feature Optimization for Constituent Parsing via Neural Networks

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1138–1147, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Feature Optimization for Constituent Parsing via Neural Networks Zhiguo Wang IBM Watson 1101 Kitchawan Yorktown Heights, NY, USA zhigwang@us.ibm.com Haitao Mi IBM Watson 1101 Kitchawan Yorktown Heights, NY, USA hmi@us.ibm.com Nianwen Xue Brandeis University 415 South St Waltham, MA, USA xuen@brandeis.edu Abstract The performance of discriminative con- stituent parsing relies crucially on feature engineering, and effective features usu- ally have to be carefully selected through a painful manual process. In this paper, we propose to automatically learn a set of effective features via neural networks. Specifically, we build a feedforward neu- ral network model, which takes as input a few primitive units (words, POS tags and certain contextual tokens) from the lo- cal context, induces the feature represen- tation in the hidden layer and makes pars- ing predictions in the output layer. The network simultaneously learns the feature representation and the prediction model parameters using a back propagation al- gorithm. By pre-training the model on a large amount of automatically parsed data, and then fine-tuning on the manually an- notated Treebank data, our parser achieves the highest F1 score at 86.6% on Chi- nese Treebank 5.1, and a competitive F1 score at 90.7% on English Treebank. More importantly, our parser generalizes well on cross-domain test sets, where we sig- nificantly outperform Berkeley parser by 3.4 points on average for Chinese and 2.5 points for English. 1 Introduction Constituent parsing seeks to uncover the phrase structure representation of sentences that can be used in a variety of natural language applications such as machine translation, information extrac- tion and question answering (Jurafsky and Martin, 2008). One of the major challenges for this task is that constituent parsers require an inference algo- rithm of high computational complexity in order to search over their large structural space, which makes it very hard to efficiently train discrimina- tive models. So, for a long time, the task was mainly solved with generative models (Collins, 1999; Charniak, 2000; Petrov et al., 2006). In the last few years, however, with the use of ef- fective parsing strategies, approximate inference algorithms, and more efficient training methods, discriminative models began to surpass the gen- erative models (Carreras et al., 2008; Zhu et al., 2013; Wang and Xue, 2014). Just like other NLP tasks, the performance of discriminative constituent parsing crucially relies on feature engineering. If the feature set is too small, it might underfit the model and leads to low performance. On the other hand, too many fea- tures may result in an overfitting problem. Usu- ally, an effective set of features have to be de- signed manually and selected through repeated ex- periments (Sagae and Lavie, 2005; Wang et al., 2006; Zhang and Clark, 2009). Not only does this procedure require a lot of expertise, but it is also tedious and time-consuming. Even af- ter …

Content truncated to 500 words