It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 387–396, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool Jinho D. Choi Emory University 400 Dowman Dr. Atlanta, GA 30322, USA jchoi31@emory.edu Joel Tetreault Yahoo Labs 229 West 43rd St. New York, NY 10036, USA tetreaul@yahoo-inc.com Amanda Stent Yahoo Labs 229 West 43rd St. New York, NY 10036, USA stent@yahoo-inc.com Abstract The last few years have seen a surge in the number of accurate, fast, publicly avail- able dependency parsers. At the same time, the use of dependency parsing in NLP applications has increased. It can be difficult for a non-expert to select a good “off-the-shelf” parser. We present a com- parative analysis of ten leading statistical dependency parsers on a multi-genre cor- pus of English. For our analysis, we de- veloped a new web-based tool that gives a convenient way of comparing depen- dency parser outputs. Our analysis will help practitioners choose a parser to op- timize their desired speed/accuracy trade- off, and our tool will help practitioners ex- amine and compare parser output. 1 Introduction Dependency parsing is a valuable form of syn- tactic processing for NLP applications due to its transparent lexicalized representation and robust- ness with respect to flexible word order languages. Thanks to over a decade of research on statisti- cal dependency parsing, many dependency parsers are now publicly available. In this paper, we re- port on a comparative analysis of leading statis- tical dependency parsers using a multi-genre cor- pus. Our purpose is not to introduce a new pars- ing algorithm but to assess the performance of ex- isting systems across different genres of language use and to provide tools and recommendations that practitioners can use to choose a dependency parser. The contributions of this work include: • A comparison of the accuracy and speed of ten state-of-the-art dependency parsers, cov- ering a range of approaches, on a large multi- genre corpus of English. • A new web-based tool, DEPENDABLE, for side-by-side comparison and visualization of the output from multiple dependency parsers. • A detailed error analysis for these parsers using DEPENDABLE, with recommendations for parser choice for different factors. • The release of the set of dependencies used in our experiments, the test outputs from all parsers, and the parser-specific models. 2 Related Work There have been several shared tasks on de- pendency parsing conducted by CoNLL (Buch- holz and Marsi, 2006; Nivre and others, 2007; Surdeanu and others, 2008; Hajiˇc and others, 2009), SANCL (Petrov and McDonald, 2012), SPMRL (Seddah and others, 2013), and Se- mEval (Oepen and others, 2014). These shared tasks have led to the public release of numerous statistical parsers. The primary metrics reported in these shared tasks are: labeled attachment score (LAS) – the percentage of predicted dependencies where the arc and the label are assigned correctly; unlabeled attachment score (UAS) – where …

Content truncated to 500 words