CoNLL-X Shared Task on Multilingual Dependency Parsing

Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL-X), pages 149–164, New York City, June 2006. c⃝2006 Association for Computational Linguistics CoNLL-X shared task on Multilingual Dependency Parsing Sabine Buchholz Speech Technology Group Cambridge Research Lab Toshiba Research Europe Cambridge CB2 3NH, UK sabine.buchholz@crl.toshiba.co.uk Erwin Marsi Communication & Cognition Tilburg University 5000 LE Tilburg, The Netherlands e.c.marsi@uvt.nl Abstract Each year the Conference on Com- putational Natural Language Learning (CoNLL)1 features a shared task, in which participants train and test their systems on exactly the same data sets, in order to bet- ter compare systems. The tenth CoNLL (CoNLL-X) saw a shared task on Multi- lingual Dependency Parsing. In this pa- per, we describe how treebanks for 13 lan- guages were converted into the same de- pendency format and how parsing perfor- mance was measured. We also give an overview of the parsing approaches that participants took and the results that they achieved. Finally, we try to draw gen- eral conclusions about multi-lingual pars- ing: What makes a particular language, treebank or annotation scheme easier or harder to parse and which phenomena are challenging for any dependency parser? Acknowledgement Many thanks to Amit Dubey and Yuval Kry- molowski, the other two organizers of the shared task, for discussions, converting treebanks, writing software and helping with the papers.2 1see http://ilps.science.uva.nl/˜erikt/signll/conll/ 2Thanks also to Alexander Yeh for additional help with the paper reviews. His work was made possible by the MITRE Cor- poration’s Sponsored Research Program. 1 Introduction Previous CoNLL shared tasks focused on NP chunk- ing (1999), general chunking (2000), clause iden- tification (2001), named entity recognition (2002, 2003), and semantic role labeling (2004, 2005). This shared task on full (dependency) parsing is the log- ical next step. Parsing is an important preprocess- ing step for many NLP applications and therefore of considerable practical interest. It is a complex task and as it is not straightforwardly mappable to a “classical” segmentation, classification or sequence prediction problem, it also poses theoretical chal- lenges to machine learning researchers. During the last decade, much research has been done on data-driven parsing and performance has in- creased steadily. For training these parsers, syntac- tically annotated corpora (treebanks) of thousands to tens of thousands of sentences are necessary; so initially, research has focused on English. Dur- ing the last few years, however, treebanks for other languages have become available and some parsers have been applied to several different languages. See Section 2 for a more detailed overview of re- lated previous research. So far, there has not been much comparison be- tween different dependency parsers on exactly the same data sets (other than for English). One of the reasons is the lack of a de-facto standard for an eval- uation metric (labeled or unlabeled, separate root ac- curacy?), for splitting the data into training and test- ing portions and, in the case of constituency tree- banks converted to dependency format, for this con- version. Another reason are the various annotation 149 schemes and logical data …

Content truncated to 500 words