The CoNLL 2007 Shared Task on Dependency Parsing

Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pp. 915–932, Prague, June 2007. c⃝2007 Association for Computational Linguistics The CoNLL 2007 Shared Task on Dependency Parsing Joakim Nivre∗† Johan Hall∗ Sandra K¨ubler‡ Ryan McDonald∗∗ Jens Nilsson∗ Sebastian Riedel†† Deniz Yuret‡‡ ∗V¨axj¨o University, School of Mathematics and Systems Engineering, first.last@vxu.se †Uppsala University, Dept. of Linguistics and Philology, joakim.nivre@lingfil.uu.se ‡Indiana University, Department of Linguistics, skuebler@indiana.edu ∗∗Google Inc., ryanmcd@google.com ††University of Edinburgh, School of Informatics, S.R.Riedel@sms.ed.ac.uk ‡‡Koc¸ University, Dept. of Computer Engineering, dyuret@ku.edu.tr Abstract The Conference on Computational Natural Language Learning features a shared task, in which participants train and test their learn- ing systems on the same data sets. In 2007, as in 2006, the shared task has been devoted to dependency parsing, this year with both a multilingual track and a domain adaptation track. In this paper, we define the tasks of the different tracks and describe how the data sets were created from existing treebanks for ten languages. In addition, we characterize the different approaches of the participating systems, report the test results, and provide a first analysis of these results. 1 Introduction Previous shared tasks of the Conference on Compu- tational Natural Language Learning (CoNLL) have been devoted to chunking (1999, 2000), clause iden- tification (2001), named entity recognition (2002, 2003), and semantic role labeling (2004, 2005). In 2006 the shared task was multilingual dependency parsing, where participants had to train a single parser on data from thirteen different languages, which enabled a comparison not only of parsing and learning methods, but also of the performance that can be achieved for different languages (Buchholz and Marsi, 2006). In dependency-based syntactic parsing, the task is to derive a syntactic structure for an input sentence by identifying the syntactic head of each word in the sentence. This defines a dependency graph, where the nodes are the words of the input sentence and the arcs are the binary relations from head to dependent. Often, but not always, it is assumed that all words except one have a syntactic head, which means that the graph will be a tree with the single independent word as the root. In labeled dependency parsing, we additionally require the parser to assign a specific type (or label) to each dependency relation holding between a head word and a dependent word. In this year’s shared task, we continue to explore data-driven methods for multilingual dependency parsing, but we add a new dimension by also intro- ducing the problem of domain adaptation. The way this was done was by having two separate tracks: a multilingual track using essentially the same setup as last year, but with partly different languages, and a domain adaptation track, where the task was to use machine learning to adapt a parser for a single lan- guage to a new domain. In total, test results were submitted for twenty-three systems in the multilin- gual track, and ten systems in the domain adaptation track (six of which also participated in the multilin- gual track). …

Content truncated to 500 words