Frustratingly Hard Domain Adaptation for Dependency Parsing

Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pp. 1051–1055, Prague, June 2007. c⃝2007 Association for Computational Linguistics Frustratingly Hard Domain Adaptation for Dependency Parsing Mark Dredze1 and John Blitzer1 and Partha Pratim Talukdar1 and Kuzman Ganchev1 and Jo˜ao V. Grac¸a2 and Fernando Pereira1 1CIS Dept., University of Pennsylvania, Philadelphia, PA 19104 {mdredze|blitzer|partha|kuzman|pereira}@seas.upenn.edu 2L2F – INESC-ID Lisboa/IST, Rua Alves Redol 9, 1000-029, Lisboa, Portugal javg@l2f.inesc-id.pt Abstract We describe some challenges of adaptation in the 2007 CoNLL Shared Task on Domain Adaptation. Our error analysis for this task suggests that a primary source of error is differences in annotation guidelines between treebanks. Our suspicions are supported by the observation that no team was able to im- prove target domain performance substan- tially over a state of the art baseline. 1 Introduction Dependency parsing, an important NLP task, can be done with high levels of accuracy. However, adapt- ing parsers to new domains without target domain labeled training data remains an open problem. This paper outlines our participation in the 2007 CoNLL Shared Task on Domain Adaptation (Nivre et al., 2007). The goal was to adapt a parser trained on a single source domain to a new target domain us- ing only unlabeled data. We were given around 15K sentences of labeled text from the Wall Street Journal (WSJ) (Marcus et al., 1993; Johansson and Nugues, 2007) as well as 200K unlabeled sentences. The development data was 200 sentences of labeled biomedical oncology text (BIO, the ONCO portion of the Penn Biomedical Treebank), as well as 200K unlabeled sentences (Kulick et al., 2004). The two test domains were a collection of medline chem- istry abstracts (pchem, the CYP portion of the Penn Biomedical Treebank) and the Child Language Data Exchange System corpus (CHILDES) (MacWhin- ney, 2000; Brown, 1973). We used the second or- der two stage parser and edge labeler of McDonald et al. (2006), which achieved top results in the 2006 CoNLL-X shared task. Preliminary experiments in- dicated that the edge labeler was fairly robust to do- main adaptation, lowering accuracy by 3% in the de- velopment domain as opposed to 2% in the source, so we focused on unlabeled dependency parsing. Our system did well, officially coming in 3rd place out of 12 teams and within 1% of the top sys- tem (Table 1). 1 In unlabeled parsing, we scored 1st and 2nd on CHILDES and pchem respectively. However, our results were obtained without adap- tation. Given our position in the ranking, this sug- gests that no team was able to significantly improve performance on either test domain beyond that of a state-of-the-art parser. After much effort in developing adaptation meth- ods, it is critical to understand the causes of these negative results. In what follows, we provide an er- ror analysis that attributes domain loss for this task to a difference in annotation guidelines between do- mains. We then overview our attempts to improve adaptation. While we were able to show limited adaptation on reduced training data …

Content truncated to 500 words