How to Avoid Burning Ducks: Combining Linguistic Analysis and Corpus Statistics for German Compound Processing

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 224–234, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics How to Avoid Burning Ducks: Combining Linguistic Analysis and Corpus Statistics for German Compound Processing Fabienne Fritzinger and Alexander Fraser Institute for Natural Language Processing University of Stuttgart {fritzife, fraser}@ims.uni-stuttgart.de Abstract Compound splitting is an important prob- lem in many NLP applications which must be solved in order to address issues of data sparsity. Previous work has shown that lin- guistic approaches for German compound splitting produce a correct splitting more often, but corpus-driven approaches work best for phrase-based statistical machine translation from German to English, a worrisome contradiction. We address this situation by combining linguistic analysis with corpus-driven statistics and obtain- ing better results in terms of both produc- ing splittings according to a gold standard and statistical machine translation perfor- mance. 1 Introduction Compounds are highly productive in German and cause problems of data sparsity in data-driven sys- tems. Compound splitting is an important com- ponent of German to English statistical machine translation systems. The central result of work by (Koehn and Knight, 2003) is that corpus-driven approaches to compound splitting perform better than approaches based on linguistic analysis, and this result has since been confirmed by other re- searchers (Popovi´c et al., 2006; Stymne, 2008). This is despite the fact that linguistic analysis per- forms better in terms of matching a gold standard splitting. Our work shows that integrating these two approaches, by employing high-recall lin- guistic analysis disambiguated using corpus statis- tics, effectively combines the benefits of both ap- proaches. This is important due to the wide us- age of the Koehn and Knight approach in statisti- cal machine translation systems. The splittings we produce are best in terms of both end-to-end machine translation performance (resulting in an improvement of 0.59 BLEU and 0.84 METEOR over the corpus-driven approach of Koehn and Knight on the development test set used for WMT 20091) and two gold standard eval- uations (see section 4). We provide an exten- sive analysis of the improvements of our approach over the corpus-driven approach. The approach we have developed may help show how to im- prove previous approaches to handling compounds in such applications as speech recognition (e.g., (Larson et al., 2000)) or information retrieval (e.g., (Braschler and Ripplinger, 2004)). The organization of the paper is as follows. Sec- tion 2 discusses previous work on compound split- ting for statistical machine translation. Section 3 presents approaches for compound splitting and also presents SMOR, the morphological analyzer that is a key knowledge source for our approach. Section 4 presents a comparison of compound splitting techniques using two gold standard cor- pora and an error analysis. Section 5 presents phrase-based statistical machine translation (SMT) results. Section 6 concludes. 2 Related Work on German Compound Splitting Rule-based compound splitting for SMT has been addressed by Nießen and Ney (2000), where GERTWOL was used for morphological analysis and the GERCG parser for lexical …

Content truncated to 500 words