Empirical Methods for Compound Splitting

plan Akti on plan Akt actionplan act ion plan action plan Figure 1: Splitting options for the German word Aktionsplan Aktionsplan Empirical Methods for Compound Splitting Philipp Koehn Information Sciences Institute Department of Computer Science University of Southern California koehn@isi.edu Kevin Knight Information Sciences Institute Department of Computer Science University of Southern California knight@isi.edu Abstract Compounded words are a challenge for NLP applications such as machine trans- lation (MT). We introduce methods to learn splitting rules from monolingual and parallel corpora. We evaluate them against a gold standard and measure their impact on performance of statisti- cal MT systems. Results show accuracy of 99.1% and performance gains for MT of 0.039 BLEU on a German-English noun phrase translation task. 1 Introduction Compounding of words is common in a number of languages (German, Dutch, Finnish, Greek, etc.). Since words may be joined freely, this vastly in- creases the vocabulary size, leading to sparse data problems. This poses challenges for a number of NLP applications such as machine translation, speech recognition, text classification, information extraction, or information retrieval. For machine translation, the splitting of an un- known compound into its parts enables the transla- tion of the compound by the translation of its parts. Take the word Aktionsplan in German (see Fig- ure 1), which was created by joining the words Ak- tion and Plan. Breaking up this compound would assist the translation into English as action plan. Compound splitting is a well defined compu- tational linguistics task. One way to define the goal of compound splitting is to break up foreign words, so that a one-to-one correspondence to En- glish can be established. Note that we are looking for a one-to-one correspondence to English con- tent words: Say, the preferred translation of Ak- tionsplan is plan for action. The lack of corre- spondence for the English word for does not de- tract from the definition of the task: We would still like to break up the German compound into the two parts Aktion and Plan. The insertion of function words is not our concern. Ultimately, the purpose of this work is to im- prove the quality of machine translation systems. For instance, phrase-based translation systems [Marcu and Wong, 2002] may recover more eas- ily from splitting regimes that do not create a one-to-one translation correspondence. One split- ting method may mistakenly break up the word Aktionsplan into the three words Akt, Ion, and Plan. But if we consistently break up the word Aktion into Akt and Ion in our training data, such a 187 system will likely learn the translation of the word pair Akt Ion into the single English word action. These considerations lead us to three different objectives and therefore three different evaluation metrics for the task of compound splitting: • One-to-One correspondence • Translation quality with a word-based trans- lation system • Translation quality with a phrase-based trans- lation system For the first objective, we compare the output of our methods to a manually created gold stan- dard. …

Content truncated to 500 words