Derivational morphology to the rescue: how it can help resolve unfound words in MT

Derivational Morphology to the Rescue: How It Can Help Resolve Unfound Words in MT Claudia Gdaniec, Esmé Manandise, Michael C. McCord IBM Thomas J. Watson Research Center Yorktown Heights, New York 10598 USA {cgdaniec, esme, mcmcord}@us.ibm.com Abstract Machine Translation (MT) systems that process unrestricted text should be able to deal with words that are not found in the MT lexicon. Without some kind of recognition, the parse may be incomplete, there is no transfer for the unfound word, and tests for transfers for surrounding words will often fail, resulting in poor translation. Interestingly, not much has been published on unfound- word guessing in the context of MT although such work has been going on for other applications. In our work on the IBM MT system, we implemented a far-reaching strategy for recognizing unfound words based on rules of word formation and for generating transfers. What distinguishes our approach from others is the use of semantic and syntactic features for both analysis and transfer, a scoring system to assign levels of confidence to possible word structures, and the creation of transfers in the transformation component. We also successfully applied rules of derivational morphological analysis to non-derived unfound words. Keywords Unfound words, derivational morphology, computational morphology, transfer system, scoring mechanism 1 Problem Natural language processing (NLP) systems whose parsers rely on the morphological and syntactic information associated with words, such as part of speech (POS), gender, number, case etc., run into problems when words are not found in the lexicon. Generally, such unfound words contribute to the bad parsing of input text. In MT systems, these words are often resolved as nouns, sometimes flagged as unfound, and never translated. Apart from possible bad parsing, the semantic information for determining the correct transfer of surrounding words is also missing. 2 Motivation and Purpose Updating the lexicon would be the most direct way to remedy this problem. However, this is a time-consuming and costly enterprise. And since productive word- formation rules allow the creation of new words almost on a daily basis, no lexicon is ever complete. Our linguistic intuition tells us that knowledge of word structures and the regularity of word formation can be formalized into a procedure to deal with unfound words not only for an improved analysis of the input but also to generate a more meaningful MT output. We implemented a language-independent strategy for responding to unfound words in arbitrary unrestricted texts through the morphological analyzer (MA) of the MT system LMT developed at IBM, whose main application is Web-page translation in several languages. Another crucial part of this work was the generation of transfers in our transformations. For the work described in this paper, French and German1 were chosen as source languages and English as the target language. 3 Related work Researchers in NLP have developed MAs that incorporate derivational morphology as a strategy to improve application performance. See for example: general purpose analyzers (Byrd et al. 1986), analyzers for improved tagging and parsing (Daciuk, forthcoming), and for enhanced …

Content truncated to 500 words