Towards a Comprehensive Evaluation Method of Memory-Based Translation Systems

Towards a comprehensive evaluation method of Memory-Based Translation Systems Gábor Hodász Pázmány Péter Catholic University Faculty of Information Technology Práter utca 50/a. Budapest 1083, Hungary hodasz@itk.ppke.hu Abstract. The paper gives an overview of the evaluation methods of memory-based translation systems: Translation Memories (TM) and Example Based Machine Translation (EBMT) systems. After a short comparison with the well-discussed methods of evaluation of Machine Translation (MT) Systems we give a brief overview of current methodology on memory-based applications. We propose a new aspect, which takes the content of memory into account: a measure to describe the correspondence between the memory and the current segment to translate. We also offer a brief survey of a linguistically enriched translation memory on which these new methods will be tested. 1. Introduction The paper gives an overview of the evaluation methods of memory-based translation systems: Translation Memories (TM) and Example Based Machine Translation (EBMT) systems. After a short comparison with the well-discussed methods of evaluation of Machine Translation (MT) Systems we give a brief overview of current methodology on memory-based applications. We propose a new aspect, which takes the content of memory into account: a measure to describe the correspondence between the memory and the current segment to translate. We also offer a brief survey of a linguistically enriched translation memory on which these new methods will be tested. The literature of MT Systems discusses the theme of evaluation exhaustively and plenty of methods and measures arose in the past decades. Whilst in the case of translation memories and (in many respects very similar) EBMT systems the methods are less comprehensive. The evaluation of a machine translation system aims to measure a distance (or similarity) between the output of the system and a human translation used as a gold standard. Whereas the evaluation of the memory-based systems examines the effectiveness of the reuse of the segments in the memory. Beyond that the aim of such systems is to help the work of a (potentially professional) translator, therefore the “usefulness” of the output can be defined as well. The method of evaluation also can rely on the co-operation of the user, i.e. the human scoring of the suggested translation sentence-by-sentence, because this is the way of usage of a TM [Somers 2003]. 2. The MetaMorpho TM system The MetaMorpho TM system is a linguistically enriched translation memory, based on sub- sentential segments and a similarity measure rested on morpho-syntactic similarity [Hodasz&Pohl 2005]. Our aim was to develop an improved TM system that uses linguistic analysis in both source and destination language sides to yield more exact matches to the source sentence. The MetaMorpho TM stores and retrieves sub-sentential segments and uses a linguistically based measure to determine similarity between two source-language segments, and attempts to assemble a sensible translation using translations of source-language chunks if the entire source segment was not found. To describe the basic operation of the proposed TM engine is not the topic of this paper. To get a detailed description, see [Hodász,Grőbler&Kis 2004]. …

Content truncated to 500 words