A method of evaluation of the quality of translated text

MT Summit VII Sept. 1999 A Method of Evaluation of the Quality of Translated Text Yamauchi Satoshi Research and Development Group RICOH COMPANY. LTD. 16-1 Shinei-cho Tsuzuki-ku Yokohama, 224-0035, JAPAN yamauchi@int.rdc.ricoh.co.jp Abstract In this paper, I present a method for the evaluation of the quality of translated text, namely, a translation ability index, which shows the relative position of the translation ability of a Machine Translation (MT) system on a measurement scale. The measurements are made by an analysis ratio which is able to make absolute measurements and a conversion and magnitude scale (CGMS) which indicates the mutual relation of the machine translated text to the text translated by the professional human translator. The translation ability index in this work has been confirmed by the evaluation of two MT systems. This is effective as a clear explanation of this work. 1 Introduction Evaluating or measuring techniques are very important for promoting and indicating progress made in technologies in general. In Japan, the MT systems have been used by a certain number of people in different limited domains. However, anyone who has ever used a MT system hopes that it will be improved upon. So, we have to make a breakthrough in MT technology, and build up an evaluation method that is easy and effective for people to use in choosing a MT system. After ALPAC (Automatic Language Processing Advisory Committee) [1] had first introduced an evaluation method for the quality of translated text in 1966, several evaluation methods were offered from many researchers and research groups [2][3][4][5]. No method, however, was objective but each one had a partly a subjective factor since two kinds of text belonging to two different cultures must be compared. When an individual evaluates a MT system, his or her judgment when using an evaluation criteria is not consistent. So the result of the evaluation may not be unique because the person who evaluates each translated text has to choose a ranking from various steps determined by the evaluation criteria. I introduce a new evaluation method that makes it easy for users to choose a MT system without their worrying about issues of the quality of the translated text. The heart of the method is to evaluate generated text in the target language in all translation procedures. There are two steps of evaluation in the total translation ability evaluation. The first step of the evaluation is to check the source text analysis ability. The next step is to check for quality of conversion and generation. The product of these results is the translation ability index. In the following description, I show the result which was obtained in the evaluation of two MT systems. 2 The Evaluation Method For the evaluation, you need three sets of documents as part of the evaluation criteria. (The original texts that were translated were selected at random in a limited area or field, ignoring any criteria such as suitability for translation by computer.) Bilingual documents must be prepared, i.e. the original …

Content truncated to 500 words