A machine translationness typology for MT evaluations

A Machine Translationness Typology for MT Evaluations Joaquim Moré López and Salvador Climent Roca Universitat Oberta de Catalunya (UOC) jmore@uoc.edu; scliment@uoc.edu Abstract. In this article we present machine translationness (MTness, henceforth) as an approach for MT evaluations, in contrast to the notion of human likeness which is the basic criteria for state-of-the-art automatic evaluations. Our approach is based on the detection of phenomena that contribute to the MTness of a translation and, consequently, its poor fluency. We present here a typology of MTness based on an empirical study in which human and machine translations were Turing tested. We also discuss the types of the typology that should be focused on in MT evaluations, the subjective quality of the human likeness assumption, and the advantages of the MTness approach, in terms of time and financial costs. Keywords: machine translationness, human likeness, evaluation . 1 Introduction Automatic MT evaluations are generally based on the assumption that everyone would agree in considering a translation made by a professional translator as a good translation (Human Translation Goodness, HTG henceforth). This is the basic idea behind metrics based on the Assumption of Reference Proximity (ARP) such as BLEU [1], evaluations which consist in classifying automatically translations as human (good) or machine (bad) [2] and the approach which combines different metrics in order to give a result which correlates the closest to the ‘human likeness’ of MT translations [3]. However, contrary to MT evaluation, little is said about evaluating human translations (HT), which should be a prior step if we want to use them in MT evaluations; their good quality is generally taken for granted, but sometimes some HT translations prove to be worse than their MT counterparts and human translators also use MT systems and present MT output as original [4]. Thus, we present an assumption we consider will be more widely – if not universally – agreed upon: translations that sound like machine translations are bad. Actually, this assumption is implied in the methodology based on HTG: what sounds human-like is good, thus what sounds machine-like is bad. Therefore, if we focus not on the human-likeness but on the machine-likeness of a translation, that is to say, its machine-translationness (MTness), we could carry out fast evaluations on the fly that would convey a significant idea about the quality of a system and its main drawbacks ([5] and [6]). Depending on the purpose of the evaluation, focusing on examples of MTness is enough and saves the time and money-consuming task of preparing human translation references or collecting huge training corpora of machine 12th EAMT conference, 22-23 September 2008, Hamburg, Germany 130 and human translations for the human-machine classifier. Likewise, the cost of evaluating the human translations would also be saved. Finally, the evaluation based on detecting MTness would use a more reliable criterion. For this reason, it is important to establish a typology of MTness features. This paper presents an MTness typology from the output of 2 Catalan-Spanish and 2 English-Spanish MT systems. We also …

Content truncated to 500 words