Register-based machine translation evaluation with text classification techniques

Register-Based Machine Translation Evaluation with Text Classification Techniques Mihaela Vela m.vela@mx.uni-saarland.de Saarland University, Saarbr¨ucken, Germany Ekaterina Lapshinova-Koltunski e.lapshinova@mx.uni-saarland.de Saarland University, Saarbr¨ucken, Germany Abstract This paper presents a novel approach to machine translation evaluation by combining register features – characterised by particular distributions of lexico-grammatical features – with text classification techniques. The goal of this method is to compare machine translation output with comparable originals in the same language, as well as with human reference translations. The degree of similarity – in terms of register features – between machine translations and originals, and machine translations and reference translations is measured by applying two text classification methods trained on 1) originals and 2) reference translations, and tested on machine translations. The results from the experiments prove our assumption that machine translations share register features rather with human translations than with non-translated texts produced by humans. This confirms that registers are one of the most important factors that should be integrated into register-based machine translation evaluation. 1 Introduction: Motivation and Goals The state-of-the-art in evaluating machine translation (MT) nowadays is to measure lexical and eventually syntactic and semantic overlap between a machine translation (called hypoth- esis translation) and a human-produced reference translation. In this paper, we present a new approach to evaluation, integrating the knowledge on register, i.e. language variation accord- ing to context, as defined by Halliday and Hasan (1989) and Quirk et al. (1985). The dif- ference in terms of register between original and translated texts has been shown by several studies (Hansen-Schirra et al., 2012; Kruger and van Rooy, 2012; Neumann, 2013), proving that translations tend to share a set of lexical, syntactic and/or textual features. More recent investigations by Baroni and Bernardini (2006), Kurokawa et al. (2009) and Lembersky et al. (2012) applied text classification methods to automatically identify these differences. The aim of the research presented here is twofold: 1) to show that machine translations and the corresponding reference translations are related to each other in terms of register-specific features and as a consequence of this 2) to show that hypothesis translations and human trans- lations share more than the lexical surface. The novel idea introduced here is the notion of register-specific features which relate reference and hypothesis translations, and therefore have implications for MT (evaluation). We measure the “closeness” between comparable non-translated originals and machine translations, as well as between human and machine translations by applying two different clas- sification methods. The classification is performed on the basis of extracted register-specific features for two data sets. First, we use original non-translated texts as training data and ma- Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 215 chine translations as test data. In a second step, human translations are used as training data and machine translations as test data. Our assumption is that in terms of register specificity quantified in the corresponding features, MT output is closer to the corresponding reference translations than to the comparable non-translated originals. …

Content truncated to 500 words