TQ-AutoTest – An Automated Test Suite for (Machine) Translation Quality

TQ-AutoTest – A Semi-Automatic Test Suite for (Machine) Translation Quality Vivien Macketanz, Renlong Ai, Aljoscha Burchardt and Hans Uszkoreit DFKI – Language Technology Lab Berlin, Germany {firstname.lastname}@dfki.de Abstract In several areas of NLP evaluation, test suites have been used to analyze the strengths and weaknesses of systems. Today, Machine Translation (MT) quality is usually assessed by shallow automatic comparisons of MT outputs with reference corpora resulting in a number. Especially the trend towards neural MT has renewed peoples’ interest in better and more analytical diagnostic methods for MT quality. In this paper we present TQ-AutoTest, a novel framework that supports a linguistic evaluation of (machine) translations using test suites. Our current test suites comprise about 5000 handcrafted test items for the language pair German–English. The framework supports the creation of tests and the semi-automatic evaluation of the MT results using regular expressions. The expressions help to classify the results as correct, incorrect or as requiring a manual check. The approach can easily be extended to other NLP tasks where test suites can be used such as evaluating (one-shot) dialogue systems. Keywords: Machine Translation, Quality Evaluation, Test Suites 1. Introduction and Background In several areas of NLP evaluation, test suites have been used to analyze the strengths and weaknesses of systems. In contrast to “real-life” gold standard corpora, test suites can contain made-up or edited input-output pairs to isolate interesting or difficult phenomena. In Machine Translation (MT) research, broadly-defined test suites have not been used apart from several singular at- tempts (King and Falkedal, 1990; Isahara, 1995; Koh et al., 2001, etc.). One of the reasons for this might be the fear that the performance of statistical MT systems depends so much on the particular input data, parameter settings, etc., that relevant conclusions about the errors they make are dif- ficult to obtain. Another concern is that “correct” MT out- put cannot be specified in the same way as the output of other language processing tasks like parsing or fact extrac- tion where the expected results can be more or less clearly defined. Due to the variation of language, ambiguity, etc., checking and evaluating MT output can be almost as diffi- cult as the translation itself. Today, MT quality is still usually assessed by shallow au- tomatic comparisons of MT outputs with reference corpora resulting in a number. Early attempts to automatically clas- sify errors based on post-edits or reference translations like (Popovi´c and Ney, 2011) have not yet become standard. For the detection of certain types of errors like grammar errors, parsers have been used (Tezcan et al., 2016). In other nar- row domains, researchers have started to explore the dif- ferences between systems and between the development stages of one system in more linguistic detail. Especially the trend towards neural MT has renewed peoples’ interest in better and more analytical diagnostic methods for MT quality. Recent work based on specific test suites includes the study of verb-particle constructions (Schottm¨uller and Nivre, 2014), pronouns (Guillou and Hardmeier, 2016) or structural …

Content truncated to 500 words