MARMOT: A Toolkit for Translation Quality Estimation at the Word Level

MARMOT: A Toolkit for Translation Quality Estimation at the Word Level Varvara Logacheva§, Chris Hokamp†, Lucia Specia§ §Department of Computer Science, University of Sheffield, UK †CNGL Centre for Global Intelligent Content, Dublin City University, Ireland {v.logacheva,l.specia}@sheffield.ac.uk, chokamp@computing.dcu.ie Abstract We present Marmot — a new toolkit for quality estimation (QE) of machine translation output. Marmot contains utilities targeted at quality estimation at the word and phrase level. However, due to its flexibility and modularity, it can also be extended to work at the sentence level. In addition, it can be used as a framework for extracting features and learning models for many common natural language processing tasks. The tool has a set of state-of-the-art features for QE, and new features can easily be added. The tool is open-source and can be downloaded from https://github.com/qe-team/marmot/. Keywords: machine translation, quality estimation, toolkit 1. Introduction Quality estimation (QE) for machine translation (MT) is a technique aimed at defining the quality of an automatic translation without comparing it to a reference. This task is particularly important for many real-world applications of MT, where no reference translation is available. If MT is used for gisting, a user who can only speak the target lan- guage cannot judge the quality or reliability of its output. If an MT system is embedded into a computer-assisted trans- lation (CAT) tool, quality estimation can increase the pro- ductivity of a user by filtering out translations that are too bad for editing or highlighting the phrases with low quality to make sure they are edited by the user (Turchi et al., 2015; O’Brien et al., 2014). Translation quality can be estimated at different levels of granularity, ranging from the word level to the document level. Thus far, the best results have been achieved in sentence-level QE. This is likely to be because the evalu- ation of the entire sentence allows the use of very many general features, such as language model (LM) scores and posterior translation probabilities. However, the more challenging task of estimating quality at the word level is also very important, both from an applica- tion perspective, and from a research point of view. Fine- grained distinctions can be used to diagnose MT systems and to guide post-editors’ attention to erroneous parts of the sentence. In addition, sentence-level translation quality can be drastically affected by translation errors involving single words or phrases. One of the first attempts to address the problem of QE was the workshop at John Hopkins University (Blatz et al., 2004). While the main focus of the workshop was sentence- level QE, it also included research on word-level QE. In (Ueffing and Ney, 2007), word-level quality is defined as the word-level posterior probability, which is computed as a probability of a word occurring in an n-best list. Raybaud et al. (2009) rely on mutual information between the eval- uated word and its source and target contexts, and an LM enhanced with linguistic features. More recently, the WMT shared task on QE (Bojar et al., 2013; Buck …

Content truncated to 500 words