Quality Estimation

With the noticeable improvement in the overall quality of Machine Translation (MT) systems in recent years, post-editing of MT output is starting to become a com- mon practice among human …...

In this paper we introduce a number of new features for quality estimation in machine translation that were developed for the WMT 2012 quality estimation shared task. We find that …...

This paper describes a study on the contribu- tion of linguistically-informed features to the task of quality estimation for machine trans- lation at sentence level. A standard regression algorithm is …...

In this paper we present the system we sub- mitted to the WMT12 shared task on Quality Estimation. Each translated sentence is given a score between 1 and 5. The …...

We present a method we used for the quality estimation shared task of WMT 2012 involving IBM1 and language model scores calculated on morphemes and POS tags. The IBM1 scores …...

This is a description of the submissions made by the pattern recognition and human lan- guage technology group (PRHLT) of the Uni- versitat Polit`ecnica de Val`encia to the qual- ity …...

We present the approach we took for our par- ticipation to the WMT12 Quality Estimation Shared Task: our main goal is to achieve rea- sonably good results without appeal to …...

The usefulness of a translated text for gist- ing purposes strongly depends on the over- all translation quality of the text, but espe- cially on the translation quality of the …...

In this paper, we describe the UPC system that participated in the WMT 2012 shared task on Quality Estimation for Machine Translation. Based on the empirical evidence that fluency- related …...

This paper describes Uppsala University’s submissions to the Quality Estimation (QE) shared task at WMT 2012. We present a QE system based on Support Vector Machine re- gression, using a …...

In this paper we present the approach and system setup of the joint participation of Fondazione Bruno Kessler and University of Edinburgh in the WMT 2013 Quality Estimation shared-task. Our …...

This paper describes the machine learning algorithm and the features used by LIMSI for the Quality Estimation Shared Task. Our submission mainly aims at evaluating the usefulness for quality estimation …...

In this paper we present the system we submitted to the WMT13 shared task on Quality Estimation. We participated in the Task 1.1. Each translated sentence is given a score …...

In this paper we present our entry to the WMT’13 shared task: Quality Estima- tion (QE) for machine translation (MT). We participated in the 1.1, 1.2 and 1.3 sub-tasks with …...

This paper is to introduce our participation in the WMT13 shared tasks on Quality Estima- tion for machine translation without using ref- erence translations. We submitted the results for Task …...

We describe TerrorCat, a submission to this year’s metrics shared task. It is a ma- chine learning-based metric that is trained on manual ranking data from WMT shared tasks 2008–2012. …...

We introduce referential translation ma- chines (RTM) for quality estimation of translation outputs. RTMs are a computa- tional model for identifying the translation acts between any two data sets with …...

This paper describes a set of experi- ments on two sub-tasks of Quality Esti- mation of Machine Translation (MT) out- put. Sentence-level ranking of alternative MT outputs is done with …...

We describe the results of our submissions to the WMT13 Shared Task on Quality Estimation (subtasks 1.1 and 1.3). Our submissions use the framework of Gaus- sian Processes to investigate …...

This paper describes the TALP-UPC par- ticipation in the WMT’13 Shared Task on Quality Estimation (QE). Our partic- ipation is reduced to task 1.2 on System Selection. We used a …...

AbstractThis paper presents a new active learning technique for machine translation based on quality estimation of automatically translated sentences. It uses an error-driven strategy, i.e., it assumes that the more …...

The automatic estimation of machine translation (MT) output quality is a hard task in which the selection of the appro- priate algorithm and the most predictive features over reasonably sized …...

AbstractThe automatic estimation of machine translation (MT) output quality is an active research area due to its many potential applications (e.g. aiding human translation and post-editing, re-ranking MT hypotheses, MT …...

Predicting the quality of machine trans- lations is a challenging topic. Quality estimation (QE) of translations is based on features of the source and target texts (without the need for …...

This paper presents the use of consensus among Machine Translation (MT) systems for the WMT14 Quality Estimation shared task. Consensus is explored here by com- paring the MT system output …...

AbstractThis article describes a large-scale evaluation of the use of Statistical Machine Translation for professional subtitling. The work was carried out within the FP7 EU-funded project SUMAT and involved two …...

Machine Translation (MT) Quality Estimation (QE) aims to automatically measure the quality of MT system output without reference translations. In spite of the progress achieved in re- cent years, current …...

AbstractWe present a first attempt at predicting the quality of translations produced by human, professional translators. We examine datasets annotated for quality at sentence- and word-level for four language pairs …...

We address the problem of estimating the quality of Automatic Speech Recognition (ASR) out- put at utterance level, without recourse to manual reference transcriptions and when information about system’s confidence …...

AbstractThis paper presents a novel approach for parallel data generation using machine translation and quality estimation. Our study focuses on pivot-based machine translation from English to Croatian through Slovene. We …...

We investigate the usefulness of syntactic knowledge in estimating the quality of English-French translations. We find that dependency and constituency tree kernels perform well but the error rate can be …...

We describe our systems for the WMT14 Shared Task on Quality Estimation (sub- tasks 1.1, 1.2 and 1.3). Our submissions use the framework of Multi-task Gaus- sian Processes, where we …...

Quality estimation evaluation commonly takes the form of measurement of the error that exists between predictions and gold standard labels for a particular test set of translations. Issues can arise …...

Scarcity of annotated corpora for many languages is a bottleneck for training fine- grained sentiment analysis models that can tag aspects and subjective phrases. We pro- pose to exploit statistical …...

We describe our systems for Tasks 1 and 2 of the WMT15 Shared Task on Qual- ity Estimation. Our submissions use (i) a continuous space language model to ex- tract …...

This paper describes the submission of the UGENT-LT3 SCATE system to the WMT15 Shared Task on Quality Estima- tion (QE), viz. English-Spanish word and sentence-level QE. We conceived QE as …...

AbstractEffectively assessing Natural Language Processing output tasks is a challenge for research in the area. In the case of Machine Translation (MT), automatic metrics are usually preferred over human evaluation, …...

This paper presents a novel approach using recurrent neural networks for estimating the quality of machine translation output. A se- quence of vectors made by the prediction method is used …...

Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation en- vironments. Such scenarios can be im- proved if quality predictions are accompa- nied …...

AbstractHuman-targeted metrics provide a compromise between human evaluation of machine translation, where high inter-annotator agreement is difficult to achieve, and fully automatic metrics, such as BLEU or TER, that lack …...

AbstractEvaluating the quality of output from language processing systems such as machine translation or speech recognition is an essential step in ensuring that they are sufficient for practical use. However, …...

AbstractWe present Marmot~― a new toolkit for quality estimation (QE) of machine translation output. Marmot contains utilities targeted at quality estimation at the word and phrase level. However, due to …...

This paper describes the SHEF submis- sions for the three sub-tasks of the Qual- ity Estimation shared task of WMT17, namely: (i) a word-level prediction sys- tem using bilexical embeddings, …...

The main aim of this paper is to investigate automatic quality assessment for spoken language translation (SLT). More precisely, we investigate SLT errors that can be due to transcription (ASR) …...

We present a new model for text re- gression that seamlessly combine engi- neered features and character-level infor- mation through deep parallel convolution stacks, multi-layer perceptrons and multi- task learning. …...

Automatic post-editing (APE) is a challenging task on WMT evaluation campaign. We find that only a small number of edit operations are required for most machine translation outputs, through analysis …...

In this paper, we present a two-stage neu- ral quality estimation model that uses mul- tilevel task learning for translation quality estimation (QE) at the sentence, word, and phrase levels. …...

The field of Quality Estimation (QE) has the goal to provide automatic methods for the evaluation of Machine Translation (MT), that do not require reference transla- tions in their computation. …...

AbstractThis paper provides an evaluation of a wide range of advanced sentence-level Quality Estimation models, including Support Vector Regression, Ride Regression, Neural Networks, Gaussian Processes, Bayesian Neural Networks, Deep Kernel …...

We investigate different strategies for combining quality estimation (QE) and automatic post- editing (APE) to improve the output of machine translation (MT) systems. The joint contribu- tion of the two …...

AbstractPredicting Machine Translation (MT) quality can help in many practical tasks such as MT post-editing. The performance of Quality Estimation (QE) methods has drastically improved recently with the introduction of …...

AbstractWord alignments identify translational correspondences between words in a parallel sentence pair and is used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems, or to perform …...

AbstractPredicting the quality of machine translation has traditionally been addressed with language-specific models, under the assumption that the quality label distribution or linguistic features exhibit traits that are not shared …...

AbstractRecent advances in pre-trained multilingual language models lead to state-of-the-art results on the task of quality estimation (QE) for machine translation. A carefully engineered ensemble of such models won the …...

AbstractWe frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as …...

AbstractThis paper presents our submission to the WMT2020 Shared Task on Quality Estimation (QE). We participate in Task and Task 2 focusing on sentence-level prediction. We explore (a) a black-box …...

AbstractWith the advent of neural machine translation, there has been a marked shift towards leveraging and consuming the machine translation results. However, the gap between machine translation systems and human …...

AbstractThe Predictor-Estimator framework for quality estimation (QE) is commonly used for its strong performance. Where the predictor and estimator works on feature extraction and quality evaluation, respectively. However, training the …...

AbstractWe report the results of the WMT20 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the …...

AbstractWe propose approaches to Quality Estimation (QE) for Machine Translation that explore both text and visual modalities for Multimodal QE. We compare various multimodality integration and fusion strategies. For both …...

AbstractThis paper describes the NICT Kyoto submission for the WMT’20 Quality Estimation (QE) shared task. We participated in Task 2: Word and Sentence-level Post-editing Effort, which involved Wikipedia data and …...

AbstractThis paper describes our system of the sentence-level and word-level Quality Estimation Shared Task of WMT20. Our system is based on the QE Brain, and we simply enhance it by …...

AbstractThe quality estimation of artifacts generated by creators via crowdsourcing has great significance for the construction of a large-scale data resource. A common approach to this problem is to ask …...

AbstractApproaches to Quality Estimation (QE) of machine translation have shown promising results at predicting quality scores for translated sentences. However, QE models are often trained on noisy approximations of quality …...

AbstractQuality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT …...

AbstractTranslating text into a language unknown to the text’s author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic …...

AbstractSentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent …...

Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety …...

AbstractWe report the results of the WMT 2021 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at …...

AbstractThis paper presents our work in WMT 2021 Quality Estimation (QE) Shared Task. We participated in all of the three sub-tasks, including Sentence-Level Direct Assessment (DA) task, Word and Sentence-Level …...

AbstractCompared to fully manual translation, post-editing (PE) machine translation (MT) output can save time and reduce errors. Automatic word-level quality estimation (QE) aims to predict the correctness of words in …...

Quality Estimation (QE) is the task of automat- ically predicting Machine Translation quality in the absence of reference translations, mak- ing it applicable in real-time settings, such as translating online …...

AbstractWe propose a method to distill a language-agnostic meaning embedding from a multilingual sentence encoder. By removing language-specific information from the original embedding, we retrieve an embedding that fully represents …...

AbstractWord alignment identify translational correspondences between words in a parallel sentence pair and are used and for example and to train statistical machine translation and learn bilingual dictionaries or to …...

AbstractCurrent Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain …...

AbstractQuality Estimation, as a crucial step of quality control for machine translation, has been explored for years. The goal is to to investigate automatic methods for estimating the quality of …...

AbstractAutomatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality …...

AbstractQuality estimation aims to measure the quality of translated content without access to a reference translation. This is crucial for machine translation systems in real-world scenarios where high-quality translation is …...

AbstractQuality estimation (QE) of machine translation (MT) aims to evaluate the quality of machine-translated sentences without references and is important in practical applications of MT. Training QE models require massive …...

AbstractThis paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation …...

AbstractWith the increase in machine translation (MT) quality over the latest years, it has now become a common practice to integrate MT in the workflow of language service providers (LSPs) …...

AbstractQuality Estimation (QE) models have the potential to change how we evaluate and maybe even train machine translation models. However, these models still lack the robustness to achieve general adoption. …...

AbstractWe present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE). Our team participated in all three subtasks: (i) Sentence and Word-level …...

AbstractNeural machine translation (NMT) is often criticized for failures that happenwithout awareness. The lack of competency awareness makes NMT untrustworthy. This is in sharp contrast to human translators who give …...

AbstractQuality estimation (QE) is a crucial method to investigate automatic methods for estimating the quality of machine translation results without reference translations. This paper presents Huawei Translation Services Center’s (HW-TSC’s) …...

AbstractDeepSPIN is a research project funded by the European Research Council (ERC) whose goal is to develop new neural structured prediction methods, models, and algorithms for improving the quality, interpretability, …...

AbstractWe report the results of the WMT 2022 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems …...

AbstractThis paper describes IIT Bombay’s submission to the WMT22 Automatic Post-Editing (APE) shared task for the English-Marathi (En-Mr) language pair. We follow the curriculum training strategy to train our APE …...

AbstractThis paper presents KU X Upstage’s submission to the quality estimation (QE): critical error detection (CED) shared task in WMT22. We leverage the XLM-RoBERTa large model without utilizing any additional …...

AbstractFrom both human translators (HT) and machine translation (MT) researchers’ point of view, translation quality evaluation (TQE) is an essential task. Translation service providers (TSPs) have to deliver large volumes …...

AbstractWe present MLQE-PE, a new dataset for Machine Translation (MT) Quality Estimation (QE) and Automatic Post-Editing (APE). The dataset contains annotations for eleven language pairs, including both high- and low-resource …...

AbstractWe present the task of PreQuEL, Pre-(Quality-Estimation) Learning. A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation, thus eschewing unnecessary resource …...

AbstractThis study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data. QE models for GEC evaluations in prior …...

AbstractWith the recent advance in neural machine translation demonstrating its importance, research on quality estimation (QE) has been steadily progressing. QE aims to automatically predict the quality of machine translation …...

AbstractRecent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results in predicting the overall quality of translated sentences. However, detecting specifically which translated words are …...

AbstractQuality estimation (QE) is the task of predicting quality of outputs produced by machine translation (MT) systems. Currently, the highest-performing QE systems are supervised and require training on data with …...

AbstractThis case study presents a Multilingual e-commerce Project, which principal aim is to create an improved system that translates product titles and descriptions, plus other content in multiple languages. The …...

AbstractQuality Estimation (QE) of Machine Translation output suffers from the lack of annotated data to train supervised models across domains and language pairs. In this work, we describe a method …...

AbstractMultilingual sentence representations from large models encode semantic information from two or more languages and can be used for different cross-lingual information retrieval and matching tasks. In this paper, we …...

AbstractNeural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. However, NMT models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user …...

AbstractQuality Estimation (QE) is the task of predicting the quality of Machine Translation (MT) system output, without using any gold-standard translation references. State-of-the-art QE models are supervised: they require human-labeled …...

AbstractMachine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. State-of-the-art QE systems based on pretrained language models have been achieving remarkable correlations with …...

AbstractWe investigate whether using semi-supervised learning (SSL) methods can be beneficial for the task of word-level Quality Estimation of Machine Translation in low resource conditions. We show that the Mean …...

AbstractWhile quality estimation (QE) can play an important role in the translation process, its effectiveness relies on the availability and quality of training data. For QE in particular, high-quality labeled …...

AbstractNeural sequence generation models are known to “hallucinate”, by producing outputs that are unrelated to the source text. These hallucinations are potentially harmful, yet it remains unclear in what conditions …...