Manual Evaluation

. For applications of the natural language processing, such as spoken translation, it is the most important characteristic to accept various inputs. We analyze a speech input as a sequence …...

. Recent research has shown the importance of using source context information to disambiguate source phrases in phrase-based Sta- tistical Machine Translation. Although encouraging results have been obtained, those studies …...

This paper analyzes the translation qual- ity of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish. We report the translation quality of …...

We improve the quality of paraphrases ex- tracted from parallel corpora by requiring that phrases and their paraphrases be the same syn- tactic type. This is achieved by parsing the …...

This paper presents the results of the WMT09 shared tasks, which included a translation task, a system combination task, and an evaluation task. We con- ducted a large-scale manual evaluation …...

The Workshop on Statistical Machine Translation (WMT) has become one of ACL’s flagship workshops, held annually since 2006. In addition to soliciting pa- pers from the research community, WMT also …...

This paper presents the results of the WMT11 shared tasks, which included a translation task, a system combination task, and a task for machine translation evaluation metrics. We conducted a …...

This paper presents the results of the WMT12 shared tasks, which included a translation task, a task for machine translation evaluation metrics, and a task for run-time estimation of machine …...

AbstractWe describe the “Shared Task on Applying Machine Learning Techniques to Optimise the Division of Labour in Hybrid Machine Translation” (ML4HMT) which aims to foster research on improved system combination …...

In this paper we present a novel approach to automatic creation of anchor texts for hyper- links in a document pointing to similar doc- uments. Methods used in this approach …...

In this paper we present a method for extracting bilingual terminologies from comparable corpora. In our approach we treat bilingual term extrac- tion as a classification problem. For classification we …...

AbstractWe present a method for the extraction of synonyms for German particle verbs based on a word-aligned German-English parallel corpus: by translating the particle verb to a pivot, which is …...

AbstractThis paper gives a general review and detailed analysis of China Workshop on Machine Translation (CWMT) Evaluation. Compared with the past CWMT evaluation campaigns, CWMT2013 evaluation is characterized as follows: …...

AbstractEvaluating the quality of output from language processing systems such as machine translation or speech recognition is an essential step in ensuring that they are sufficient for practical use. However, …...

AbstractWe present PROTEST, a test suite for the evaluation of pronoun translation by MT systems. The test suite comprises 250 hand-selected pronoun tokens and an automatic evaluation method which compares …...

AbstractKeyphrase extraction is the task of finding phrases that represent the important content of a document. The main aim of keyphrase extraction is to propose textual units that represent the …...

AbstractThe paper presents a deep factored machine translation (MT) system between English and Bulgarian languages in both directions. The MT system is hybrid. It consists of three main steps: (1) …...

Recent works in spoken language translation (SLT) have attempted to build end-to-end speech-to-text translation without using source language transcription during learning or decoding. However, while large quantities of parallel texts …...

AbstractThis paper presents a case study in translating short image captions of the Visual Genome dataset from English into Hindi using out-of-domain data sets of varying size. We experiment with …...

AbstractConducting a manual evaluation is considered an essential part of summary evaluation methodology. Traditionally, the Pyramid protocol, which exhaustively compares system summaries to references, has been perceived as very reliable, …...

AbstractMost existing work on automated fact checking is concerned with predicting the veracity of claims based on metadata, social network spread, language used in claims, and, more recently, evidence supporting …...

AbstractIn this paper we present a pipeline developed at Acolad to test a Machine Translation (MT) engine and compute the discount to be applied when its output is used in …...

AbstractSplit and Rephrase is a text simplification task of rewriting a complex sentence into simpler ones. As a relatively new task, it is paramount to ensure the soundness of its …...

AbstractWe propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction (GEC). Previous studies have shown that reference-less metrics are promising; however, existing metrics are …...

AbstractThis paper presents TaPaCo, a freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. Tatoeba is a crowdsourcing project mainly geared towards language learners. Its aim is …...

AbstractFive years after the first published proofs of concept, direct approaches to speech translation (ST) are now competing with traditional cascade solutions. In light of this steady progress, can we …...

Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto- labeled data. However, its evaluation has long been a …...

AbstractELTE Poetry Corpus is a database that stores canonical Hungarian poetry with automatically generated annotations of the poems’ structural units, grammatical features and sound devices, i.e. rhyme patterns, rhyme pairs, …...

AbstractRecently, pre-trained transformer-based models have achieved great success in the task of definition generation (DG). However, previous encoder-decoder models lack effective representation learning to contain full semantic components of the …...

AbstractPowerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow …...

AbstractThis study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data. QE models for GEC evaluations in prior …...