METEOR

Recently, confusion network decoding has been applied in machine translation system combination. Due to errors in the hypoth- esis alignment, decoding may result in un- grammatical combination outputs. This pa- …...

Meteor is an automatic metric for Ma- chine Translation evaluation which has been demonstrated to have high levels of corre- lation with human judgments of translation quality, significantly outperforming the …...

We present a series of empirical studies aimed at illuminating more precisely the likely contribution of semantic roles in im- proving statistical machine translation ac- curacy. The experiments reported study …...

. The strict character of most of the existing Machine Translation (MT) evaluation metrics does not permit them to capture lexical variation in translation. However, a central issue in MT …...

We describe the LIU systems for English- German and German-English translation in the WMT09 shared task. We focus on two methods to improve the word align- ment: (i) by applying …...

We describe a synthetic method for com- bining machine translations produced by different systems given the same input. One-best outputs are explicitly aligned to remove duplicate words. Hypotheses follow system …...

We explored novel automatic evaluation measures for machine translation output oriented to the syntactic structure of the sentence: the BLEU score on the detailed Part-of-Speech (POS) tags as well as …...

We present TESLA-M and TESLA, two novel automatic machine translation eval- uation metrics with state-of-the-art perfor- mances. TESLA-M builds on the suc- cess of METEOR and MaxSim, but em- ploys …...

Translation systems are generally trained to optimize BLEU, but many alternative metrics are available. We explore how optimizing toward various automatic evaluation metrics (BLEU, METEOR, NIST, TER) affects the re- …...

Automated grammar correction techniques have seen improvement over the years, but there is still much room for increased perfor- mance. Current correction techniques mainly focus on identifying and correcting a …...

This paper describes Meteor 1.3, our submis- sion to the 2011 EMNLP Workshop on Sta- tistical Machine Translation automatic evalua- tion metric tasks. New metric features include improved text normalization, …...

A machine learning mechanism is learned from human annotations in order to perform preference ranking. The mechanism operates on a sentence level and ranks the alternative machine translations of each …...

We introduce the first fully automatic, fully seman- tic frame based MT evaluation metric, MEANT, that outperforms all other commonly used auto- matic metrics in correlating with human judgment on …...

This paper describes the UPC participation in the WMT 12 evaluation campaign. All sys- tems presented are based on standard phrase- based Moses systems. Variations adopted sev- eral improvement techniques …...

AbstractIn the last decades, a wide range of automatic metrics that use linguistic knowledge has been developed. Some of them are based on lexical information, such as METEOR; others rely …...

AbstractDefining the reordering search space is a crucial issue in phrase-based SMT between distant languages. In fact, the optimal trade-off between accuracy and complexity of decoding is nowadays reached by …...

This paper describes the TALP participa- tion in the WMT13 evaluation campaign. Our participation is based on the combi- nation of several statistical machine trans- lation systems: based on standard …...

We present a human judgments dataset and an adapted metric for evaluation of Arabic machine translation. Our medium- scale dataset is the first of its kind for Ara- bic with …...

This paper describes the LAYERED met- ric which is used for the shared WMT’14 metrics task. Various metrics exist for MT evaluation: BLEU (Papineni, 2002), ME- TEOR (Alon Lavie, 2007), …...

This paper describes Parmesan, our sub- mission to the 2014 Workshop on Sta- tistical Machine Translation (WMT) met- rics task for evaluation English-to-Czech translation. We show that the Czech Me- …...

Most of the widely-used automatic evaluation metrics consider only the local fragments of the references and translations, and they ignore the evaluation on the syntax level. Current syntax- based evaluation …...

This paper presents the system SSMT measuring the semantic similarity between a paragraph and a sentence submitted to the SemEval 2014 task3: Cross-level Se- mantic Similarity. The special difficulty of …...

Evaluation of segment-level machine transla- tion metrics is currently hampered by: (1) low inter-annotator agreement levels in human as- sessments; (2) lack of an effective mechanism for evaluation of translations …...

This paper describes Meteor-WSD and RATATOUILLE, the LIMSI submissions to the WMT15 metrics shared task. Meteor- WSD extends synonym mapping to lan- guages other than English based on align- ments …...

Human-designed sub-structures are re- quired by most of the syntax-based ma- chine translation evaluation metrics. In this paper, we propose a novel evalua- tion metric based on dependency parsing model, …...

Compared to tree grammars, graph gram- mars have stronger generative capacity over structures. Based on an edge re- placement grammar, in this paper we pro- pose to use a synchronous …...

This paper proposes an extension of METEOR, a well-known MT evaluation metric, for multiple target languages using an in-house lexical resource called DBnary (an extraction from Wiktionary provided to the …...

In this paper, we describe our submis- sion to WMT 2015 Tuning Task. We integrate a dependency-based MT evalu- ation metric, RED, to Moses and com- pare it with BLEU …...

This paper describes the importance of in- troducing a phrase-based language model in the process of machine translation. In fact, nowadays SMT are based on phrases for translation but their …...

In this paper we describe the JUNITMZ 1 sys- tem that was developed for participation in Se- mEval 2016 Task 1: Semantic Textual Simi- larity. Methods for measuring the textual …...

AbstractIn this paper, we propose a novel neural approach for paraphrase generation. Conventional paraphrase generation methods either leverage hand-written rules and thesauri-based alignments, or use statistical machine learning principles. To …...

We report experiments with multi-modal neural machine translation models that in- corporate global visual features in differ- ent parts of the encoder and decoder, and use the VGG19 network to …...

This paper describes the monomodal and multimodal Neural Machine Translation systems developed by LIUM and CVC for WMT17 Shared Task on Multimodal Translation. We mainly explored two mul- timodal architectures …...

AbstractWe present a video captioning approach that encodes features by progressively completing syntactic structure (LSTM-CSS). To construct basic syntactic structure (i.e., subject, predicate, and object), we use a Conditional Random …...

AbstractGenerating high-quality paraphrases is a fundamental yet challenging natural language processing task. Despite the effectiveness of previous work based on generative models, there remain problems with exposure bias in recurrent …...

AbstractInformation need of humans is essentially multimodal in nature, enabling maximum exploitation of situated context. We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain. …...

AbstractMost of the existing models for document-level machine translation adopt dual-encoder structures. The representation of the source sentences and the document-level contexts are modeled with two separate encoders. Although these …...

Constrained decoding forces a certain set words or phrases to appear in the translation results and is very useful when adapting MT to a certain domain. In recent years, the …...

AbstractVirtual Assistants can be quite literal at times. If the user says “tell Bob I love him,” most virtual assistants will extract the message “I love him” and send it …...

AbstractPre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often “rambling” without coherently arranged content. In this work, we present a novel content-controlled …...

AbstractQuestion generation (QG) attempts to solve the inverse of question answering (QA) problem by generating a natural language question given a document and an answer. While sequence to sequence neural …...

AbstractThis paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions …...

AbstractThe metrics standardly used to evaluate Natural Language Generation (NLG) models, such as BLEU or METEOR, fail to provide information on which linguistic factors impact performance. Focusing on Surface Realization …...

AbstractThis work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based …...

AbstractMulti-document summarization (MDS) has made significant progress in recent years, in part facilitated by the availability of new, dedicated datasets and capacious language models. However, a standing limitation of these …...

AbstractThe automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as …...

AbstractHumans constantly deal with multimodal information, that is, data from different modalities, such as texts and images. In order for machines to process information similarly to humans, they must be …...

AbstractWe propose uFACT (Un-Faithful Alien Corpora Training), a training corpus construction method for data-to-text (d2t) generation models. We show that d2t models trained on uFACT datasets generate utterances which represent …...