Human Judgment
Related Papers
Sentence-level MT evaluation without reference translations: beyond language modeling
. In this paper we investigate the possibility of evaluating MT quality and fluency at the sentence level in the absence of reference translations. We measure the correlation between automatically-generated …...
Generating Chinese Couplets using a Statistical MT Approach
Part of the unique cultural heritage of China is the game of Chinese couplets (duìlián). One person challenges the oth- er person with a sentence (first sentence). The other person …...
From Human to Automatic Error Classification for Machine Translation Output
Future improvement of machine transla- tion systems requires reliable automatic evaluation and error classification mea- sures to avoid time and money consuming human classification. In this article, we propose a …...
Dependency Analysis of Scrambled References for Better Evaluation of Japanese Translation
In English-to-Japanese translation, BLEU (Papineni et al., 2002), the de facto standard evaluation metric for machine translation (MT), has very weak corre- lation with human judgments (Goto et al., 2011; …...
Evaluating Textual Representations through Image Generation
AbstractWe present a methodology for determining the quality of textual representations through the ability to generate images from them. Continuous representations of textual input are ubiquitous in modern Natural Language …...
Facet-Aware Evaluation for Extractive Summarization
AbstractCommonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage …...
Invisible to People but not to Machines: Evaluation of Style-aware HeadlineGeneration in Absence of Reliable Human Judgment
AbstractWe automatically generate headlines that are expected to comply with the specific styles of two different Italian newspapers. Through a data alignment strategy and different training/testing settings, we aim at …...
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
AbstractWe study knowledge-grounded dialogue generation with pre-trained language models. To leverage the redundant external knowledge under capacity constraint, we propose equipping response generation defined by a pre-trained language model with …...
Learning and Evaluating Emotion Lexicons for 91 Languages
AbstractEmotion lexicons describe the affective meaning of words and thus constitute a centerpiece for advanced sentiment and emotion analysis. Yet, manually curated lexicons are only available for a handful of …...
Topic-relevant Response Generation using Optimal Transport for an Open-domain Dialog System
AbstractConventional neural generative models tend to generate safe and generic responses which have little connection with previous utterances semantically and would disengage users in a dialog system. To generate relevant …...
Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model
AbstractEvaluating the quality of responses generated by open-domain conversation systems is a challenging task. This is partly because there can be multiple appropriate responses to a given dialogue history. Reference-based …...