Human Judgements
Related Papers
Disamibiguating and Interpreting Verb Definitions
To achieve our goal of building a compre- hensive lexical database out of various on-line resources, it is necessary to interpret and disambiguate the information found in these resources. In …...
Learning to Say It Well: Reranking Realizations by Predicted Synthesis Quality
This paper presents a method for adapting a language generator to the strengths and weaknesses of a synthetic voice, thereby improving the naturalness of synthetic speech in a spoken language …...
Learning Labelled Dependencies in Machine Translation Evaluation
Recently novel MT evaluation metrics have been presented which go beyond pure string matching, and which corre- late better than other existing metrics with human judgements. Other research in this …...
Generating Shifting Sentiment for a Conversational Agent
We investigate techniques for generating al- ternative output sentences with varying sen- timent, using (an approximation to) the Valentino method, based on SentiWordNet, of Guerini et al. We extend this …...
The Secret’s in the Word Order: Text-to-Text Generation for Linguistic Steganography
Linguistic steganography is a form of covert communication using natural language to conceal the existence of the hidden message, which is usually achieved by systematically making changes to a cover …...
Optimizing a Distributional Semantic Model for the Prediction of German Particle Verb Compositionality
AbstractIn the work presented here we assess the degree of compositionality of German Particle Verbs with a Distributional Semantics Model which only relies on word window information and has no …...
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
AbstractText classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for …...
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation
AbstractAutomatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system’s quality over another. The community choice of automatic metric guides research directions …...
Constructing Distributions of Variation in Referring Expression Type from Corpora for Model Evaluation
AbstractThe generation of referring expressions (REs) is a non-deterministic task. However, the algorithms for the generation of REs are standardly evaluated against corpora of written texts which include only one …...
MS-COMET: More and Better Human Judgements Improve Metric Performance
AbstractWe develop two new metrics that build on top of the COMET architecture. The main contribution is collecting a ten-times larger corpus of human judgements than COMET and investigating how …...
REUSE: REference-free UnSupervised Quality Estimation Metric
AbstractThis paper describes our submission to the WMT2022 shared metrics task. Our unsupervised metric estimates the translation quality at chunk-level and sentence-level. Source and target sentence chunks are retrieved by …...
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
AbstractThe predictions of question answering (QA) systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating …...
What Drives the Use of Metaphorical Language? Negative Insights from Abstractness, Affect, Discourse Coherence and Contextualized Word Representations
AbstractGiven a specific discourse, which discourse properties trigger the use of metaphorical language, rather than using literal alternatives? For example, what drives people to say grasp the meaning rather than …...