Человеческое суждение

In evaluating the output of language tech- nology applications—MT, natural language generation, summarisation—automatic eval- uation techniques generally conflate mea- surement of faithfulness to source content with fluency of the resulting …...

We describe DCU’s LFG dependency- based metric submitted to the shared eval- uation task of WMT-MetricsMATR 2010. The metric is built on the LFG F-structure- based approach presented in (Owczarzak …...

Learner corpora are receiving special atten- tion as an invaluable source of educational feedback and are expected to improve teach- ing materials and methodology. However, they include various types of …...

AbstractA dialogue is essentially a multi-turn interaction among interlocutors. Effective evaluation metrics should reflect the dynamics of such interaction. Existing automatic metrics are focused very much on the turn-level quality, …...

AbstractIn translating text where sentiment is the main message, human translators give particular attention to sentiment-carrying words. The reason is that an incorrect translation of such words would miss the …...

AbstractAutomated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we …...

AbstractIn natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater …...