Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems

Proceedings of Recent Advances in Natural Language Processing, pages 1217–1226 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_137 1217 Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems Hadeel Saadany RGCL University of Wolverhampton Wolverhampton, UK h.s.saadany@wlv.ac.uk Constantin Or˘asan Centre for Translation Studies University of Surrey Guildford, UK c.orasan@surrey.ac.uk Emad Mohamed RGCL University of Wolverhampton Wolverhampton, UK e.mohamed2@wlv.ac.uk Ashraf Tantavy School of Computer Science and Informatics De Montfort University Leicester, UK ashraf.tantavy@dmu.ac.uk Abstract In translating text where sentiment is the main message, human translators give particular at- tention to sentiment-carrying words. The rea- son is that an incorrect translation of such words would miss the fundamental aspect of the source text, i.e. the author’s sentiment. In the online world, MT systems are exten- sively used to translate User-Generated Con- tent (UGC) such as reviews, tweets, and so- cial media posts, where the main message is often the author’s positive or negative attitude towards the topic of the text. It is important in such scenarios to accurately measure how far an MT system can be a reliable real-life util- ity in transferring the correct affect message. This paper tackles an under-recognised prob- lem in the field of machine translation eval- uation which is judging to what extent auto- matic metrics concur with the gold standard of human evaluation for a correct translation of sentiment. We evaluate the efficacy of conven- tional quality metrics in spotting a mistransla- tion of sentiment, especially when it is the sole error in the MT output. We propose a numeri- cal ‘sentiment-closeness’ measure appropriate for assessing the accuracy of a translated af- fect message in UGC text by an MT system. We will show that incorporating this sentiment- aware measure can significantly enhance the correlation of some available quality metrics with the human judgement of an accurate trans- lation of sentiment. 1 Introduction Standard quality measures for assessing the per- formance of machine translation systems, such as BLEU (Papineni et al., 2002), are domain agnos- tic; they evaluate the translation accuracy regard- less of the semantic domain or linguistic peculiar- ities of the source text. Consequently, they give equal penalty weight to inaccurate translation of n-grams, which may lead to performance overes- timation (or underestimation). For example, the Arabic high-rated Goodreads book review show- ing the reviewers overall satisfaction of a novel: ‘��� ���� �������� ������� ������� �������� ��������� (‘The novel is great, its only flaw is the last part’) is mistrans- lated by the available online translation tool which wrongly outputs a negative sentiment (‘The novel is terrible, its only flaw is the last part’). Despite the distortion of the affect message, this translation receives an equally high score as a correct trans- lation (The story is great, its only flaw is the last part), but which uses story instead of novel. This is because BLEU mildly penalises the wrong transla- tion as it swaps only one uni-gram (‘great’) with its opposite (‘terrible’). Yet, the mistranslation of this particular uni-gram is a critical error as it is …

Content truncated to 500 words