On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1656–1671 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1656 On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation Wei Zhao†, Goran Glavaˇs‡, Maxime PeyrardΦ, Yang Gao⋆, Robert WestΦ, Steffen Eger† † Technische Universit¨at Darmstadt ‡ University of Mannheim, Germany Φ EPFL, Switerland ⋆ Royal Holloway University of London, UK {zhao,eger}@aiphes.tu-darmstadt.de goran@informatik.uni-mannheim.de, yang.gao@rhul.ac.uk {maxime.peyrard,robert.west}@epfl.ch Abstract Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual simi- larity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system trans- lations, which represents a natural adversarial setup for multilingual encoders. Reference- free evaluation holds the promise of web-scale comparison of MT systems. We systemati- cally investigate a range of metrics based on state-of-the-art cross-lingual semantic repre- sentations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limita- tions, namely, (a) a semantic mismatch be- tween representations of mutual translations and, more prominently, (b) the inability to punish “translationese”, i.e., low-quality literal translations. We propose two partial remedies: (1) post-hoc re-alignment of the vector spaces and (2) coupling of semantic-similarity based metrics with target-side language modeling. In segment-level MT evaluation, our best metric surpasses reference-based BLEU by 5.7 cor- relation points. We make our MT evaluation code available.1 1 Introduction A standard evaluation setup for supervised machine learning (ML) tasks assumes an evaluation metric which compares a gold label to a classifier predic- tion. This setup assumes that the task has clearly defined and unambiguous labels and, in most cases, that an instance can be assigned few labels. These assumptions, however, do not hold for natural lan- guage generation (NLG) tasks like machine trans- 1https://github.com/AIPHES/ ACL20-Reference-Free-MT-Evaluation lation (MT) (Bahdanau et al., 2015; Johnson et al., 2017) and text summarization (Rush et al., 2015; Tan et al., 2017), where we do not predict a single discrete label but generate natural language text. Thus, the set of labels for NLG is neither clearly defined nor finite. Yet, the standard evaluation protocols for NLG still predominantly follow the described default paradigm: (1) evaluation datasets come with human-created reference texts and (2) evaluation metrics, e.g., BLEU (Papineni et al., 2002) or METEOR (Lavie and Agarwal, 2007) for MT and ROUGE (Lin and Hovy, 2003) for sum- marization, count the exact “label” (i.e., n-gram) matches between reference and system-generated text. In other words, established NLG evaluation compares semantically ambiguous labels from an unbounded set (i.e., natural language texts) via hard symbolic matching (i.e., string overlap). The first remedy is to replace the hard symbolic comparison of natural language “labels” with a soft comparison of texts’ meaning, using seman- tic vector space representations. Recently, a num- ber of MT evaluation methods appeared focusing on semantic comparison of reference and system translations (Shimanaka …

Content truncated to 500 words