Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2191–2212 July 9-14, 2023 ©2023 Association for Computational Linguistics Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation Josef Jon and Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {jon,bojar}@ufal.mff.cuni.cz Abstract We propose a genetic algorithm (GA) based method for modifying n-best lists produced by a machine translation (MT) system. Our method offers an innovative approach to im- proving MT quality and identifying weaknesses in evaluation metrics. Using common GA oper- ations (mutation and crossover) on a list of hy- potheses in combination with a fitness function (an arbitrary MT metric), we obtain novel and diverse outputs with high metric scores. With a combination of multiple MT metrics as the fitness function, the proposed method leads to an increase in translation quality as measured by other held-out automatic metrics. With a single metric (including popular ones such as COMET) as the fitness function, we find blind spots and flaws in the metric. This allows for an automated search for adversarial examples in an arbitrary metric, without prior assumptions on the form of such example. As a demon- stration of the method, we create datasets of adversarial examples and use them to show that reference-free COMET is substantially less ro- bust than the reference-based version. 1 Introduction Attaining good translation quality in machine trans- lation (MT) arguably relies on good automatic met- rics of MT quality. Recently, a new generation of evaluation metrics was introduced. These met- rics are based on embeddings computed by large pretrained language models and human annotation scores. The improvements in metric quality re- sulted in renewed interest in metric-driven transla- tion hypothesis selection methods, like Minimum Bayes Risk (MBR) decoding (Goel and Byrne, 2000; Kumar and Byrne, 2004). Our method relies on MBR decoding and the genetic algorithm (GA; Fraser, 1957; Bremermann, 1958; Holland, 1975. Through combinations and mutations of translations produced by an MT model, we search for optimal translation under a selected metric. This is a novel approach to gener- ating translation hypotheses in NMT. We find that by combining neural and surface form-based metrics in a GA’s fitness function, it is possible to create better quality translations than by simple reranking of the initial hypotheses (as evaluated by held-out metrics). It also allows the combination of multiple sources for the translation, for example, MT, paraphrasing models and dictio- naries. Another use-case for our method is the identifi- cation of weak points in MT metrics. Flaws and biases of the novel neural metrics are being studied, for example, by Hanna and Bojar (2021), Amrhein and Sennrich (2022a), Alves et al. (2022) or Kano- jia et al. (2021). In summary, these metrics have low sensitivity to errors in named entities and num- bers. Also, they are not sufficiently sensitive to changes in meaning and critical errors, like nega- tions. These previous works on deficiencies of …

Content truncated to 500 words