Cheating to Identify Hard Problems for Neural Machine Translation

Findings of the Association for Computational Linguistics: EACL 2023, pages 1620–1631 May 2-6, 2023 ©2023 Association for Computational Linguistics Cheating to Identify Hard Problems for Neural Machine Translation Proyag Pal and Kenneth Heafield School of Informatics, University of Edinburgh, Scotland {proyag.pal,kheafiel}@ed.ac.uk Abstract We identify hard problems for neural machine translation models by analyzing progressively higher-scoring translations generated by letting models cheat to various degrees. If a system cheats and still gets something wrong, that sug- gests it is a hard problem. We experiment with two forms of cheating: providing the model a compressed representation of the target as an additional input, and fine-tuning on the test set. Contrary to popular belief, we find that the most frequent tokens are not necessarily the most accurately translated due to these often being function words and punctuation that can be used more flexibly in translation, or content words which can easily be paraphrased. We systematically analyze system outputs to iden- tify categories of tokens which are particularly hard for the model to translate, and find that this includes certain types of named entities, sub- ordinating conjunctions, and unknown and for- eign words. We also encounter a phenomenon where words, often names, which were not in- frequent in the training data are still repeatedly mistranslated by the models — we dub this the Fleetwood Mac problem. 1 Introduction Some types and components of text are more dif- ficult to translate than others. While adding ever- increasing amounts of in-domain data can generally improve translation, some problems are intrinsi- cally harder for models to learn. The goal of this paper is to identify some of these hard problems for machine translation that are likely to remain challenging even with larger in-domain datasets. The way we approach this is to cheat. Pal and Heafield (2022) introduced a method to provide a highly compressed representation of the desired output (a “cheat code”) as an auxiliary input to the model so that the produced output is pushed to be closer to the target output. While their work was motivated as a method to estimate the amount of information present in the target that is missing in the source, we adopt the same method to produce unrealistically accurate models, and contend that if the models get particular things wrong even with hints from cheat codes, those are the harder things to translate. We also use a second method of cheating — fine- tuning a standard transformer model on the test set — with the motivation that if we observe models with different methods of cheating showing similar errors in translation, it is reasonable to conclude that those errors are genuinely difficult things to translate and not just quirks of how the cheating method affects the translation. While large amounts of in-domain data can improve overall quality sig- nificantly (Edunov et al., 2018), this fine-tuning method lets us expose the model to the most rele- vant data possible, the test set itself. The longer we fine-tune, the more it learns …

Content truncated to 500 words