On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2214 - 2225 July 10-15, 2022 ©2022 Association for Computational Linguistics On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation Kelly Marchisio∗ Johns Hopkins University kmarc@jhu.edu Markus Freitag David Grangier Google Research {freitag, grangier}@google.com Abstract Modern unsupervised machine translation (MT) systems reach reasonable translation qual- ity under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU. We compare translations from super- vised and unsupervised MT systems of simi- lar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is super- vised MT. We then demonstrate a way to com- bine the benefits of both methods into a sin- gle system which results in improved adequacy and fluency as rated by human evaluators. Our results open the door to interesting discussions about how supervised and unsupervised MT might be different yet mutually-beneficial. 1 Introduction Supervised machine translation (MT) utilizes paral- lel bitext to learn to translate. Ideally, this data con- sists of natural texts and their human translations. In a way, the goal of supervised MT training is to produce a machine that mimicks human translators in their craft. Unsupervised MT, on the other hand, uses monolingual data alone to learn to translate. Critically, unsupervised MT never sees an exam- ple of human translation, and therefore must create its own style of translation. Unlike supervised MT where one side of each training sentence pair must be a translation, unsupervised MT can be trained with natural text alone. In this work, we investigate the output of su- pervised and unsupervised MT systems of similar quality to assess whether systematic differences in translation exist. Our exploration of this research area focuses on English→German for which abun- dant bilingual training examples exist, allowing us ∗ Work completed at Google Translate Research. to train high-quality systems with both supervised and unsupervised training. Our main contributions are: • We observe systematic differences between the output of supervised and unsupervised MT systems of similar quality. High-quality un- supervised output appears more natural, and more structurally diverse when compared to human translation. • We show a way to incorporate unsupervised back-translation into a standard supervised MT system, improving adequacy, naturalness, and fluency as measured by human evaluation. Our results provoke interesting questions about what unsupervised methods might contribute be- yond the traditional context of low-resource lan- guages which lack bilingual training data, and sug- gest that unsupervised MT might have contribu- tions to make for high-resource scenarios as well. It is worth exploring how combining supervised and unsupervised setups might contribute to a sys- tem better than either creates alone. We discuss related work in §2. In §3, we in- …

Content truncated to 500 words