Cheat Codes to Quantify Missing Source Information in Neural Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2472 - 2477 July 10-15, 2022 ©2022 Association for Computational Linguistics Cheat Codes to Quantify Missing Source Information in Neural Machine Translation Proyag Pal and Kenneth Heafield School of Informatics, University of Edinburgh, Scotland {proyag.pal,kheafiel}@ed.ac.uk Abstract This paper describes a method to quantify the amount of information H(t|s) added by the tar- get sentence t that is not present in the source s in a neural machine translation system. We do this by providing the model the target sen- tence in a highly compressed form (a “cheat code”), and exploring the effect of the size of the cheat code. We find that the model is able to capture extra information from just a single float representation of the target and nearly re- produces the target with two 32-bit floats per target token. 1 Introduction Given a sentence s in the source language, a ma- chine translation system generates a translation t in the target language. However, for any sentence of non-trivial complexity, the translation t is not unique. Therefore, to reproduce a reference trans- lation, a model requires some amount of extra in- formation. The aim of this work is to quantify the amount of information that is missing in the source s that is required to generate the translation t. To quantify this information, we modify the model architecture to provide the target sentence to the model as an auxiliary input, and observe the effect of varying the size of the representation of the target sentence from the minimum that provides any useful information to the decoder to the size that enables a near-perfect reproduction of the tar- get. Since the model seeing the target as an input is a form of “cheating”, we refer to these compressed representations of the target as “cheat codes”. 2 Related Work Zoph and Knight (2016) use multiple encoders to provide input in multiple languages to machine translation models to improve translation quality. Dual encoder networks have been used in lan- guage generation tasks to inject extra information (Sharath T et al., 2017), encode input at different levels of granularity (Yao et al., 2020), or for con- text awareness (Li et al., 2020). Junczys-Dowmunt and Grundkiewicz (2017) use very similar dual- encoder architectures for automatic post-editing, but without bottlenecking the second encoder out- put, and the second input in that case is machine translation output instead of a reference. Dinu et al. (2019) train models to inject custom terminology by providing an additional input, but instead of using a second encoder, this is done using inline annotations for the terms to be generated and using factors to demarcate these annotations. 3 Method 3.1 Architecture We use the Marian framework (Junczys-Dowmunt et al., 2018) to implement1 a modified dual-encoder transformer architecture (Zoph and Knight, 2016) similar to the one used by Junczys-Dowmunt and Grundkiewicz (2018), but without the tied encoder parameters. The first encoder …

Content truncated to 500 words