Adversarial Training on Disentangling Meaning and Language Representations for Unsupervised Quality Estimation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5240–5245 October 12–17, 2022. 5240 Adversarial Training on Disentangling Meaning and Language Representations for Unsupervised Quality Estimation Yuto Kuroda Ehime University kuroda@ai.cs.ehime-u.ac.jp Tomoyuki Kajiwara Ehime University kajiwara@cs.ehime-u.ac.jp Yuki Arase Osaka University arase@ist.osaka-u.ac.jp Takashi Ninomiya Ehime University ninomiya@cs.ehime-u.ac.jp Abstract We propose a method to distill language- agnostic meaning embeddings from multilin- gual sentence encoders for unsupervised qual- ity estimation of machine translation. Our method facilitates that the meaning embed- dings focus on semantics by adversarial train- ing that attempts to eliminate language-specific information. Experimental results on unsuper- vised quality estimation reveal that our method achieved higher correlations with human evalu- ations. 1 Introduction Quality Estimation (QE) is a task of estimat- ing translation quality without reference sen- tences (Specia et al., 2018). Reference-based au- tomatic evaluation methods, such as BLEU (Pap- ineni et al., 2002) and BLEURT (Sellam and Parikh, 2020), have contributed to research and develop- ment of machine translation; however, end-users of machine translation systems unlikely have such ref- erence translations. Hence, the development of QE methods that correlate well with human evaluation is practically important. Supervised QE models (Ranasinghe et al., 2020; Fomicheva et al., 2020a; Nakamachi et al., 2020) based on pre-trained multilingual sentence en- coders (Conneau et al., 2020; Feng et al., 2022) have been actively proposed in the QE competi- tions (Specia et al., 2020). However, these mod- els require bilingual sentence pairs with manually labeled translation quality scores for fine-tuning. Creating such a QE dataset is expensive because it requires annotators who are fluent in both of source and target languages. Therefore, supervised QE models are limited to several major language pairs included in the competitions. In contrast, unsupervised QE allows quality es- timation without human-assessed machine trans- lation outputs. Instead of the annotated outputs, unsupervised QE utilizes widely available parallel corpora. Multilingual sentence encoders (Artetxe and Schwenk, 2019a,b; Reimers and Gurevych, 2020; Conneau et al., 2020; Feng et al., 2022) are promising for developing unsupervised QE models; however, their sentence embeddings are dominated by language-specific information. Due to this char- acteristic, these sentence embeddings form clusters by language rather than by meaning, which hinders precise estimation of semantic similarity across languages (Tiyajamorn et al., 2021). To address this problem, DREAM (Tiyajamorn et al., 2021) disentangles sentence embeddings to meaning and language embeddings. It conducts self-supervised learning using parallel sentence pairs in bilingual corpora as positive examples and random pairs as negative examples; meaning embeddings of posi- tive pairs should be close while those of negative pairs should be distant. However, DREAM lacks direct supervision to eliminate language-specific information from the meaning embeddings and its architecture is complex. We improve DREAM by introducing an adver- sarial training that attempts to remove language- specific information from the meaning embed- dings.1 Our adversarial training eliminates the ran- dom pairs that DREAM needs, which results in a simpler architecture and lighter computational costs for training. Experimental results on the WMT20 QE task (Specia et al., 2020) revealed …

Content truncated to 500 words