Exploring Paracrawl for Document-level Neural Machine Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1304–1310 May 2-6, 2023 ©2023 Association for Computational Linguistics Exploring Paracrawl for Document-level Neural Machine Translation Yusser Al Ghussin1,2, Jingyi Zhang3 and Josef van Genabith1,2 1German Research Center for Artificial Intelligence (DFKI), Saarland Informatics Campus, Saarbrucken, Germany 2Department of Language Science and Technology, Saarland University, Germany 3Hasso-Plattner-Institut (HPI), Potsdam, Germany yusser.al_ghussin/Josef.Van_Genabith@dfki.de,Jingyi.Zhang@hpi.de Abstract Document-level neural machine translation (NMT) has outperformed sentence-level NMT on a number of datasets. However, document- level NMT is still not widely adopted in real- world translation systems mainly due to the lack of large-scale general-domain training data for document-level NMT. We examine the effectiveness of using Paracrawl for learn- ing document-level translation. Paracrawl is a large-scale parallel corpus crawled from the In- ternet and contains data from various domains. The official Paracrawl corpus was released as parallel sentences (extracted from parallel web- pages) and therefore previous works only used Paracrawl for learning sentence-level transla- tion. In this work, we extract parallel para- graphs from Paracrawl parallel webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel docu- ments for training document-level translation models. We show that document-level NMT models trained with only parallel paragraphs from Paracrawl can be used to translate real documents from TED, News and Europarl, out- performing sentence-level NMT models. We also perform a targeted pronoun evaluation and show that document-level models trained with Paracrawl data can help context-aware pronoun translation. We release our data and code here1. 1 Introduction The Transformer translation model (Vaswani et al., 2017), which performs sentence-level translation based on attention networks, has achieved great success and significantly improved the state-of-the- art in machine translation. Compared to sentence- level translation, document-level translation (Xu et al., 2021; Bao et al., 2021; Jauregi Unanue et al., 2020; Ma et al., 2020; Maruf et al., 2019; Tu et al., 2018; Maruf and Haffari, 2018) performs trans- lation at document-level and can potentially fur- 1https://github.com/Yusser96/Exploring-Paracrawl-for- Document-level-Neural-Machine-Translation ther improve translation quality, e.g., document- level context can help word disambiguation for translating words with multiple senses, document- level translation can help pronoun translation which requires context outside of the current sentence (Müller et al., 2018), document-level translation can improve document-level lexical cohesion in the translation (Voita et al., 2019). Document-level neural machine translation (NMT) has received much attention in recent years (Bao et al., 2021; Donato et al., 2021; Fernandes et al., 2021; Kang et al., 2020; Saunders et al., 2020; Yu et al., 2020; Zheng et al., 2020; Yang et al., 2019; Kuang et al., 2018; Bawden et al., 2018; Zhang et al., 2018; Voita et al., 2018; Kuang and Xiong, 2018). Existing works showed that document-level translation can outperform sentence-level transla- tion for a number of datasets, such as TED, News, Europarl (Bao et al., 2021; Donato et al., 2021; Xu et al., 2021). Although document-level NMT has shown promising results on a number of bench- marks, document-level NMT is still not widely adopted …

Content truncated to 500 words