The United Nations Parallel Corpus v1.0

The United Nations Parallel Corpus v1.0 Michał Ziemski⋆, Marcin Junczys-Dowmunt†‡, Bruno Pouliquen‡ ∗United Nations, DGACM, New York, United States of America †Adam Mickiewicz University, Pozna´n, Poland ‡World Intellectual Property Organization, Geneva, Switzerland mziemski@unog.ch,junczys@amu.edu.pl,bruno.pouliquen@wipo.int Abstract This paper describes the creation process and statistics of the official United Nations Parallel Corpus, the first parallel corpus composed from United Nations documents published by the original data creator. The parallel corpus presented consists of manually translated UN documents from the last 25 years (1990 to 2014) for the six official UN languages, Arabic, Chinese, English, French, Russian, and Spanish. The corpus is freely available for download under a liberal license. Apart from the pairwise aligned documents, a fully aligned subcorpus for the six official UN languages is distributed. We provide baseline BLEU scores of our Moses-based SMT systems trained with the full data of language pairs involving English and for all possible translation directions of the six-way subcorpus. Keywords: United Nations, parallel corpus, statistical machine translation 1. Motivation The United Nations1 (UN) is mandated to publish docu- ments in six official languages and built up a considerable archive of parallel documents from its own translation oper- ations. Multilingualism is a strategic priority for the United Nations, as an essential factor in harmonious communica- tion among peoples. The official publication of this corpus is a reaction to the growing importance of statistical machine translation (SMT) within the UN Department for General Assembly and Conference Management (DGACM) translation ser- vices. In 2011, a research project — in cooperation with the World Intellectual Property Organization (WIPO) — to ex- plore a prototype SMT system based on the TAPTA system used at WIPO (Pouliquen et al., 2011) for the language pair English-Spanish (Pouliquen et al., 2012) was spearheaded by the Spanish Translation Service (STS) in New York and quickly noticed by other United Nations language services. Further development underlined the good performance of the SMT approach and its applicability to UN translation services. The system was expanded (Pouliquen et al., 2013) to a total of 10 language pairs, resulting in a production- grade cloud-based SMT service called TAPTA4UN. Espe- cially since its integration with the in-house computer as- sisted translation (CAT) tool eLUNa, TAPTA4UN has be- come a critical global tool in the UN translation toolkit. DGACM publishes documents in the six official UN lan- guages and additionally in German2 and is running ma- jor translation operations in various locations. The global DGACM translation output for 2014 alone was 231 mil- lion words. The translated documents are hosted on the Official Document System3 (ODS) and are publicly avail- able. Historically, this parallel data has been a major re- 1Referring to the Department for General Assembly and Con- ference Management which is responsible for the document pro- cessing chain, including translation, of the UN Secretariat 2Only some documents are translated by the German Transla- tion Section in New York. 3http://ods.un.org source for SMT and NLP research, and has resulted in var- ious (unofficial) corpora, most of them incomplete due …

Content truncated to 500 words