A Large Parallel Corpus of Full-Text Scientific Articles

A Large Parallel Corpus of Full-Text Scientific Articles Felipe Soares, Viviane Pereira Moreira, Karin Becker Institute of Informatics, Universidade Federal do Rio Grande do Sul Porto Alegre - RS, Brazil {felipe.soares, viviane, karin.becker}@inf.ufrgs.br Abstract The Scielo database is an important source of scientific information in Latin America, containing articles from several research domains. A striking characteristic of Scielo is that many of its full-text contents are presented in more than one language, thus being a potential source of parallel corpora. In this article, we present the development of a parallel corpus from Scielo in three languages: English, Portuguese, and Spanish. Sentences were automatically aligned using the Hunalign algorithm for all language pairs, and for a subset of trilingual articles also. We demonstrate the capabilities of our corpus by training a Statistical Machine Translation system (Moses) for each language pair, which outperformed related works on scientific articles. Sentence alignment was also manually evaluated, presenting an average of 98.8% correctly aligned sentences across all languages. Our parallel corpus is freely available in the TMX format, with complementary information regarding article metadata. Keywords: parallel corpus, scientific articles, Scielo 1. Introduction Cross-language corpora is one of the basis of Statistical Machine Translation (SMT) systems. The acquisition of quality corpora is not a trivial task, as it may demand con- siderable use of expert human curating, especially for paral- lel corpora. In that sense, the automated building of parallel corpora from open resources is of great interest in Natu- ral Language Processing (NLP). Europarl (Koehn, 2005), is one of the largest parallel corpora availabe, including up to 21 European languages. Similarly, the United Na- tions (UN) parallel corpus (Ziemski et al., 2016) makes use of UN’s official documents and records, providing aligned sentenced in six languages. Other parallel corpora initia- tives have been reported in distinct domains, such as patents (Utiyama and Isahara, 2007), movie subtitles (Zhang et al., 2014), and books (Skadins et al., 2014). Parallel corpora based on scientific articles can be a valu- able language resource. Several text mining tasks may ben- efit from the availability of parallel corpora of scientific ar- ticles. The most straightforward example is cross language plagiarism detection, when an original text is translated in another language and presented as novel. Other possible applications are related to article indexing or classification, as well as the development and extension of Named Entity Recognition (NER) tools for multiple languages. The latter is of particular interest to the increasingly active biomedical sciences field, which has already standardized vocabularies and ontologies, thus favoring NER initiatives. The construction of parallel corpora of scientific texts has been addressed by different authors. Wu et al. (2011) con- structed a parallel corpus of biomedical article titles from PUBMED in six languages. Based on this corpus, the au- thors built SMT systems and achieved higher BLEU scores than Google Translator. With a different intent, Kors et al. (2015) produced a gold-standard annotated parallel corpus for biomedical concept recognition in five languages. They used Medline abstracts, drug labels, …

Content truncated to 500 words