OpusTools and Parallel Corpus Diagnostics

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3782–3789 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3782 OpusTools and Parallel Corpus Diagnostics Mikko Aulamo, Umut Sulubacak, Sami Virpioja, J¨org Tiedemann Department of Digital Humanities University of Helsinki, Helsinki / Finland {mikko.aulamo, umut.sulubacak, sami.virpioja, jorg.tiedemann}@helsinki.fi Abstract This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consistency of the data collection. Keywords: Corpus (Creation, Annotation, etc.); Machine Translation; Tools, Systems, Applications 1. Introduction OPUS (Tiedemann, 2012) is the biggest collection of openly available parallel corpora. The collection has been growing constantly over the years and is widely used in work on machine translation and cross-linguistic research. Currently it contains 57 released corpora covering over 700 languages and language variants creating more than 70,000 bitexts in the sense of aligned language pairs across all cor- pora in the collection. The size and popularity of OPUS makes it necessary to build an efficient infrastructure that enables the various users to obtain and access the data and this paper introduces two packages that provide tools for that purpose. The goal of those packages is it to make it easy to download, convert and process the data included in OPUS from the command line or from applications using the library implementing those tools. The two packages re- fer to a Python library with command-line tools and a com- plementary Perl module, both provided as open source and with permissive licenses. In the sections below, we introduce the tools and their basic use and also discuss how we applied those tools to create new data sets and to run systematic diagnostics of the entire data base. With the availability of the OpusTools it is now possible to run careful sanity checks on the extensive data sets to verify validity of encoding, to find broken links and structures and to identify other issues with the data. 2. Characteristics of OPUS OPUS includes parallel corpora from a wide variety of sources. Each of them comes with their own peculiari- ties and the properties can differ substantially depending on the original data and their distribution. The philosophy in OPUS is to keep markup and annotation as much as pos- sible but to unify the essential data format to make access to parallel data as transparent as possible. This means that corpus data is …

Content truncated to 500 words