Two Multilingual Corpora Extracted from the Tenders Electronic Daily for Machine Learning and Machine Translation Applications.

Two Multilingual Corpora Extracted from the Tenders Electronic Daily for Machine Learning and Machine Translation Applications Oussama Ahmia1,2, Nicolas Béchet1, Pierre-François Marteau1 1IRISA, Université Bretagne Sud Campus de Tohannic, 56000 Vannes FRANCE 2 Jurismarchés, 2 Place Saint-Pierre, Nantes, 44000, FRANCE {firstname.name}@irisa.fr Abstract The European "Tenders Electronic Daily" (TED) is a large source of semi-structured and multilingual data that is very valuable to the Natural Language Processing community. This data sets can effectively be used to address complex machine translation, multilingual terminology extraction, text-mining, or to benchmark information retrieval systems. Despite of the services offered by the user-friendliness of the web site that is made available to the public to access the publishing of the EU call for tenders, collecting and managing such kind of data is a great burden and consumes a lot of time and computing resources. This could explain why such a resource is not very (if any) exploited today by computer scientists or engineers in NLP. The aim of this paper is to describe two documented and easy-to-use multilingual corpora (one of them is a parallel corpus), extracted from the TED web source that we will release for the benefit of the NLP community. Keywords: Multilingual corpora, Parallel Corpus, Call for Tender, European Languages, Natural Language Resource 1. Introduction The world procurement generates daily large amounts of data, that represent useful knowledge for business intelli- gence tasks. Among other sources, the European "Tenders Electronic Daily" (TED) system, publishes approximately 1,700 tenders five times a week in the TED servers1. More precisely, the publication office of the TED publishes about 460,000 contract notices per year, among which 175,000 tenders worth around 420 billion euros. These raw data are available as bulk downloads of these data that contain contract notices for tender data in XML format with a complex structure. Unfortunately, different versions of the XML data structure from year to year have been used, making the aggregation of the different bulks of data difficult. Furthermore, the collected documents are associated to a variable number of translations as well as variable sets of meta data that is used for indexing. Consequently, the collecting and managing of such data is a great burden and consumes a lot of time and computing resources. The aim of this paper is to describe a processed version of this database in a raw text format that can be directly and easily used for text mining and natural language processing tasks. We also aim at making this processed dataset available to the scientific community and can be downloaded2 along with a simple Python API for easier manipulation. The provided dataset is declined into two sub-datasets cre- ated from the TED’s documents that have been published between January 2011 and August 2017. 1http://ted.europa.eu/TED/main/HomePage.do 2https://github.com/oussamaahmia/TED-dataset 1. The first sub-dataset, fd-TED, is a (multilingual) cor- pus or aligned translated documents. It contains around 3 millions of documents translated to 24 lan- guages (DA, DE, EN, ES, FI, FR, EL, IT, NL, PT, SV, CS, ET, HU, LT, LV, …

Content truncated to 500 words