CEF Data Marketplace: Powering a Long-term Supply of Language Data

CEF Data Marketplace: Powering a Long-term Supply of Language Data Amir Kamran†, Dace Dzeguze†, Jaap van der Meer†, Milica Panic†, Alessandro Cattelan‡, Daniele Patrioli‡, Luisa Bentivogli⋆, and Marco Turchi⋆ † TAUS - Language Data Network, Netherlands {amir,dace,jaap,milica}@taus.net ‡ Translated, Italy {alessandro,daniele.patrioli}@translated.com ⋆ FBK, Italy {bentivo,turchi}@fbk.eu Abstract We describe the CEF Data Marketplace project, which focuses on the develop- ment of a trading platform of translation data for language professionals: transla- tors, machine translation (MT) developers, language service providers (LSPs), trans- lation buyers and government bodies. The CEF Data Marketplace platform will be designed and built to manage and trade data for all languages and domains. This project will open a continuous and long- term supply of language data for MT and other machine learning applications. 1 Introduction The CEF Data Marketplace project is an initiative co-funded by the European Union under the Con- necting Europe Facility programme, under Grant Agreement INEA/CEF/ICT/A2018/1816453. The project has a duration of 24 months and started in November 2019. With over 3501 million new Internet users in 2019 and the annual digital growth of 9%, there is insufficient content available in the local lan- guages. The automated translation platforms sup- port merely about a hundred of the 4,000 lan- guages with an established writing system. The CEF Data Marketplace will be the first platform that facilitates the buying and selling of language data to help businesses and communities reach c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1https://wearesocial.com/blog/2019/ 01/digital-2019-global-internet-use- accelerates scale with their language technologies while offer- ing a way for the language data creators to mone- tize their work. 2 Platform Description The platform focuses on the integration and main- tenance of the already available technologies for managing and trading translation data. Specifi- cally, the following features will be added to an existing underlying translation data repository: • An easy-to-use mechanism to upload and an- notate data-sets for data sellers, as well as op- tions to upload updates to the data-sets; • an easy-to-explore mechanism to find the right data for specific languages and domains for data buyers; • an easy-to-trade transaction system for data sellers to earn monetary rewards by trading their data with data buyers; • an easy-to-trust reputation system to improve the confidence of data buyers towards the marketplace and to ensure quality of data. 3 State-of-the-art Processing Tools Advanced data processing services will be in- tegrated to enable and facilitate data exchange through the marketplace and to encourage data sellers and buyers to join the platform. These ser- vices consist of software for cleaning, anonymiz- ing and clustering the data to ensure that the data-sets available in the Marketplace are of high quality. These services will be provided through APIs and will be available free of charge to data providers or against a fee for users not publishing their data through marketplace. The software will also be released open source for the industrial and research communities. 4 Data …

Content truncated to 500 words