MAGMATic: A Multi-domain Academic Gold Standard with Manual Annotation of Terminology for Machine Translation Evaluation

MAGMATic: A Multi-domain Academic Gold Standard with Manual Annotation of Terminology for Machine Translation Evaluation Randy Scansani University of Bologna Forl`ı, Italy randy.scansani@unibo.it Luisa Bentivogli Fondazione Bruno Kessler Trento, Italy bentivo@fbk.eu Silvia Bernardini University of Bologna Forl`ı, Italy silvia.bernardini@unibo.it Adriano Ferraresi University of Bologna Forl`ı, Italy adriano.ferraresi@unibo.it Abstract This paper presents MAGMATic (Multi- domain Academic Gold Standard with Manual Annotation of Terminology), a novel Italian–English benchmark which al- lows MT evaluation focused on terminol- ogy translation. The data set comprises 2,056 parallel sentences extracted from in- stitutional academic texts, namely course unit and degree program descriptions. This text type is particularly interesting since it contains terminology from multiple do- mains, e.g. education and different aca- demic disciplines described in the texts. All terms in the English target side of the data set were manually identified and an- notated with a domain label, for a total of 7,517 annotated terms. Due to their pe- culiar features, institutional academic texts represent an interesting test bed for MT. As a further contribution of this paper, we in- vestigate the feasibility of exploiting MT for the translation of this type of docu- ments. To this aim, we evaluate two state- of-the-art Neural MT systems on MAG- MATic, focusing on their ability to trans- late domain-specific terminology. 1 Introduction The availability of bilingual versions of course cat- alogues has started to play a major role for Euro- pean universities after the Bologna Process and the resulting growth in students’ mobility. Course cat- alogues fall into the category of institutional aca- c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. demic text collections and they usually include de- gree program and course unit descriptions, where information regarding degree courses and modules are provided to students. Such texts have to be produced and published every year in each coun- try language and in English. Universities would thus undoubtedly benefit from the use of machine translation (MT). Further proof of the need for an MT engine able to translate course catalogues are two projects funded by the European Commission, namely the Bologna Translation Service1 (Depraetere et al., 2011), aimed at developing an MT system to trans- late course catalogues in 9 language combinations, and TraMOOC,2 aimed at using MT for the trans- lation of massive online open courses from English into eleven European and BRIC languages. Developing an engine in this field poses sev- eral challenges. First, the fact that degree program and course unit descriptions are usually trans- lated by non-native speakers of the target language (Fernandez Costales, 2012) reduces the number of available high-quality and alignable bilingual texts. Moreover, the lack of guidelines and best practices to draft these texts results in substantial unmotivated variation among course catalogues from different universities. Finally, institutional academic texts usually contain terminology from different domains, with disciplinary terms, e.g. Hydrosilylation, Fotoredox catalysis, for a course on chemistry, appearing together with educational ones - e.g. ECTS, module. The potential and …

Content truncated to 500 words