The MT@BZ corpus: machine translation & legal language

The MT@BZ corpus: machine translation & legal language Flavia De Camillis1 and Egon W. Stemle1 and Elena Chiocchetti1 and Francesco Fernicola1,2 Flavia.DeCamillis,Egon.Stemle,Elena.Chiocchetti,Francesco.Fernicola@eurac.edu 1 Institute for Applied Linguistics, Eurac Research, Bolzano/Bozen, Italy 2 Università di Bologna, Forlì, Italy Abstract The paper reports on the creation, annota- tion and curation of the MT@BZ corpus, a bilingual (Italian–South Tyrolean German) corpus of machine-translated legal texts from the officially multilingual Province of Bolzano, Italy. It is the first human error-annotated corpus (with an adapted SCATE taxonomy) of machine-translated legal texts in this language combination that includes a lesser-used standard variety. Project data are available on GitHub and CLARIN.1 The output of the customized engine achieved notably better BLEU, TER and chrF2 scores than the baseline. Over 50% of the segments needed no human revision. The most frequent error cate- gories were mistranslations and bilingual (legal) terminology errors. Our contribu- tion brings fine-grained insights to Machine Translation Evaluation research, as it con- cerns a less common language combination, a lesser-used language variety and a soci- etally relevant specialized domain. Such results are necessary to implement and in- form the use of MT in institutional contexts of smaller language communities. 1 Introduction Machine translation evaluation (MTE) assesses the performance of machine translation (MT) systems. It can be human or automatic. While automatic © 2023 The authors. This article is licensed under a Cre- ative Commons 4.0 licence, no derivative works, attribution, CC-BY-ND. 1Code https://gitlab.inf.unibz.it/commul/mt-bz/, corpus data http://hdl.handle.net/20.500.12124/60, annotation guide- lines http://hdl.handle.net/20.500.12124/62. metrics are quickly computed and offer an idea of how a system performs, human evaluation is time- consuming and expensive but offers detailed in- sights into what machines get right or wrong when translating. Human MTE usually considers accu- racy and fluency. Accuracy measures “the extent to which the translation transfers the meaning of the source-language unit into the target”, while flu- ency assesses “the extent to which the translation follows the rules and norms of the target language” (Castilho et al., 2018, 18). Error classification and analysis may be considered a subtask of hu- man MTE. It requires a detailed error taxonomy and a group of annotators (Popovi´c, 2018, 131– 32). In the past, different error taxonomies have been developed, but none was adapted or tested on the combination Italian-South Tyrolean German or on a lesser-used standard variety of a major Euro- pean language.2 A widely used error classification framework is the Multidimensional Quality Metrics (MQM) (Lommel et al., 2014), with a hierarchical list of categories and a flexible and customizable application that ensure different levels of granular- ity. Despite its flexibility, in our project we opted for the SCATE taxonomy, as the possibility of link- ing target annotations to source spans helps inter- pret terminology issues, our main interest. Besides, the availability of a ready-to-use annotation project with the SCATE taxonomy was an added value. Despite the substantial improvements achieved thanks to neural technologies (Kenny, 2022, 43), MT still struggles with some language combina- tions in the legal domain (Wiesmann, 2019), more than …

Content truncated to 500 words