IRAC: A Domain-Specific Annotated Corpus of Implicit Reasoning in Arguments

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 29–38 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 29 GECO-MT: The Ghent Eye-tracking Corpus of Machine Translation Toon Colman1, Margot Fonteyne1, Joke Daems1, Nicolas Dirix2, Lieve Macken1 1LT3, Language and Translation Technology Team, Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium 2Department of Experimental Psychology, Ghent University Henri Dunantlaan 2, 9000 Ghent, Belgium {toon.colman, margot.fonteyne, joke.daems, nicolas.dirix, lieve.macken}@ugent.be Abstract In the present paper, we describe a large corpus of eye movement data, collected during natural reading of a human translation and a machine translation of a full novel. This data set, called GECO-MT (Ghent Eye-tracking Corpus of Machine Translation) expands upon an earlier corpus called GECO (Ghent Eye-tracking Corpus) by Cop et al. (2017). The eye movement data in GECO-MT will be used in future research to investigate the effect of machine translation on the reading process and the effects of various error types on reading. In this article, we describe in detail the materials and data collection procedure of GECO-MT. Extensive information on the language proficiency of our participants is given, as well as a comparison with the participants of the original GECO. We investigate the distribution of a selection of important eye movement variables and explore the possibilities for future analyses of the data. GECO-MT is freely available at https://www.lt3.ugent.be/resources/geco-mt. Keywords: literary machine translation, quality assessment, reading behaviour, eye-tracking, corpus study 1. Introduction The quality of machine translation (MT) output has in- creased greatly over the last decade, mainly thanks to a paradigm shift from statistical machine translation (SMT) systems to neural machine translation (NMT) systems (Wu et al., 2016). NMT generally outperforms SMT, as it is better able to account for (sentence) con- text and can map the meaning of words more finely. Both human evaluation methods and automatic metrics have shown NMT output to be more qualitative than SMT output (Bentivogli et al., 2016; Burchardt et al., 2017; Toral and S´anchez-Cartagena, 2017; Klubiˇcka et al., 2018; Van Brussel et al., 2018; Shterionov et al., 2018; Jia et al., 2019; Daems and Macken, 2019). Thanks to the increasing quality of MT, readers might be more often confronted with ‘raw’ MT output, with- out any post-editing1 (Macken et al., 2020). However, despite overall quality improvements, re- markable differences can be observed when compar- ing machine translations (MT) and human translations (HT), especially when considering more creative text types such as literary text. Webster et al. (2020) com- pared the Dutch HTs of four classic English novels with their MT versions, generated by Google Translate and DeepL (two NMT systems). They found that a large proportion of MT sentences contained errors. Using the SCATE (Smart Computer-aided Translation Environ- ment) MT error taxonomy by Tezcan et al. (2017), they observed that the most frequent error types in their data set were (1) mistranslations, (2) coherence errors, and 1The ArisToCAT project (Assessing the Comprehensibil- ity of Automatic Translations) – which this study is …

Content truncated to 500 words