Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 192–203 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64020 Exploring Gap Filling as a Cheaper Alternative to Reading Comprehension Questionnaires when Evaluating Machine Translation for Gisting Mikel L. Forcada Dept. Lleng. i Sist. Inf., Universitat d’Alacant E-03690 St. Vicent del Raspeig, Spain mlf@ua.es Carolina Scarton, Lucia Specia Dept. of Comput. Sci., Univ. of Sheffield, Sheffield S1 4DP, U.K {l.specia,c.scarton} @shef.ac.uk Barry Haddow, Alexandra Birch School of Computing, Univ. Edinburgh, Edinburgh EH8 4AB, U.K {bhaddow,a.birch} @ed.ac.uk Abstract A popular application of machine translation (MT) is gisting: MT is consumed as is to make sense of text in a foreign language. Evaluation of the usefulness of MT for gist- ing is surprisingly uncommon. The classical method uses reading comprehension question- naires (RCQ), in which informants are asked to answer professionally-written questions in their language about a foreign text that has been machine-translated into their language. Recently, gap-filling (GF), a form of cloze test- ing, has been proposed as a cheaper alter- native to RCQ. In GF, certain words are re- moved from reference translations and read- ers are asked to fill the gaps left using the machine-translated text as a hint. This paper reports, for the first time, a comparative evalu- ation, using both RCQ and GF, of translations from multiple MT systems for the same for- eign texts, and a systematic study on the effect of variables such as gap density, gap-selection strategies, and document context in GF. The main findings of the study are: (a) both RCQ and GF clearly identify MT to be useful; (b) global RCQ and GF rankings for the MT sys- tems are mostly in agreement; (c) GF scores vary very widely across informants, making comparisons among MT systems hard, and (d) unlike RCQ, which is framed around docu- ments, GF evaluation can be framed at the sen- tence level. These findings support the use of GF as a cheaper alternative to RCQ. 1 Introduction 1.1 Machine translation for gisting Machine translation (MT) applications fall in two main groups: assimilation or gisting, and dissem- ination. Assimilation refers to the use of the raw MT output to make sense of foreign texts. Dis- semination refers to the use of the MT output as a draft translation that can be post-edited into a publishable translation. The needs of both groups of applications are quite different; for instance, an otherwise perfect Russian to English transla- tion but with no articles (some, a, the), is likely to be fine for assimilation, but would need substan- tial post-editing for dissemination. State-of-the-art MT systems are however usually evaluated —even if manually— (and optimized) with respect to their ability to produce translations that resemble refer- ences, regardless of the intended application for the system. Assimilation is by far the main use of MT in number of words translated. It is either ex- plicitly invoked, for instance, by visiting web- pages such as Google …

Content truncated to 500 words