Measuring Machine Translation User Experience (MTUX): A Comparison between AttrakDiff and User Experience Questionnaire

© 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC-BY-ND. Measuring Machine Translation User Experience (MTUX): A Comparison between AttrakDiff and User Experience Questionnaire Vicent Briva-Iglesias SFI CRT D-REAL, SALIS Dublin City University vicent.brivaiglesias2@mail.dcu.ie Sharon O’Brien SALIS Dublin City University sharon.obrien@dcu.ie Abstract Perceptions and experiences of machine translation (MT) users before, during, and af- ter their interaction with MT systems, prod- ucts or services has been overlooked both in academia and in industry. Traditionally, the focus has been on productivity and quality, often neglecting the human factor. We pro- pose the concept of Machine Translation User Experience (MTUX) for assessing, evaluat- ing, and getting further information about the user experiences of people interacting with MT. By conducting a human-computer inter- action (HCI)-based study with 15 profes- sional translators, we present a methodologi- cal paper in which we analyse which is the best method for measuring MTUX, and con- clude by suggesting the use of the User Expe- rience Questionnaire (UEQ). The measure- ment of MTUX will help every stakeholder in the MT industry - developers will be able to identify pain points for the users and solve them in the development process, resulting in better MTUX and higher adoption of MT sys- tems or products by MT users. 1 Introduction Recently, artificial intelligence has captured the attention of many stakeholders in our society, not only in specialised academic journals and conferences, but also among laypeople (Fast and Horvitz 2016). Large language models have driven technological breakthroughs, and the state-of-the-art has evolved mainly through training bigger and bigger models, with more parameters, more training time, and ultimately more computational resources (Brown et al. 2020). Research in language technologies has become a race to see who owns and releases the biggest language model (Roose 2023). This has also provoked the reaction of academics who reflect on language technology research from a socio-technical perspective, promoting a move to a more human- centered development of such language technologies (Bender & Gebru et al. 2021), which goes beyond ‘human in the loop’ concepts. In the language services or Translation Studies domains, MT is a technology that has had significant impact in the past few years, and its adoption and implementation in workflows has provoked some rejection from professional translators (Cadwell, O’Brien, and Teixeira 2018). Many professional users feel that their needs have not been considered in the development and deployment of these technologies, and have therefore felt dehumanised, commodified, with an accompanying loss of agency and status (Fırat 2021; Moorkens 2020). This results in a lack of acceptance and trust in these technologies, which is usually not a rejection of the technology, but a veto on the way in which MT is applied and used (Vieira 2020). Human factors such as users' perceptions or experiences of MT as a tool that facilitates multilingual communication - regardless of whether we are talking about professional translators or other types of users - have often been overlooked. The focus …

Content truncated to 500 words