Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT

Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT Marianna J. Martindale mmartind@umd.edu iSchool, University of Maryland, College Park, 20740, USA Marine Carpuat marine@cs.umd.edu Dept. of Computer Science, University of Maryland, College Park, 20740, USA Abstract Although measuring intrinsic quality has been a key factor in the advancement of Machine Translation (MT), successfully deploying MT requires considering not just intrinsic quality but also the user experience, including aspects such as trust. This work introduces a method of studying how users modulate their trust in an MT system after seeing errorful (disfluent or inadequate) output amidst good (fluent and adequate) output. We conduct a survey to determine how users respond to good translations compared to translations that are either adequate but not fluent, or fluent but not adequate. In this pilot study, users responded strongly to disfluent translations, but were, surprisingly, much less concerned with adequacy. 1 Introduction Machine translation has reached a level of quality such that it can now provide fast, cheap, “good enough” translations for a wide variety of information sources in an increasingly diverse set of languages and dialects. These systems are becoming available for real users who have no other way to understand content in languages in which they are not proficient. While the overall quality of MT output is important, in practice the utility of MT also depends on the willingness of users to accept the technology. Trust is an important factor in that willingness regardless of use case. In the case of MT for communication, users may choose not to communicate if they do not trust the MT available to them. If translators do not trust the MT in their computer-assisted translation environment, they may be less efficient. When there is a need for MT for understanding, insufficient trust in MT may lead the user to simply avoid content in languages they do not understand. However lack of trust is not the only issue. Implicit trust in incorrect MT can be at least as problem- atic. Users may misunderstand inaccurately translated information, an automatically translated website could reflect badly on the company, and inaccurate translations used in communication could be embarrassing. However, there could be more serious repercussions if the MT is used in critical tasks. This is not merely theoretical: In October 2017, a Palestinian man was mistakenly arrested by Israeli police based on an inaccurate Facebook translation (Berger, 2017). Because trust is so important, we need to understand how users modulate their trust in MT. However trust has been a neglected element of MT evaluation to date. To be most effective, trust evaluation must be taken into consideration not only at deployment time, but also dur- ing research and development. Understanding what factors affect user trust can help identify the most problematic error types, informing research directions. In this paper, we introduce a method of studying user trust in MT in a lab setting, based on previous work on trust in other Proceedings of AMTA 2018, vol. 1: …

Content truncated to 500 words