Outbound Translation User Interface Ptakopět: A Pilot Study

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6967–6975 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6967 Outbound Translation User Interface Ptakopˇet: A Pilot Study Vilém Zouhar, Ondˇrej Bojar Charles University Prague, Czech Republic {zouhar, bojar}@ufal.mff.cuni.cz Abstract It is not uncommon for Internet users to have to produce a text in a foreign language they have very little knowledge of and are unable to verify the translation quality. We call the task “outbound translation” and explore it by introducing an open-source modular system Ptakopˇet. Its main purpose is to inspect human interaction with MT systems enhanced with additional subsystems, such as backward translation and quality estimation. We follow up with an experiment on (Czech) human annotators tasked to produce questions in a language they do not speak (German), with the help of Ptakopˇet. We focus on three real-world use cases (communication with IT support, describing administrative issues and asking encyclopedic questions) from which we gain insight into different strategies users take when faced with outbound translation tasks. Round trip translation is known to be unreliable for evaluat- ing MT systems but our experimental evaluation documents that it works very well for users, at least on MT systems of mid-range quality. Keywords: Quality Estimation, Machine Translation, Outbound Translation 1. Introduction For most language pairs, machine translation (MT) quality is limited. Nevertheless, MT in everyday use greatly helps by providing low quality, preview translation also called gisting. The complement of gisting is outbound translation. In both cases, a message is transferred between the author and the recipient and each of them has sufficient knowl- edge of only their language. In outbound translation, the author is responsible for creating correct messages in the recipient’s language. In gisting, the message is sent in the author’s language and the responsibility to correctly inter- pret it lies on the recipient. An example of gisting would be browsing on a website in a foreign language, whilst fill- ing in a form in a foreign language would be an example of outbound translation. When translating to foreign languages, users cooperate with machine translation tools to produce the best result. Machine translation can prepare a first version of the text, or it can be used to verify the user’s own translation to some extent. Users translating into languages which they do not master enough to validate the translation need some additional sys- tem for verification and assurance that the machine transla- tion output is valid. For this, Ptakopˇet offers word-level quality estimation (QE), simulated source complexity and backward translation. While round trip translation may be unreliable for fully automatic evaluation of MT quality (Somers, 2005), it is still a widespread strategy for users. The paper is structured as follows: We briefly introduce the components we rely on in Section 2. and describe Ptakopˇet in Section 3., including the underlying models. The exper- iment setup is presented in Section 4. and the results in Section 5. We conclude in …

Content truncated to 500 words