A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

Learning an Interactive Attention Policy for Neural Machine Translation Samee Ibraheem* sibraheem@berkeley.edu Nicholas Altieri* naltieri@berkeley.edu John DeNero denero@berkeley.edu *Equal contribution. Abstract Interactive machine translation research has focused primarily on predictive typing, which re- quires a human to type parts of the translation. This paper explores an interactive setting in which humans guide the attention of a neural machine translation system in a manner that requires no text entry at all. The system generates a translation from left to right, but waits pe- riodically for a human to select the word in the source sentence to be translated next. A central technical challenge is that the system must learn when and how often to request guidance from the human. These decisions allow the system to trade off translation speed and accuracy. We cast these decisions as a reinforcement learning task and develop a policy gradient approach to train the system. Critically, the system can be trained on parallel data alone by simulating human guidance at training time. Our experiments demonstrate the viability of this interac- tive setting to improve translation quality and show that an effective policy for periodically requesting human guidance can be learned automatically. 1 Introduction Despite rapid advances in neural machine translation, human input is still needed to meet the translation quality requirements of many applications. Interactive machine translation seeks to combine the quality of human translation with the speed and lexical coverage of machine translation. This paper explores an interactive setting in which the human translator does not type at all, but instead guides the attention of a neural machine translation system by selecting relevant source words as the system translates. While we should not expect that the resulting translations will be as accurate as those produced by predictive typing, this interactive approach could provide fast and accurate draft translations that could later be improved by post-editing. Moreover, source word selection enables new user interface options because it can be performed using a wide variety of input devices, including a mouse, a touch screen, or an eye tracker, which may be used in tandem with traditional text entry methods. We first address the question of whether guiding the attention of a neural machine trans- lation system can provide enough useful information to improve translation quality. Rather than experimenting directly with human subjects, we compute an experimental upper bound on the accuracy gains from guided attention. For each word that the system is meant to generate, we find an oracle attention that maximizes the probability of generating that word. We find that guiding attention toward this oracle provides a great deal of information to the translation system, yielding substantial gains in translation quality. Second, we define an interactive translation process in which the system generates a trans- lation left-to-right, but pauses on occasion to request guidance from a human collaborator. Ide- Proceedings of MT Summit XVI, vol.1: Research Track Nagoya, Sep. 18-22, 2017 | p. 108 ally, the system would not pause after every word; if the system …

Content truncated to 500 words