Measuring the Effect of Conversational Aspects on Machine Translation Quality

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2571–2581, Osaka, Japan, December 11-17 2016. Measuring the Effect of Conversational Aspects on Machine Translation Quality Marlies van der Wees Arianna Bisazza Christof Monz Informatics Institute, University of Amsterdam {m.e.vanderwees,a.bisazza,c.monz}@uva.nl Abstract Research in statistical machine translation (SMT) is largely driven by formal translation tasks, while translating informal text is much more challenging. In this paper we focus on SMT for the informal genre of dialogues, which has rarely been addressed to date. Concretely, we investigate the effect of dialogue acts, speakers, gender, and text register on SMT quality when translating fictional dialogues. We first create and release a corpus of multilingual movie dialogues annotated with these four dialogue-specific aspects. When measuring translation performance for each of these variables, we find that BLEU fluctuations between their categories are often significantly larger than randomly expected. Following this finding, we hypothesize and show that SMT of fictional dialogues benefits from adaptation towards dialogue acts and registers. Finally, we find that male speakers are harder to translate and use more vulgar language than female speakers, and that vulgarity is often not preserved during translation. 1 Introduction Research in statistical machine translation (SMT) has mostly been driven by formal translation tasks. These are, however, not representative for the abundance of informal data emerging on the Internet, for which state-of-the-art SMT systems perform markedly worse (van der Wees et al., 2015a). Recent years have therefore shown an increasing effort in improving SMT for informal text, for example by normalizing noisy text to more formal text (Bertoldi et al., 2010; Banerjee et al., 2012; Ling et al., 2013a), or by enhancing formal training data with user-generated data (Banerjee et al., 2011; Jehl et al., 2012; Ling et al., 2013b). In this paper we focus on SMT for dialogues, an informal genre that involves, by definition, multiple speakers, and is thus noticeably different from formal text (Fern´andez, 2014). Formal text is typically written by a single writer with a clear intention (e.g., informing or persuading), and moreover has been editorially controlled according to standards of language use. In dialogues, on the other hand, different speakers have different intentions and language use, affected, for example, by their gender. Such vari- ations are reflected by register, a term referring to socio-situational language variation (Lee, 2001), and dialogue acts, functional actions such as questions or answers (Bunt, 1979). While these and other dialogue-specific aspects have been analyzed in dialogue research (Schlangen, 2005; Fern´andez, 2014), their impact on SMT has hardly been studied. A likely explanation is the lack of adequate evaluation data, i.e., parallel conversations annotated with dialogue-specific variabels. In this paper, we take a first step towards investigating the effect of dialogue acts, speakers, gender, and register on SMT performance by measuring their respective impact on annotated dialogues from movie subtitles. Since movie dialogues are fictional, we can only consider them as an approximation of real face- to-face conversation. However, several corpus-based studies have shown that, …

Content truncated to 500 words