Complexity of spoken versus written language for machine translation

Complexity of Spoken Versus Written Language for Machine Translation Nicholas Ruiz University of Trento Fondazione Bruno Kessler Trento, Italy nicruiz@fbk.eu Marcello Federico Fondazione Bruno Kessler Trento, Italy federico@fbk.eu Abstract When machine translation researchers par- ticipate in evaluation tasks, they typically design their primary submissions using ideas that are not genre-specific. In fact, their systems look much the same from one evaluation campaign to another. In this pa- per, we analyze two popular genres: spo- ken language and written news, using pub- licly available corpora which stem from the popular WMT and IWSLT evaluation campaigns. We show that there is a suf- ficient amount of difference between the two genres that particular statistical mod- eling strategies should be applied to each task. We identify translation problems that are unique to each translation task and ad- vise researchers of these phenomena to fo- cus their efforts on the particular task. 1 Introduction The machine translation community has consis- tently used the translation of news texts and news commentaries as some of its prime methods of evaluating the progression of MT research. News translation evaluation tasks have existed since the first NIST evaluations in the early 2000s, followed by the Workshop on Machine Translation (WMT) (Bojar et al., 2013). In recent years, TED talks have attracted the in- terest of the MT research community for measur- ing progress. The International Workshop on Spo- ken Language Translation (IWSLT) is currently in its fifth year of hosting TED talk evaluation cam- paigns, with a growing number of translation lan- guages and participants (Cettolo et al., 2013). Both c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. the WMT and IWSLT evaluations have enjoyed strong performance results that have progressively improved year after year and are established today as the de-facto evaluation tasks for text and speech translation, respectively. In practice, the top per- forming MT systems use many of the same train- ing and decoding approaches in these evaluations. But are the WMT and IWSLT translation tasks just different flavors of the same translation prob- lem? Are the strategies used to translate written language directly applicable to the genre of spo- ken language – in particular, prepared speeches? This paper investigates the question of what makes MT difficult for speech corpora as opposed to text corpora. We try to understand the differ- ences between the genres of news texts and pre- pared speeches, both in qualitative and quantita- tive terms. The ultimate goal is to find information that could explain differences in MT system per- formance and the types of errors occurring often in MT systems trained on text and speech corpora. We begin by surveying some of the aspects of language that make MT hard and how they relate to the problem of human understanding of text (Sec- tions 2 and 3). We follow up the discussion with a detailed analysis to determine if these aspects are distinctive of IWSLT or WMT, or …

Content truncated to 500 words