Wikipedia as an SMT Training Corpus

Proceedings of Recent Advances in Natural Language Processing, pages 702–709, Hissar, Bulgaria, 7-13 September 2013. Wikipedia as an SMT Training Corpus Dan Tufiș Institute for AI Romanian Academy Bucharest, Romania tufis@racai.ro Radu Ion Institute for AI Romanian Academy Bucharest, Romania radu@racai.ro Ștefan Daniel Dumitrescu Institute for AI Romanian Academy Bucharest, Romania sdumitrescu@racai.ro Dan Ștefănescu University of Memphis Memphis, USA dstfnscu@memphis.edu Abstract This article reports on mass experiments sup- porting the idea that data extracted from strongly comparable corpora may successfully be used to build statistical machine translation systems of reasonable translation quality for in-domain new texts. The experiments were performed for three language pairs: Spanish- English, German-English and Romanian- English, based on large bilingual corpora of similar sentence pairs extracted from the entire dumps of Wikipedia as of June 2012. Our ex- periments and comparison with similar work show that adding indiscriminately more data to a training corpus is not necessarily a good thing in SMT. 1 Introduction Wikipedia is one of the most accessed websites of the Internet according to Alexa.com with a global rank of 6 (being outrun only by major search engines such as Google, Yahoo and Baidu and by Face-book and YouTube). Approximately 14% of all Internet users use it on a daily basis and out of these, more than 50% browse through the English version of Wikipedia which is the most comprehensive one, judged by the number of articles. Wikipedia is not a real parallel cor- pus, although many documents in different lan- guages are translations from English. Many doc- uments in one language are shortened or adapted translations 1 of documents from other (not al- ways the same) languages and this property of Wikipedia together with its size makes it the ide- al candidate of a strongly comparable corpus from which parallel sentences can be mined. In the following, we use the term MT useful data to denote sets of bilingual sentences/phrases with a high level of cross-lingual similarity, out of which a word/phrase aligner can extract transla- tion lexicons relevant for the SMT task. SMT 1 http://en.wikipedia.org/wiki/Wikipedia:Translation engines like Moses (Koehn et al., 2007) produce better translations when presented with larger and larger training parallel corpora. For a given training corpus, it is also known that Moses pro- duces better translations when presented with in- domain new texts (texts from the same domain as the training data, e.g. news, laws, medicine, etc.). Collecting parallel data from a given domain, in sufficiently large quantities to be of use for sta- tistical translation, is not an easy task. To date, OPUS2 (Tiedemann, 2012) is the largest online collection of parallel corpora, comprising of ju- ridical texts (EUROPARL and EUconst)3, medi- cal texts (EMEA), technical texts (e.g. software KDE manuals, PHP manuals), movie subtitles corpora (e.g. OpenSubs) or news (SETIMES) but these corpora are not available for all language pairs nor their sizes are similar with respect to the domain. In a previous paper (Ștefănescu et al., 2012) we described in details an open-source parallel data …

Content truncated to 500 words