Crowdsourcing High-Quality Parallel Data Extraction from Twitter

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 426–436, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Crowdsourcing High-Quality Parallel Data Extraction from Twitter∗ Wang Ling123 Luís Marujo123 Chris Dyer2 Alan Black2 Isabel Trancoso 13 (1)L2F Spoken Systems Lab, INESC-ID, Lisbon, Portugal (2)Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA (3)Instituto Superior Técnico, Lisbon, Portugal {lingwang,lmarujo,cdyer,awb}@cs.cmu.edu isabel.trancoso@inesc-id.pt Abstract High-quality parallel data is crucial for a range of multilingual applications, from tuning and evaluating machine translation systems to cross-lingual annotation pro- jection. Unfortunately, automatically ob- tained parallel data (which is available in relative abundance) tends to be quite noisy. To obtain high-quality parallel data, we introduce a crowdsourcing paradigm in which workers with only basic bilin- gual proficiency identify translations from an automatically extracted corpus of par- allel microblog messages. For less than $350, we obtained over 5000 parallel seg- ments in five language pairs. Evaluated against expert annotations, the quality of the crowdsourced corpus is significantly better than existing automatic methods: it obtains an performance comparable to expert annotations when used in MERT tuning of a microblog MT system; and training a parallel sentence classifier with it leads also to improved results. The crowdsourced corpora will be made avail- able in http://www.cs.cmu.edu/ ~lingwang/microtopia/. 1 Introduction High-quality parallel data is essential for tun- ing and evaluating statistical MT systems, and it plays a role in a wide range of multilingual NLP applications, such as word sense disambigua- tion (Gale et al., 1992; Ng et al., 2003; Specia et al., 2005), paraphrasing (Bannard and Callison- burch, 2005; Ganitkevitch et al., 2012), annota- tion projection (Das and Petrov, 2011), and other language-specific applications (Schwarck et al., ∗ A sample of the crowdsourced corpora and the inter- faces used are available as supplementary material. 2010; Liu et al., 2011). While large amounts of parallel data can be easily obtained by mining the web (Resnik and Smith, 2003), comparable corpora (Munteanu and Marcu, 2005), and even social media sites (Ling et al., 2013), automati- cally extracted parallel tends to be noisy, and, as a result, “evaluation-quality” parallel corpora have generally been produced at considerable expense by targeted translation efforts (Bojar et al., 2013, inter alia). Unfortunately, in some domains such as microblogs, the only corpora that are available are automatically extracted and noisy. While phrase-based translation models can ef- fectively learn translation rules from noisy parallel data (Goutte et al., 2012), having a subset of high- quality parallel segments is nevertheless crucial. Firstly, the automatic parallel data extraction sys- tem’s parameters can be tuned by optimizing on the gold standard data. Secondly, even though the parallel data used to train MT systems can contain a considerable amount of noise, it is conventional to use human annotated parallel data to tune and evaluate the system. Finally, other NLP applica- tions may not be as noise-robust as MT. We introduce a new crowdsourcing protocol for obtaining high-quality parallel data from noisy, automatically extracted parallel data (§3), focus- ing on the challenging …

Content truncated to 500 words