How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1259–1270, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models Shafiq Joty, Hassan Sajjad, Nadir Durrani, Kamla Al-Mannai, Ahmed Abdelali and Stephan Vogel Qatar Computing Research Institute - Hamad Bin Khalifa University {sjoty,hsajjad,ndurrani,kmannai,aabdelali,svogel}@qf.org.qa Abstract We present novel models for domain adap- tation based on the neural network joint model (NNJM). Our models maximize the cross entropy by regularizing the loss function with respect to in-domain model. Domain adaptation is carried out by as- signing higher weight to out-domain se- quences that are similar to the in-domain data. In our alternative model we take a more restrictive approach by additionally penalizing sequences similar to the out- domain data. Our models achieve better perplexities than the baseline NNJM mod- els and give improvements of up to 0.5 and 0.6 BLEU points in Arabic-to-English and English-to-German language pairs, on a standard task of translating TED talks. 1 Introduction Rapid influx of digital data has galvanized the use of empirical methods in many fields including Ma- chine Translation (MT). The increasing availabil- ity of bilingual corpora has made it possible to automatically learn translation rules that required years of linguistic analysis previously. While ad- ditional data is often beneficial for a general pur- pose Statistical Machine Translation (SMT) sys- tem, a problem arises when translating new do- mains such as lectures (Cettolo et al., 2014), patents (Fujii et al., 2010) or medical text (Bojar et al., 2014), where either the bilingual text does not exist or is available in small quantity. All do- mains have their own vocabulary and stylistic pref- erences which cannot be fully encompassed by a system trained on the general domain. Machine translation systems trained from a sim- ple concatenation of small in-domain and large out-domain data often perform below par be- cause the out-domain data is distant or over- whelmingly larger than the in-domain data. Ad- ditional data increases lexical ambiguity by in- troducing new senses to the existing in-domain vocabulary. For example, an Arabic-to-English SMT system trained by simply concatenating in- and out-domain data translates the Arabic phrase “������ ���� �������� ������ ���� ��� ���” to “about the problem of unwanted pregnancy”. This translation is incorrect in the context of the in-domain data, where it should be translated to “about the prob- lem of choice overload”. The sense of the Ara- bic phrase taken from out-domain data completely changes the meaning of the sentence. In this paper, we tackle this problem by proposing domain adap- tation models that make use of all the data while preserving the in-domain preferences. A significant amount of research has been car- ried out recently in domain adaptation. The com- plexity of the SMT pipeline, starting from cor- pus preparation to word-alignment, and then train- ing a wide range of models opens a wide horizon to carry out domain specific adaptations. This is typically done using either data …

Content truncated to 500 words