Using BabelNet to Improve OOV Coverage in SMT

Using BabelNet to Improve OOV Coverage in SMT Jinhua Du,1 Andy Way,1 Andrzej Zydron2 1ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland 2XTM International Ltd., London, UK jdu@computing.dcu.ie, away@computing.dcu.ie, azydron@xtm-intl.com Abstract Out-of-vocabulary words (OOVs) are a ubiquitous and difficult problem in statistical machine translation (SMT). This paper studies different strategies of using BabelNet to alleviate the negative impact brought about by OOVs. BabelNet is a multilingual encyclopedic dictionary and a semantic network, which not only includes lexicographic and encyclopedic terms, but connects concepts and named entities in a very large network of semantic relations. By taking advantage of the knowledge in BabelNet, three different methods – using direct training data, domain-adaptation techniques and the BabelNet API – are proposed in this paper to obtain translations for OOVs to improve system performance. Experimental results on English–Polish and English–Chinese language pairs show that domain adaptation can better utilize BabelNet knowledge and performs better than other methods. The results also demonstrate that BabelNet is a really useful tool for improving translation performance of SMT systems. Keywords: BabelNet, SMT, unknown words, OOVs, domain adaptation 1. Introduction OOVs – source words that have no translation in the phrase table – are a ubiquitous and difficult problem for SMT. Due to the fact that they are trained on pre-defined static data set- s, SMT systems necessarily encounter OOVs when translat- ing new documents. In such circumstances, there are two main strategies deployed: (i) to output the source word ‘as is’ on the target side; or (ii) to omit it altogether. Of course, in both cases, erroneous and disfluent translations are pro- duced. The problem is exacerbated when bilingual data are scarce, or if the text to be translated is not from the same domain as the training data. OOVs are often named entities, specialized terms and ne- ologisms. For example, some person names or technical terms are phonetically transliterated from other languages. In the past, plenty of work has been done to alleviate the im- pact of OOVs, including orthographic (lexicon-induction- based) and morphosyntactic preprocessing (Popovic and Ney, 2004; Sadat and Habash, 2006; Habash, 2008; Garera et al., 2009), pivot languages (Callison-Burch et al., 2006), grapheme-based model for phonetic transliterations (Lehal and Saini, 2012; Luo et al., 2013), paraphrases (Habash, 2008; Marton et al., 2009; Du et al., 2010) and context- based semantic models (Haghighi et al., 2008; Daume-III and Jagarlamudi, 2011; Zhang et al., 2012). In this paper, we propose to use BabelNet to handle OOVs. BabelNet is both a multilingual encyclopedic dictionary, with lexicographic and encyclopedic coverage of terms, as well as a semantic network comprising 14 million en- tries which connects concepts and named entities in a very large network of semantic relations. These are called Ba- bel synsets, each of which represents a given meaning and contains all the synonyms which express that meaning in a range of different languages (Navigli and Ponzetto, 2010; Roberto Navigli, 2012; Navigli, 2012). The most recen- t version of BabelNet is 3.5 which integrates …

Content truncated to 500 words