Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources?

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1159–1168, Osaka, Japan, December 11-17 2016. Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources? 1LIG – Univ. Grenoble Alpes Domaine Universitaire 38401 St Martin d’Hères, France firstname.lastname @imag.fr Christophe Servan1,2, Alexandre Bérard1, Zied Elloumi1,3, Hervé Blanchon1 & Laurent Besacier1 2SYSTRAN 5 Rue Feydeau 75002 Paris, France firstname.lastname @systran.fr 3LNE 29 Avenue Roger Hennequin 78190 Trappes, France firstname.lastname @lne.fr Abstract This paper presents an approach combining lexico-semantic resources and distributed rep- resentations of words applied to the evaluation in machine translation (MT). This study is made through the enrichment of a well-known MT evaluation metric: METEOR. This metric enables an approximate match (synonymy or morphological similarity) between an automatic and a reference translation. Our experiments are made in the framework of the Metrics task of WMT 2014. We show that distributed representations are a good alternative to lexico-semantic resources for MT evaluation and they can even bring interesting additional information. The augmented versions of METEOR, using vector representations, are made available on our Github page. 1 Introduction Learning vector representations of words using neural networks has generated a strong enthusiasm in the NLP research community. In particular, many contributions were proposed after the work of (Mikolov et al., 2013a; Mikolov et al., 2013b; Mikolov et al., 2013c) on training word embeddings. The main reasons for this strong interest are: the proposal of a simple and efficient neural architecture to learn word vector representations, the availability of an open source tool Word2Vec1 and the rapid structuring of a user community2. Later on, several contributions have extended the work of Mikolov on word vectors to phrases (sequences of words) (Mikolov et al., 2013b; Le and Mikolov, 2014a) and to bilingual representations (Luong et al., 2015). All these vector representations capture similarities between words, phrases or sentences at different levels (morphological, semantic). However, although these representations can be semantically informative, they do not exactly replace fine-grained information available in lexical-semantic resources such as WordNet (Fellbaum, 1998), Ba- belNet (Navigli and Ponzetto, 2010), or DBnary (Sérasset, 2012). Such lexical resources are also more easily interpretable by humans as shown in (Panchenko, 2016), but their construction is costly while word embeddings can be trained ad infinitum on any monolingual or bilingual corpora. In short, both approaches (lexical resources and word embeddings) have their pros and cons. However, few studies have attempted to compare and combine them. Pioneering work of Faruqui et al. (2014) proposed to refine representations learning using lexical resources. The idea is to force words connected in the lexical network, to have a close representation (for example through a synonymy link). The technique proposed is evaluated on several benchmarks (word similarity, sentiment analysis, finding of synonyms). More recently, Panchenko (2016) and Rothe and Schütze (2015) extended word embeddings to sense embeddings and tried to compare them to lexical synsets. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ 1http://word2vec.googlecode.com/svn/trunk/ 2https://groups.google.com/d/forum/word2vec-toolkit 1159 …

Content truncated to 500 words