One Size Does Not Fit All: Comparing NMT Representations of Different Granularities

Proceedings of NAACL-HLT 2019, pages 1504–1516 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1504 One Size Does Not Fit All: Comparing NMT Representations of Different Granularities Nadir Durrani Fahim Dalvi Hassan Sajjad Yonatan Belinkov* Preslav Nakov {ndurrani,faimaduddin,hsajjad,pnakov}@qf.org.qa Qatar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar *MIT Computer Science and Artificial Intelligence Laboratory and Harvard John A. Paulson School of Engineering and Applied Sciences, Cambridge, MA, USA belinkov@mit.edu Abstract Recent work has shown that contextualized word representations derived from neural ma- chine translation are a viable alternative to such from simple word predictions tasks. This is because the internal understanding that needs to be built in order to be able to translate from one language to another is much more comprehensive. Unfortunately, computational and memory limitations as of present prevent NMT models from using large word vocabu- laries, and thus alternatives such as subword units (BPE and morphological segmentations) and characters have been used. Here we study the impact of using different kinds of units on the quality of the resulting representations when used to model morphology, syntax, and semantics. We found that while representa- tions derived from subwords are slightly bet- ter for modeling syntax, character-based repre- sentations are superior for modeling morphol- ogy and are also more robust to noisy input. 1 Introduction Recent years have seen the revolution of deep neu- ral networks and the subsequent rise of represen- tation learning based on network-internal activa- tions. Such representations have been shown use- ful when addressing various problems from fields such as image recognition (He et al., 2016), speech recognition (Bahdanau et al., 2016), and natu- ral language processing (NLP) (Mikolov et al., 2013a). The central idea is that the internal rep- resentations trained to solve an NLP task could be useful for other tasks as well. For example, word embeddings learned for a simple word pre- diction task in context, word2vec-style (Mikolov et al., 2013b), have now become almost obligatory in state-of-the-art NLP models. One issue with such word embeddings is that the resulting repre- sentation is context-independent. Recently, it has been shown that huge performance gains can be achieved by contextualizing the rep- resentations, so that the same word could have a different embedding in different contexts. This is best achieved by changing the auxiliary task. For example, ELMo (Peters et al., 2018) learns contex- tualized word embeddings from language model- ing (LM) using long short-term memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997). It has been further argued that complex aux- iliary tasks such as neural machine translation (NMT) are better tailored for representation learn- ing, as the internal understanding of the input lan- guage that needs to be built by the network to be able to translate from one language to another needs to be much more comprehensive compared to what would be needed for a simple word predic- tion task. This idea is implemented in the seq2seq- based CoVe model (McCann et al., 2017). More recently, the …

Content truncated to 500 words