Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 427–436 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 427 Norm-Based Curriculum Learning for Neural Machine Translation Xuebo Liu1∗ Houtim Lai2∗ Derek F. Wong1† Lidia S. Chao1 1NLP2CT Lab, Department of Computer and Information Science, University of Macau 2NewTranx Information Technology, Shenzhen, China nlp2ct.xuebo@gmail.com, haotian.li@newtranx.com, {derekfw,lidiasc}@um.edu.mo Abstract A neural machine translation (NMT) system is expensive to train, especially with high- resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to im- prove the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sen- tence, 2) the competence of the model, and 3) the weight of the sentence. The norm- based sentence difficulty takes the advantages of both linguistically motivated and model- based sentence difficulties. It is easy to de- termine and contains learning-dependent fea- tures. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sen- tence weight further enhances the learning of the vector representation of the NMT. Ex- perimental results for the WMT’14 English– German and WMT’17 Chinese–English trans- lation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score (+1.17/+1.56) and training speedup (2.22x/3.33x). 1 Introduction The past several years have witnessed the rapid development of neural machine translation (NMT) based on an encoder–decoder framework to trans- late natural languages (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Since NMT benefits from a massive amount of training data and works in a cross-lingual setting, it becomes much hungrier for training time than other natural language processing (NLP) tasks. ∗Equal Contribution †Corresponding author Based on self-attention networks (Parikh et al., 2016; Lin et al., 2017), Transformer (Vaswani et al., 2017) has become the most widely used architec- ture for NMT. Recent studies on improving Trans- former, e.g. deep models equipped with up to 30- layer encoders (Bapna et al., 2018; Wu et al., 2019; Wang et al., 2019; Zhang et al., 2019a), and scaling NMTs which use a huge batch size to train with 128 GPUs (Ott et al., 2018; Edunov et al., 2018), face a challenge to the efficiency of their training. Curriculum learning (CL), which aims to train ma- chine learning models better and faster (Bengio et al., 2009), is gaining an intuitive appeal to both academic and industrial NMT systems. The basic idea of CL is to train a model using examples ranging from “easy” to “difficult” in dif- ferent learning stages, and thus the criterion of dif- ficulty is vital to the selection of examples. Zhang et al. (2018) summarize two kinds of difficulty cri- teria in CL for NMT: 1) linguistically motivated sentence difficulty, e.g. sentence length, word fre- quency, and the number of coordinating conjunc- …

Content truncated to 500 words