Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 349–362 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers Andr´e F. T. Martins Unbabel & Instituto de Telecomunicac¸˜oes Lisbon, Portugal andre.martins@unbabel.com Julia Kreutzer∗ Computational Linguistics Heidelberg University, Germany kreutzer@cl.uni-heidelberg.de Abstract We introduce a novel neural easy-first de- coder that learns to solve sequence tag- ging tasks in a flexible order. In con- trast to previous easy-first decoders, our models are end-to-end differentiable. The decoder iteratively updates a “sketch” of the predictions over the sequence. At its core is an attention mechanism that con- trols which parts of the input are strategi- cally the best to process next. We present a new constrained softmax transformation that ensures the same cumulative attention to every word, and show how to efficiently evaluate and backpropagate over it. Our models compare favourably to BILSTM taggers on three sequence tagging tasks. 1 Introduction In the last years, neural models have led to ma- jor advances in several structured NLP problems, including sequence tagging (Plank et al., 2016; Lample et al., 2016), sequence-to-sequence pre- diction (Sutskever et al., 2014), and sequence-to- tree (Dyer et al., 2015). Part of the success comes from clever architectures such as (bidirectional) long-short term memories (BILSTMs; Hochreiter and Schmidhuber (1997); Graves et al. (2005)) and attention mechanisms (Bahdanau et al., 2015), which are able to select the pieces of context rele- vant for prediction. A noticeable aspect about many of the systems above is that they typically decode from left to right, greedily or with a narrow beam. While this is computationally convenient and reminis- cent of the way humans process spoken language, the combination of unidirectional decoding and ∗This research was partially carried out during an in- ternship at Unbabel. greediness leads to error propagation and subopti- mal classification performance. This can partly be mitigated by globally normalized models (Andor et al., 2016) and imitation learning (Daum´e et al., 2009; Ross et al., 2011; Bengio et al., 2015), how- ever these techniques still have a left-to-right bias. Easy-first decoders (Tsuruoka and Tsujii, 2005; Goldberg and Elhadad, 2010, §2) are an in- teresting alternative: instead of a fixed decoding order, these methods schedule their own actions by prefering “easier” decisions over more diffi- cult ones. A disadvantage is that these models are harder to learn, due to the factorial number of orderings leading to correct predictions. Usually, gradients are not backpropagated over this combi- natorial latent space (Kiperwasser and Goldberg, 2016a), or a separate model is used to determine the easiest next move (Clark and Manning, 2016). In this paper, we develop novel, fully differen- tiable, neural easy-first sequence taggers (§3). Instead of taking discrete actions, our decoders use an attention mechanism to decide (in a soft man- ner) which word to focus on for the next tagging decision. Our models are able to learn their own sense of “easiness”: the words receiving focus may not be the ones …

Content truncated to 500 words