On Biasing Transformer Attention Towards Monotonicity

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4474–4488 June 6–11, 2021. ©2021 Association for Computational Linguistics 4474 On Biasing Transformer Attention Towards Monotonicity Annette Rios1, Chantal Amrhein1, Noëmi Aepli1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh {rios,amrhein,naepli,sennrich}@cl.uzh.ch Abstract Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is com- patible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, mor- phological inflection, transliteration, and di- alect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead atten- tion, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior. 1 Introduction Many sequence-to-sequence tasks in natural lan- guage processing are roughly monotonic in the alignment between source and target sequence, and previous work has focused on learning monotonic attention behavior either through specialized atten- tion functions (Aharoni and Goldberg, 2017; Raffel et al., 2017; Wu and Cotterell, 2019) or pretraining (Aji et al., 2020). However, it is non-trivial to port specialized attention functions to different models, and recently, Yolchuyeva et al. (2019); Wu et al. (2021) found that a transformer model (Vaswani et al., 2017) outperforms previous work on mono- tone tasks such as grapheme-to-phoneme conver- sion, despite having no mechanism that biases the model towards monotonicity. In the transformer, it is less straightforward to what extent individual encoder states, especially in deeper layers, still represent distinct source in- puts after passing through several self-attention layers. Consequently, it is unclear whether enforc- ing monotonicity in the transformer is as beneficial as for recurrent neural networks (RNNs). In this paper, we investigate the following re- search questions: 1. How can we incorporate a monotonicity bias into attentional sequence-to-sequence models such as the transformer? 2. To what extent does a transformer model ben- efit from such a bias? Specifically, we want to incorporate a mono- tonicity bias in a way that is agnostic of the task and model architecture, allowing for its applica- tion to different sequence-to-sequence models and tasks. To this end, we introduce a loss function that measures and rewards monotonic behavior of the attention mechanism.1 We perform experiments and analysis on a va- riety of sequence-to-sequence tasks where we ex- pect the alignment between source and target to be highly monotonic, such as grapheme-to-phoneme conversion, transliteration, morphological inflec- tion, and dialect normalization and compare our results to previous work that successfully applied hard monotonic attention to recurrent sequence-to- sequence models for these tasks (Wu et al., 2018a; Wu and Cotterell, 2019). Our results show that a monotonicity bias learned through a loss function …

Content truncated to 500 words