Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3608 - 3626 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation Verna Dankers1, Christopher G. Lucas1, and Ivan Titov1,2 1ILCC, University of Edinburgh 2ILLC, University of Amsterdam vernadankers@gmail.com, {clucas2, ititov}@inf.ed.ac.uk Abstract Unlike literal expressions, idioms’ meanings do not directly follow from their parts, pos- ing a challenge for neural machine translation (NMT). NMT models are often unable to trans- late idioms accurately and over-generate com- positional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention pat- terns for models with English as source lan- guage and one of seven European languages as target language. When Transformer emits a non-literal translation – i.e. identifies the ex- pression as idiomatic – the encoder processes idioms more strongly as single lexical units compared to literal expressions. This mani- fests in idioms’ parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder’s cross-attention, figurative inputs result in re- duced attention on source-side tokens. These results suggest that Transformer’s tendency to process idioms as compositional expressions contributes to literal translations of idioms. 1 Introduction An idiom is a group of words of which the figura- tive meaning differs from the literal reading, such as “kick the bucket,” which means to die, instead of physically kicking a bucket. An idiom’s figu- rative meaning is established by convention and is typically non-compositional – i.e. the meaning cannot be computed from the meanings of the id- iom’s parts. Idioms are challenging for the task of neural machine translation (NMT) (Barreiro et al., 2013; Isabelle et al., 2017; Constant et al., 2017; Avramidis et al., 2019). On the one hand, figures of speech are ubiquitous in natural language (Colson, 2019). On the other hand, idioms occur much less frequently than their parts, their meanings need to be memorised due to the non-compositionality, context idiom translated idiom 1 2 3 4 more attention for literal more attention for figurative </s> translated context cross-attention self-attention Figure 1: How do attention patterns of figurative PIEs that are paraphrased by the model compare to atten- tion patterns of literal PIEs that are translated word for word? We find (1) decreased interaction between the PIE and its context, (2) increased attention within the PIE, (3) decreased cross-attention between the PIE and its paraphrase, (4) increased cross-attention from the paraphrase to </s>. and they require disambiguation before translation. After all, not all potentially idiomatic expressions (PIEs) are figurative – e.g. consider “When I kicked the bucket, it fell over”. Whether PIEs should re- ceive a figurative or literal translation depends on the context. Yet, little is known about neural mecha- nisms enabling idiomatic translations and methods for improving them, other than data annotation (Za- ninello and Birch, 2020). …

Content truncated to 500 words