Seeing Both the Forest and the Trees: Multi-head Attention for Joint Classification on Different Compositional Levels

Proceedings of the 28th International Conference on Computational Linguistics, pages 3761–3775 Barcelona, Spain (Online), December 8-13, 2020 3761 Seeing Both the Forest and the Trees: Multi-head Attention for Joint Classification on Different Compositional Levels Miruna Pˆıslar♠ Marek Rei♣♦ ♠Dept. of Computer Science, University of Cambridge, UK ♣Dept. of Computing, Imperial College London, UK ♦The ALTA Institute, Dept. of Computer Science, University of Cambridge, UK miruna.pislar@gmail.com, marek.rei@imperial.ac.uk Abstract In natural languages, words are used in association to construct sentences. It is not words in iso- lation, but the appropriate combination of hierarchical structures that conveys the meaning of the whole sentence. Neural networks can capture expressive language features; however, insights into the link between words and sentences are difficult to acquire automatically. In this work, we design a deep neural network architecture that explicitly wires lower and higher linguistic components; we then evaluate its ability to perform the same task at different hierarchical levels. Settling on broad text classification tasks, we show that our model, MHAL, learns to simulta- neously solve them at different levels of granularity by fluidly transferring knowledge between hierarchies. Using a multi-head attention mechanism to tie the representations between single words and full sentences, MHAL systematically outperforms equivalent models that are not in- centivized towards developing compositional representations. Moreover, we demonstrate that, with the proposed architecture, the sentence information flows naturally to individual words, al- lowing the model to behave like a sequence labeller (which is a lower, word-level task) even without any word supervision, in a zero-shot fashion. 1 Introduction Compositional reasoning is fundamental in human cognition: we use it to interact with objects, take actions, reason about numbers, and move in space (Spelke and Kinzler, 2007). This is also reflected in some aspects of the human language (Wagner et al., 2011; Piantadosi and Aslin, 2016; Sandler, 2018) since we use words and phrases in association to construct sentences. Consequently, there are lower linguistic components that act as building blocks for higher levels. In this work, we focus on two levels of the compositional hierarchy – words and sentences – and ask the following question: are deep neural networks (DNNs) trained for a higher-level task (i.e., at the sentence level) able to pick up the features of the compounds needed to solve the same task but at a lower level, such as at the word level? Moreover, how are the different hierarchical levels interacting under varying supervision signals? To the best of our knowledge, very few studies have investigated the transferability of a task solution between compositional levels using DNNs in controlled experiments. It has been shown that neural networks are universal function approximators (Hornik, 1991; Leshno et al., 1993); they can perform arbitrary function combinations to learn expressive features. DNNs trained for language tasks are not an exception to this rule, and recent studies have shown their power in extracting linguistically-rich representations (Mikolov et al., 2013; Devlin et al., 2018). However, when trained end-to-end, learning the connection between the different compositional levels is not trivial …

Content truncated to 500 words