Self-Attentional Models for Lattice Inputs

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1185–1197 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1185 Self-Attentional Models for Lattice Inputs Matthias Sperber1, Graham Neubig2, Ngoc-Quan Pham1, Alex Waibel1,2 1Karlsruhe Institute of Technology, Germany 2Carnegie Mellon University, USA {first}.{last}@kit.edu, gneubig@cs.cmu.edu Abstract Lattices are an efficient and effective method to encode ambiguity of upstream systems in natural language processing tasks, for example to compactly capture multiple speech recogni- tion hypotheses, or to represent multiple lin- guistic analyses. Previous work has extended recurrent neural networks to model lattice in- puts and achieved improvements in various tasks, but these models suffer from very slow computation speeds. This paper extends the recently proposed paradigm of self-attention to handle lattice inputs. Self-attention is a se- quence modeling technique that relates inputs to one another by computing pairwise simi- larities and has gained popularity for both its strong results and its computational efficiency. To extend such models to handle lattices, we introduce probabilistic reachability masks that incorporate lattice structure into the model and support lattice scores if available. We also pro- pose a method for adapting positional embed- dings to lattice structures. We apply the pro- posed model to a speech translation task and find that it outperforms all examined baselines while being much faster to compute than pre- vious neural lattice models during both train- ing and inference. 1 Introduction In many natural language processing tasks, graph- based representations have proven useful tools to enable models to deal with highly structured knowledge. Lattices are a common instance of graph-based representations that allows capturing a large number of alternative sequences in a com- pact form (Figure 1). Example applications in- clude speech recognition lattices that represent al- ternative decoding choices (Saleem et al., 2004; Zhang et al., 2005; Matusov et al., 2008), word segmentation lattices that capture ambiguous de- cisions on word boundaries or morphological al- ternatives (Dyer et al., 2008), word class lattices f a S c E d b e g 1 0.4 1 1 1 0.6 0.2 0.8 1 e a S E c b d 1 0.4 1 1 0.6 0.2 0.8 e a S c b d 1 0.45 0.88 1 0.55 0.12 1 Figure 1: Example of a node-labeled lattice. Nodes are labeled with word tokens and posterior scores. (Navigli and Velardi, 2010), and lattices for alter- native video descriptions (Senina et al., 2014). Prior work has made it possible to handle these through the use of recurrent neural network (RNN) lattice representations (Ladhak et al., 2016; Su et al., 2017; Sperber et al., 2017), inspired by earlier works that extended RNNs to tree struc- tures (Socher et al., 2013; Tai et al., 2015; Zhu et al., 2015). Unfortunately, these models are computationally expensive, because the extension of the already slow RNNs to tree-structured in- puts prevents convenient use of batched compu- tation. An alternative model, graph convolutional networks (GCN) (Duvenaud et al., 2015; Deffer- rard et al., …

Content truncated to 500 words