The (Non-)Utility of Structural Features in BiLSTM-based Dependency Parsers

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 117–128 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 117 The (Non-)Utility of Structural Features in BiLSTM-based Dependency Parsers Agnieszka Falenska and Jonas Kuhn Institut f¨ur Maschinelle Sprachverarbeitung University of Stuttgart name.surname@ims.uni-stuttgart.de Abstract Classical non-neural dependency parsers put considerable effort on the design of feature functions. Especially, they benefit from infor- mation coming from structural features, such as features drawn from neighboring tokens in the dependency tree. In contrast, their BiLSTM-based successors achieve state-of- the-art performance without explicit informa- tion about the structural context. In this pa- per we aim to answer the question: How much structural context are the BiLSTM representa- tions able to capture implicitly? We show that features drawn from partial subtrees become redundant when the BiLSTMs are used. We provide a deep insight into information flow in transition- and graph-based neural architec- tures to demonstrate where the implicit infor- mation comes from when the parsers make their decisions. Finally, with model ablations we demonstrate that the structural context is not only present in the models, but it signifi- cantly influences their performance. 1 Introduction When designing a conventional non-neural parser substantial effort is required to design a power- ful feature extraction function. Such a function (McDonald et al., 2005; Zhang and Nivre, 2011, among others) is constructed so that it captures as much structural context as possible. The con- text allows the parser to make well-informed deci- sions.1 It is encoded in features built from partial subtrees and explicitly used by the models. Recently, Kiperwasser and Goldberg (2016, K&G) showed that the conventional feature ex- traction functions can be replaced by modeling the left- and right-context of each word with BiLSTMs (Hochreiter and Schmidhuber, 1997; 1See Figure 1 for the concept of structural context, details of the architectures will be described in Section 2. Graves and Schmidhuber, 2005). Although the proposed models do not use any conventional structural features they achieve state-of-the-art performance. The authors suggested that it is be- cause the BiLSTM encoding is able to estimate the missing information from the given features and did not explore this issue further. Since the introduction of the K&G architecture BiLSTM-based parsers have become standard in the field.2 Yet, it is an open question how much conventional structural context the BiLSTMs rep- resentations actually are able to capture implicitly. Small architectures that ignore the structural con- text are attractive since they come with lower time complexity. But to build such architectures it is important to investigate to what extent the explicit structural information is redundant. For example, K&G also proposed an extended feature set de- rived from structural context, which has subse- quently been re-implemented and used by others without questioning its utility. Inspired by recent work (Gaddy et al., 2018) on constituency parsing we aim at understanding what type of information is captured by the inter- nal representations of BiLSTM-based dependency parsers and how it translates into their …

Content truncated to 500 words