Recursive Subtree Composition in LSTM-Based Dependency Parsing

Recursive Subtree Composition in LSTM-Based Dependency Parsing Miryam de Lhoneux♠ Miguel Ballesteros♦ Joakim Nivre♠ ♠ Department of Linguistics and Philology, Uppsala University ♦ IBM Research AI, Yorktown Heights, NY {miryam.de_lhoneux,joakim.nivre}@lingfil.uu.se miguel.ballesteros@ibm.com Abstract The need for tree structure modelling on top of sequence modelling is an open issue in neural dependency parsing. We investigate the impact of adding a tree layer on top of a sequential model by recursively compos- ing subtree representations (composition) in a transition-based parser that uses features ex- tracted by a BiLSTM. Composition seems superfluous with such a model, suggesting that BiLSTMs capture information about sub- trees. We perform model ablations to tease out the conditions under which composition helps. When ablating the backward LSTM, performance drops and composition does not recover much of the gap. When ablating the forward LSTM, performance drops less dra- matically and composition recovers a substan- tial part of the gap, indicating that a forward LSTM and composition capture similar infor- mation. We take the backward LSTM to be related to lookahead features and the forward LSTM to the rich history-based features both crucial for transition-based parsers. To capture history-based information, composition is bet- ter than a forward LSTM on its own, but it is even better to have a forward LSTM as part of a BiLSTM. We correlate results with lan- guage properties, showing that the improved lookahead of a backward LSTM is especially important for head-final languages. 1 Introduction Recursive neural networks allow us to construct vector representations of trees or subtrees. They have been used for constituency parsing by Socher et al. (2013) and Dyer et al. (2016) and for de- pendency parsing by Stenetorp (2013) and Dyer et al. (2015), among others. In particular, Dyer et al. (2015) showed that composing representa- tions of subtrees using recursive neural networks can be beneficial for transition-based dependency parsing. These results were further strengthened in Kuncoro et al. (2017) who showed, using ablation experiments, that composition is key in the Recur- rent Neural Network Grammar (RNNG) genera- tive parser by Dyer et al. (2016). In a parallel development, Kiperwasser and Goldberg (2016b) showed that using BiLSTMs for feature extraction can lead to high parsing ac- curacy even with fairly simple parsing architec- tures, and using BiLSTMs for feature extraction has therefore become very popular in dependency parsing. It is used in the state-of-the-art parser of Dozat and Manning (2017), was used in 8 of the 10 highest performing systems of the 2017 CoNLL shared task (Zeman et al., 2017) and 10 out of the 10 highest performing systems of the 2018 CoNLL shared task (Zeman et al., 2018). This raises the question of whether features ex- tracted with BiLSTMs in themselves capture in- formation about subtrees, thus making recursive composition superfluous. Some support for this hypothesis comes from the results of Linzen et al. (2016) which indicate that LSTMs can capture hi- erarchical information: they can be trained to pre- dict long-distance number agreement in English. Those results were extended to …

Content truncated to 500 words