Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3306–3316 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3306 Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs Michael A. Lepori1 Tal Linzen1,2 R. Thomas McCoy2 1Department of Computer Science 2Department of Cognitive Science Johns Hopkins University {mlepori1, tal.linzen, tom.mccoy}@jhu.edu Abstract Sequence-based neural networks show signifi- cant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based net- works can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependency- based one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially im- proved by fine-tuning on a small amount of constructed data, suggesting that data augmen- tation is a viable alternative to explicit con- stituency structure for imparting the syntactic biases that sequential models are lacking. 1 Introduction Natural language syntax is structured hierarchi- cally, rather than sequentially (Chomsky, 1957; Everaert et al., 2015). One phenomenon that il- lustrates this fact is English subject-verb agree- ment, the requirement that verbs and their subjects must match in number. The hierarchical structure of a sentence determines which noun phrase each verb must agree with; sequential heuristics such as agreeing with the most recent noun may succeed on simple sentences such as (1a) but fail in more complex cases such as (1b): (1) a. The boys kick the ball. b. The boys by the red truck kick the ball. We investigate whether a neural network must pro- cess input according to the structure of a syntac- tic parse in order for it to learn the appropriate No Constituency Constituency No Heads BiLSTM Constituency LSTM Heads Dependency LSTM Head-Lexicalized LSTM Table 1: Linguistic properties of our four models. rules governing these dependencies, or whether there is sufficient signal in natural language corpora for low-bias networks (such as sequential LSTMs) to learn these structures. We compare sequential LSTMs, which process sentences from left to right, with tree-based LSTMs that process sentences in accordance with an externally-provided, ground- truth syntactic structure. We consider two types of syntactic structure: constituency structure (Chomsky, 1993; Pollard and Sag, 1994) and dependency structure (Tes- niere, 1959; Hudson, 1984). We investigate models provided with either structure, both structures, or neither structure (see Table 1), and assess how ro- bustly these models learn subject-verb agreement when trained on natural language.1 Even with the syntactic biases present in tree- based LSTMs, it is possible that natural language might not impart a strong enough signal to teach a network how to robustly track subject-verb de- pendencies. How might the performance of these tree-based LSTMs change if they were fine-tuned on a …

Content truncated to 500 words