Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2888–2899, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2888 Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study Aixiu An1 Peng Qian2 Ethan Wilcox3 Roger Levy2 1 Universit´e de Paris, LLF, CNRS, aixiu.an@etu.univ-paris-diderot.fr 2 Department of Brain and Cognitive Sciences, MIT, {pqian, rplevy}@mit.edu 3 Department of Linguistics, Harvard University, wilcoxeg@g.harvard.edu Abstract Neural language models have achieved state- of-the-art performances on many NLP tasks, and recently have been shown to learn a num- ber of hierarchically-sensitive syntactic depen- dencies between individual words. However, equally important for language processing is the ability to combine words into phrasal con- stituents, and use constituent-level features to drive downstream expectations. Here we investigate neural models’ ability to repre- sent constituent-level features, using coordi- nated noun phrases as a case study. We as- sess whether different neural language mod- els trained on English and French represent phrase-level number and gender features, and use those features to drive downstream expec- tations. Our results suggest that models use a linear combination of NP constituent num- ber to drive CoordNP/verb number agreement. This behavior is highly regular and even sensi- tive to local syntactic context, however it dif- fers crucially from observed human behavior. Models have less success with gender agree- ment. Models trained on large corpora per- form best, and there is no obvious advantage for models trained using explicit syntactic su- pervision. 1 Introduction Humans deploy structure-sensitive expectations to guide processing during natural language com- prehension (Levy, 2008). While it has been shown that neural language models show similar structure-sensitivity in their predictions about up- coming material (Linzen et al., 2016; Futrell et al., 2018), previous work has focused on dependen- cies that are conditioned by features attached to a single word, such as subject number (Gulordava et al., 2018; Marvin and Linzen, 2018) or wh- question words (Wilcox et al., 2018). There has been no systematic investigation into models’ abil- ity to compute phrase-level features—features that The star near the moon is · · · (a) The star and the moon are · · · (b) Figure 1: Subject-verb agreement with (a) the head of a noun phrase structure, and (b) the coordination struc- ture. are attached to a set of words—and whether mod- els can deploy these more abstract properties to drive downstream expectations. In this work, we assess whether state-of-the-art neural models can compute and employ phrase- level gender and number features of coordinated subject Noun Phrases (CoordNPs) with two nouns. Typical syntactic phrases are ENDOCENTRIC: they are HEADED by a single child, whose features de- termine the agreement requirements for the entire phrase. In Figure 1a, for example, the word star heads the subject NP The star; since star is sin- gular, the verb must be singular. CoordNPs lack endocentricity: neither conjunct NP solely deter- mines the features of the NP as …

Content truncated to 500 words