Pragmatic competence of pre-trained language models through the lens of discourse connectives

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 17–26 November 10–11, 2021. ©2021 Association for Computational Linguistics 17 Dependency Induction Through the Lens of Visual Perception Ruisi Su1 Shruti Rijhwani2 Hao Zhu2 Junxian He2 Xinyu Wang2 Yonatan Bisk2 Graham Neubig2 1Electrical and Computer Engineering, Carnegie Mellon University 2Language Technologies Institute, Carnegie Mellon University ruisis@alumni.cmu.edu, zhuhao@cmu.edu Abstract Most previous work on grammar induction focuses on learning phrasal or dependency structure purely from text. However, be- cause the signal provided by text alone is limited, recently introduced visually grounded syntax models make use of multimodal infor- mation leading to improved performance in constituency grammar induction. However, as compared to dependency grammars, con- stituency grammars do not provide a straight- forward way to incorporate visual information without enforcing language-specific heuristics. In this paper, we propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based heuristic to jointly learn constituency-structure and dependency-structure grammars. Our ex- periments find that concreteness is a strong in- dicator for learning dependency grammars, im- proving the direct attachment score (DAS) by over 50% as compared to state-of-the-art mod- els trained on pure text. Next, we propose an extension of our model that leverages both word concreteness and visual semantic role la- bels in constituency and dependency parsing. Our experiments show that the proposed exten- sion outperforms the current state-of-the-art vi- sually grounded models in constituency pars- ing even with a smaller grammar size.1 1 Introduction Grammar induction aims to discover the under- lying grammatical structure of a language from strings of symbols. Previous work has focused on two grammar formalisms: constituency and dependency grammars. Probabilistic context-free grammars (Charniak, 1996; Clark, 2001, PCFG) have been used for modeling probabilistic rules in a constituency grammar, including more recent approaches that combine PCFGs with neural mod- els (Kim et al., 2019). The dependency grammar 1Code is available at https://github.com/ ruisi-su/concrete_dep. fans | observing in progress in fans | observing observing a basketball game a observing fans Figure 1: A learned tree structure with its lexicalized heads. Solid lines represent the constituents. Arrows represent the propagation path of the lexical heads from the constituents to the root. and are the paths of the neural L-PCFG which uses pure text and our pro- posed CONCRETE L-PCFG which uses word concrete- ness, respectively. neural L-PCFG incorrectly predicts observing as the root of the sentence, while CONCRETE L-PCFG selects the correct root fans. approach to syntactic modeling allows for a binary probabilistic treatment between each word and its head. This binary relationship introduces flexibility for modeling languages with rich morphology and relatively free word order languages (Jurafsky and Martin, 2009). A commonly used model for de- pendency induction is the dependency model with valence (Klein and Manning, 2004, DMV). The two formalisms are complementary and uni- fied models have been shown to achieve higher induction accuracy than models that are trained for either constituency or dependency parsing alone. For instance, the lexicalized PCFG (Collins, 2003) is a supervised …

Content truncated to 500 words