Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5286 - 5302 July 10-15, 2022 ©2022 Association for Computational Linguistics Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases Qing Lyu1 Hua Zheng2 Daoxin Li3 Li Zhang1 Marianna Apidianaki1 Chris Callison-Burch1 1Department of Computer and Information Science, University of Pennsylvania 2Key Lab of Computational Linguistics (MOE), Peking University 3Department of Linguistics, University of Pennsylvania {lyuqing,zharry,marapi,ccb}@seas.upenn.edu zhenghua@pku.edu.cn daoxinli@sas.upenn.edu Abstract Recursive noun phrases (NPs) have interest- ing semantic properties. For example, my fa- vorite new movie is not necessarily my fa- vorite movie, whereas my new favorite movie is. This is common sense to humans, yet it is unknown whether language models have such knowledge. We introduce the Recursive Noun Phrase Challenge (RNPC), a dataset of three textual inference tasks involving textual entail- ment and event plausibility comparison, pre- cisely targeting the understanding of recursive NPs. When evaluated on RNPC, state-of-the- art Transformer models only perform around chance. Still, we show that such knowledge is learnable with appropriate data. We further probe the models for relevant linguistic fea- tures that can be learned from our tasks, in- cluding modifier semantic category and mod- ifier scope. Finally, models trained on RNPC achieve strong zero-shot performance on an ex- trinsic Harm Detection evaluation task, show- ing the usefulness of the understanding of re- cursive NPs in downstream applications.1 1 Introduction Recursion, the self-embedding of a linguistic struc- ture, constitutes a fundamental property of human language. Due to its hierarchical structure, it poses many challenges to human language acquisition. One such challenge occurs in the context of recur- sive Noun Phrases (NPs), i.e., NPs with multiple prenominal modifiers. For instance, in Figure 1, when asked to point to the second green ball in a se- ries of balls, children sometimes erroneously point to the second and green ball (intersective interpreta- tion), instead of the second among green balls (re- cursive interpretation) (Matthei, 1982; Hamburger and Crain, 1984; Marcilese et al., 2013). 1Our code and data are available at https://github. com/veronica320/Recursive-NPs. Figure 1: The intersective (incorrect) and the recursive (correct) interpretation of the second green ball. We investigate whether language models (LMs) make similar errors, since the understanding of recursive NPs is also fundamental in real-world AI applications. For example, a summarization system should know that the former US president cannot be shortened as the president, since they are no longer in power. Also, a self-driving car asked to take the first left-hand exit should not assume that it is always the first exit. Previous work has studied the syntactic parsing of recursive NPs (Nakov and Hearst, 2005; Pitler et al., 2010), as well as the semantic categorization of modifiers in NPs with only one prenominal mod- ifier (Kamp and Partee, 1995; McCrae et al., 2014). However, neither parsing nor modifier categoriza- tion alone can sufficiently capture the meaning of recursive NPs (§2). In this paper, using recursive NPs …

Content truncated to 500 words