Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension Najoung Kim†,∗, Roma Patelφ, Adam Poliak†, Alex Wang∂, Patrick Xia†, R. Thomas McCoy†, Ian Tenney∆, Alexis Ross⋄, Tal Linzen†, Benjamin Van Durme†, Samuel R. Bowman∂, Ellie Pavlickφ,∗ †Johns Hopkins University φBrown University ∂New York University ∆Google AI Language ⋄Harvard University Abstract We introduce a set of nine challenge tasks that test for the understanding of function words. These tasks are created by structurally mutat- ing sentences from existing datasets to target the comprehension of specific types of func- tion words (e.g., prepositions, wh-words). Us- ing these probing tasks, we explore the ef- fects of various pretraining objectives for sen- tence encoders (e.g., language modeling, CCG supertagging and natural language inference (NLI)) on the learned representations. Our re- sults show that pretraining on language mod- eling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably. Overall, no pretraining objective dominates across the board, and our function word probing tasks highlight several intuitive differences between pretraining ob- jectives, e.g., that NLI helps the comprehen- sion of negation. 1 Introduction Many recent advances in NLP have been driven by new approaches to representation learning— i.e., the design of models whose primary aim is to yield representations of words or sentences that are useful for a range of downstream applications (Bowman et al., 2017). Approaches to representa- tion learning typically differ in the architecture of the model used to learn the representations, the ob- jective used to train that network, or both. Varying these factors can significantly impact performance on a broad range of NLP tasks (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019). This paper investigates the role of pretraining objectives of sentence encoders, with respect to ∗Corresponding authors: Najoung Kim (n.kim@jhu.edu), Ellie Pavlick (ellie pavlick@brown.edu) their capacity to understand function words (e.g., prepositions, conjunctions). Although the impor- tance of finding an effective pretraining objective for learning better (or more generalizable) repre- sentations is well acknowledged, relatively few studies offer a controlled comparison of diverse pretraining objectives, holding model architecture constant. We ask whether the linguistic properties im- plicitly captured by pretraining objectives measur- ably affect the types of linguistic information en- coded in the learned representations. To this end, we explore whether qualitatively different objec- tives lead to demonstrably different sentence rep- resentations. We focus our analysis on function words because they play a key role in composi- tional meaning—e.g., introducing and identifying discourse referents or representing relationships between entities or ideas—and are not yet con- sidered to be well-modeled by distributional se- mantics (Bernardi et al., 2015). Our results sug- gest that different pretraining objectives give rise to differences in function word comprehension; for instance, we see that natural language infer- ence helps understanding negation, and CCG su- pertagging helps recognizing meaningful sentence boundaries. However, overall, we find that the observed differences are not always straightfor- wardly interpretable, and …

Content truncated to 500 words