What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4403 - 4417 July 10-15, 2022 ©2022 Association for Computational Linguistics What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris Mikael Brunila McGill University Department of Geography mikael.brunila@gmail.com Jack LaViolette Columbia University Department of Sociology jack.laviolette@columbia.edu Abstract The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts. This idea is specif- ically invoked by noting that “you shall know a word by the company it keeps,” a quote from British linguist J.R. Firth who, along with his American colleague Zellig Harris, is often cred- ited with the invention of “distributional se- mantics.” While both Firth and Harris are cited in all major NLP textbooks and many founda- tional papers, the content and differences be- tween their theories is seldom discussed. En- gaging in a close reading of their work, we dis- cover two distinct and in many ways divergent theories of meaning. One focuses exclusively on the internal workings of linguistic forms, while the other invites us to consider words in new company—not just with other linguistic elements, but also in a broader cultural and situ- ational context. Contrasting these theories from the perspective of current debates in NLP, we discover in Firth a figure who could guide the field towards a more culturally grounded notion of semantics. We consider how an expanded no- tion of “context” might be modeled in practice through two different strategies: comparative stratification and syntagmatic extension. 1 Introduction We are in the world and the world is in us. Alfred North Whitehead (1938; cited in Firth 1957c, 29) If you have read any papers in computational linguistics in the past thirty years, you have likely come upon the following quote from British lin- guist J.R.Firth (1957c, 11): “You shall know a word by the company it keeps”. Cited in most major textbooks (Manning and Schütze, 1999; Ju- rafsky and Martin, 2009; Eisenstein, 2019; Russell and Norvig, 2020), several foundational papers, and hundreds of other NLP articles, this phrase has come to index a theoretical orientation in a field that is increasingly focused on computation, often at the expense of linguistic theory (on these trends in NLP see Halevy et al., 2009; Manning, 2015; Norvig, 2012; Henderson, 2020; Church and Liberman, 2021). Together with American linguist Zellig Harris, Firth is regularly called upon to jus- tify a distributional theory of semantics, whereby the meaning of lexical units is conceived in terms of relative co-occurrence and shared contexts of use (Sahlgren, 2008). While Harris and Firth are often invoked, their ideas are seldom closely engaged. Hailing from disparate traditions, Harris and Firth had radically different ideas on the scope and context of linguis- tic analysis, and presented incongruent versions of the distributional method. Drawing on the informa- tion theory pioneered by Claude Shannon (1948), Harris was determined to work out a structuralist …

Content truncated to 500 words