From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5614 - 5633 May 22-27, 2022 c⃝2022 Association for Computational Linguistics From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology Mark Dingemanse ∗ Center for Language Studies Radboud University mark.dingemanse@ru.nl Andreas Liesenfeld ∗ Center for Language Studies Radboud University andreas.liesenfeld@ru.nl Abstract Informal social interaction is the primordial home of human language. Linguistically di- verse conversational corpora are an impor- tant and largely untapped resource for com- putational linguistics and language technology. Through the efforts of a worldwide language documentation movement, such corpora are in- creasingly becoming available. We show how interactional data from 63 languages (26 fam- ilies) harbours insights about turn-taking, tim- ing, sequential structure and social action, with implications for language technology, natural language understanding, and the design of con- versational interfaces. Harnessing linguistically diverse conversational corpora will provide the empirical foundations for flexible, localizable, humane language technologies of the future. 1 The natural habitat of language The primary ecology of natural language is in real- life episodes of human interaction. This is where people learn language and where they use it to co- ordinate joint actions, build social relations, and exchange information (Schieffelin and Ochs, 1986; Schegloff, 2006). In contrast, when machines en- counter language, it tends to be radically divorced from this habitat and reduced to large amounts of decontextualised non-interactive text (Bender and Koller, 2020; Marge et al., 2022). Natural languages are also characterized by diversity at many levels, from sound and sign systems to syntax and semantics (Nettle, 1999; Evans and Levinson, 2009). In contrast, the language samples that in- form language technology tend to be limited to a handful of well-resourced languages, representing only a tiny sliver of the world’s linguistic diversity (Blasi et al., 2021; Joshi et al., 2020). The time is ripe for language technology to ben- efit from linguistically diverse interactional data. ∗Both authors contributed equally. Insights from such data can strengthen the empiri- cal foundations of language technology, help break down the hegemony of the resourceful few, and provide room for linguistic diversity in localized applications (Bird, 2020; Danielescu and Christian, 2018). Today there is a growing set of conversa- tional corpora of diverse languages, thanks in large part to important primary work on language doc- umentation and description (Seifart et al., 2018). We argue such corpora represent an important and mostly untapped resource for language technology. Corpus size is often seen as a challenge, but data comes in levels of granularity. A well-curated corpus amounting to an hour of lively conversa- tion may not contain enough text to train a lan- guage model. But it does provide thousands of conversational turns organized in larger sequential structures of social action, along with fine details about timing, participation and linguistic structure. Since conversational corpora are one of the few places where we can study language in a way that approaches its natural habitat, collectively, these corpora harbour important insights …

Content truncated to 500 words