ConveRT: Efficient and Accurate Conversational Representations from Transformers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2161–2174 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2161 ConveRT: Efficient and Accurate Conversational Representations from Transformers Matthew Henderson, I˜nigo Casanueva, Nikola Mrkˇsi´c, Pei-Hao Su, Tsung-Hsien Wen and Ivan Vuli´c matt@poly-ai.com PolyAI Limited, London, UK Abstract General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are com- putationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pre- training framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response se- lection task, effectively leveraging quantiza- tion and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that Con- veRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of ex- tended dialog history as context yields further performance gains. Finally, we show that pre- trained representations from the proposed en- coder can be transferred to the intent classifi- cation task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or pre- vious state-of-the-art dual encoders. With its reduced size and superior performance, we be- lieve this model promises wider portability and scalability for Conversational AI applications. 1 Introduction Dialog systems, also referred to as conversational systems or conversational agents, have found use in a wide range of applications. They assist users in accomplishing well-defined tasks such as find- ing and booking restaurants, hotels, and flights (Hemphill et al., 1990; Williams, 2012; El Asri et al., 2017), with further use in tourist informa- tion (Budzianowski et al., 2018), language learning (Raux et al., 2003; Chen et al., 2017), entertainment (Fraser et al., 2018), and healthcare (Laranjo et al., 2018; Fadhil and Schiavo, 2019). They are also key components of intelligent virtual assistants such as Siri, Alexa, Cortana, and Google Assistant. Data-driven task-oriented dialog systems require domain-specific labelled data: annotations for in- tents, explicit dialog states, and mentioned entities (Williams, 2014; Wen et al., 2017b,a; Ramadan et al., 2018; Liu et al., 2018; Zhao et al., 2019b). This makes the scaling and maintenance of such systems very challenging. Transfer learning on top of pretrained models (Devlin et al., 2019; Liu et al., 2019, inter alia) provides one avenue for reduc- ing the amount of annotated data required to train models capable of generalization. Pretrained models making use of language- model (LM) based learning objectives have be- come prevalent across the NLP research commu- nity. When it comes to dialog systems, response selection provides a more suitable pretraining task for learning representations that can encapsulate conversational cues. Such models can be pretrained using large corpora of natural unlabelled conversa- tional data (Henderson et al., 2019b; Mehri et al., 2019). Response selection is also directly appli- cable to retrieval-based dialog systems, a popular and elegant approach to framing dialog (Wu et al., 2017; Weston et al., …

Content truncated to 500 words