MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2471–2486, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2471 MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale Andreas R¨uckl´e and Jonas Pfeiffer and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science, Technische Universit¨at Darmstadt www.ukp.tu-darmstadt.de Abstract We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source do- mains from community question answering fo- rums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surpris- ingly well, where the large majority of mod- els substantially outperforms common IR base- lines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer per- formances, which contrasts the standard pro- cedure that merely relies on the largest and most similar domains. In addition, we ex- tensively study how to best combine multi- ple source domains. We propose to incorpo- rate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably out- performs in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in ad- ditional large gains and achieves the new state of the art on all nine benchmarks. 1 Introduction Semantic matching of two text sequences is cru- cial among a wide range of NLP problems, such as question answering (Nakov et al., 2017; Wang and Jiang, 2017) or semantic textual similarity (Cer et al., 2017). Due to the ubiquity of applications, it is crucial to study how to obtain re-usable text matching models that transfer well to unseen do- mains or tasks. Zero-shot transfer of text matching models is particularly challenging in setups of non-factoid answer selection (Cohen et al., 2018; Tay et al., 2017; Feng et al., 2015; Verberne et al., 2010) and question similarity (Nakov et al., 2017; Lei et al., 2016). These tasks compare questions and answers, or two potentially related questions in community question answering (cQA) forums, FAQ pages, and general collections of text passages. In contrast to other text matching tasks in NLP, they compare texts of different lengths—e.g., answers can be long explanations or descriptions—and often deal with expert domains. This makes it difficult to transfer models across domains (Shah et al., 2018), and to apply common approaches such as universal sentence embeddings without further domain or task adaptations (Poerner and Sch¨utze, 2019). Non-factoid answer selection and question simi- larity are also particularly promising to study zero- shot transfer. Reasons are that (1) there exist a large number of domains, and (2) in-domain training data is often scarce. Previous work proposed do- main adaptation techniques (Poerner and Sch¨utze, 2019; Shah et al., 2018), training with unlabeled data (R¨uckl´e et al., 2019b), and shallow architec- tures (R¨uckl´e et al., 2019a). However, these ap- proaches result in …

Content truncated to 500 words