Paragraph-based Transformer Pre-training for Multi-Sentence Inference

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2521 - 2531 July 10-15, 2022 ©2022 Association for Computational Linguistics Paragraph-based Transformer Pre-training for Multi-Sentence Inference Luca Di Liello1∗, Siddhant Garg2, Luca Soldaini3†, Alessandro Moschitti2 1University of Trento , 2Amazon Alexa AI, 3Allen Institute for AI luca.diliello@unitn.it {sidgarg,amosch}@amazon.com lucas@allenai.org Abstract Inference tasks such as answer sentence se- lection (AS2) or fact verification are typically solved by fine-tuning transformer-based mod- els as individual sentence-pair classifiers. Re- cent studies show that these tasks benefit from modeling dependencies across multiple candi- date sentences jointly. In this paper, we first show that popular pre-trained transformers per- form poorly when used for fine-tuning on multi-candidate inference tasks. We then pro- pose a new pre-training objective that models the paragraph-level semantics across multiple input sentences. Our evaluation on three AS2 and one fact verification datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks. 1 Introduction Pre-trained transformers (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020) have become the de facto standard for several NLP applications, by means of fine-tuning on downstream data. The most popular architecture uses self-attention mech- anisms for modeling long range dependencies be- tween compounds in the text, to produce deep con- textualized representations of the input. There are several downstream NLP applications that require reasoning across multiple inputs candidates jointly towards prediction. Some popular examples in- clude (i) Answer Sentence Selection (AS2) (Garg et al., 2020), which is a Question Answering (QA) task that requires selecting the best answer from a set of candidates for a question; and (ii) Fact Verifi- cation (Thorne et al., 2018), which reasons whether a claim is supported/refuted by multiple evidences. Inherently, these tasks can utilize information from multiple candidates (answers/evidences) to support the prediction of a particular candidate. ∗Work done as an intern at Amazon Alexa AI †Work completed at Amazon Alexa AI Pre-trained transformers such as BERT are used for these tasks as cross-encoders by setting them as sentence-pair classification problems, i.e, ag- gregating inferences independently over each can- didate. Recent studies (Zhang et al., 2021; Ty- moshenko and Moschitti, 2021) have shown that these tasks benefit from encoding multiple candi- dates together, e.g., encoding five answer candi- dates per question in the transformer, so that the cross-attention can model dependencies between them. However, Zhang et al. only improved over the pairwise cross-encoder by aggregating multiple pairwise cross-encoders together (one for each can- didate), and not by jointly encoding all candidates together in a single model. In this paper, we first show that popular pre- trained transformers such as RoBERTa perform poorly when used for jointly modeling inference tasks (e.g., AS2) using multi-candidates. We show that this is due to a shortcoming of their pre-training objectives, being unable to capture meaningful de- pendencies among …

Content truncated to 500 words