The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks

Proceedings of the 28th International Conference on Computational Linguistics, pages 3652–3659 Barcelona, Spain (Online), December 8-13, 2020 3652 The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks Brihi Joshi† Neil Shah♣ Francesco Barbieri♣ Leonardo Neves♣ †Indraprastha Institute of Information Technology Delhi, New Delhi, India ♣Snap Inc., Santa Monica, CA 90405, USA †brihi16142@iiitd.ac.in, ♣{nshah,fbarbieri,lneves}@snap.com, Abstract Contextual embeddings derived from transformer-based neural language models have shown state-of-the-art performance for various tasks such as question answering, sentiment analysis, and textual similarity in recent years. Extensive work shows how accurately such models can represent abstract, semantic information present in text. In this expository work, we explore a tangent direction and analyze such models’ performance on tasks that require a more granular level of representation. We focus on the problem of textual similarity from two perspectives: matching documents on a granular level (requiring embeddings to capture fine-grained attributes in the text), and an abstract level (requiring embeddings to capture overall textual semantics). We empirically demonstrate, across two datasets from different domains, that despite high per- formance in abstract document matching as expected, contextual embeddings are consistently (and at times, vastly) outperformed by simple baselines like TF-IDF for more granular tasks. We then propose a simple but effective method to incorporate TF-IDF into models that use contextual embeddings, achieving relative improvements of up to 36% on granular tasks. 1 Introduction In recent years, contextual embeddings (Peters et al., 2018; Devlin et al., 2018) have made immense progress in semantic understanding-based tasks. After being trained using large amounts of data, for example via a self-supervised task like masked language-modeling, such models learn crucial elements of language, such as syntax and semantics (Jawahar et al., 2019; Goldberg, 2019; Wiedemann et al., 2019) from just raw text. The best performing contextual embeddings are trained with Transformer- based methods (TBM) (Vaswani et al., 2017; Devlin et al., 2018). These embeddings have been shown to frequently achieve state-of-the-art results in downstream tasks like question answering and sentiment analysis (van Aken et al., 2019; Sun et al., 2019). Contextual embeddings are also often used to capture the similarity between pairs of documents; for example, on the Semantic Textual Similarity (STS) task (Cer et al., 2017) included in the GLUE benchmark (Wang et al., 2018), TBMs have shown competitive performance, substantially outperforming embedding baselines like Word2Vec (Mikolov et al., 2013) and GloVE (Pennington et al., 2014). However, their performance on similarity tasks beyond abstract, semantic ones (Mickus et al., 2019) – for example, on granular news article matching – is less understood. In this work, we study the performance of TBMs in textual similarity tasks with the following research question: Are transformer-based methods as performant for granular tasks as they are for abstract ones? Here, granular and abstract reflect varying amounts of coarseness in the concept of similarity. For example, consider the news domain: A granular notion of similarity might be whether a pair of articles both report the exact same news event. Conversely, an abstract notion might …

Content truncated to 500 words