Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1418–1427 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1418 Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings Rob van der Goot,♣ Max M¨uller-Eberstein,♣ Barbara Plank♣♦ ♣Computer Science Department, IT University of Copenhagen ♦Center for Information and Language Processing (CIS), LMU Munich, Germany robv@itu.dk, mamy@itu.dk, bapl@itu.dk Abstract As input representation for each sub-word, the original BERT architecture proposes the sum of the sub-word embedding, position embedding and a segment embedding. Sub-word and position embeddings are well-known and studied, and encode lexical information and word position, respectively. In contrast, segment embeddings are less known and have so far received no attention. The key idea of segment embeddings is to encode to which of the two sentences (segments) a word belongs—the intuition is to inform the model about the separation of sentences for the next sentence prediction pre-training task. However, little is known on whether the choice of segment impacts downstream prediction performance. In this work, we try to fill this gap and empirically study the impact of alternating the segment embedding during inference time for a variety of pre-trained embeddings and target tasks. We hypothesize that for single-sentence prediction tasks performance is not affected—neither in mono- nor multilingual setups—while it matters when changing the segment IDs in paired-sentence tasks. To our surprise, this is not the case. Although for classification tasks and monolingual BERT models no large differences are observed, particularly word-level multilingual prediction tasks are heavily impacted. For low-resource syntactic tasks, we observe impacts of segment embedding and multilingual BERT choice. We find that the default setting for the most used multilingual BERT model underperforms heavily, and a simple swap of the segment embeddings yields an average improvement of 2.5 points absolute LAS score for dependency parsing over 9 different treebanks. Keywords: multilingual representations, BERT embeddings, segment embeddings, parsing, classification 1. Introduction Transformer-based contextualized embeddings have led to an increase in performance on countless Natu- ral Language Processing (NLP) tasks. However, un- derstanding why they work, and what is encoded in their embeddings, is still an active area of research also coined “BERTology” (Rogers et al., 2020) for the most popular type of contextualized embeddings based on BERT (Devlin et al., 2019). We refer to Rogers et al. (2020) for an overview of recent work studying the in- ner workings of BERT and focus on a so-far ignored part of the architecture: segment embeddings. In BERT, the input text is first tokenized using a basic punctuation/whitespace tokenizer, and then split into sub-words which stem from a vocabulary of word- pieces (Wu et al., 2016). These sub-words are then encoded as the sum of three distinct embeddings (see also Figure 1): 1) sub-word embeddings, 2) positional embeddings, 3) segment embeddings. Compared to the sub-word and positional embeddings, the intuition be- hind segment embeddings is less obvious, especially for unsegmented (i.e. single sentence) input (detailed in Section 2.1). …

Content truncated to 500 words