Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings?

Findings of the Association for Computational Linguistics: EACL 2023, pages 2306–2316 May 2-6, 2023 ©2023 Association for Computational Linguistics Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? Sonal Sannigrahi,1 Josef van Genabith,1,2 Cristina España-Bonet2 1Saarland University, Saarland Informatics Campus, Germany 2German Research Center for Artificial Intelligence (DFKI) sosa00001@stud.uni-saarland.de {cristinae, Josef.Van_Genabith}@dfki.de Abstract Dense vector representations for textual data are crucial in modern NLP. Word embeddings and sentence embeddings estimated from raw texts are key in achieving state-of-the-art results in various tasks requiring semantic understand- ing. However, obtaining embeddings at the doc- ument level is challenging due to computational requirements and lack of appropriate data. In- stead, most approaches fall back on computing document embeddings based on sentence rep- resentations. Although there exist architectures and models to encode documents fully, they are in general limited to English and few other high- resourced languages. In this work, we provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We compare input token number truncation, sentence aver- aging as well as some simple windowing and in some cases new augmented and learnable ap- proaches, on 3 multi- and cross-lingual tasks in 8 languages belonging to 3 different language families. Our task-based extrinsic evaluations show that, independently of the language, a clever combination of sentence embeddings is usually better than encoding the full document as a single unit, even when this is possible. We demonstrate that while a simple sentence aver- age results in a strong baseline for classification tasks, more complex combinations are neces- sary for semantic tasks. Our code is publicly available.1 1 Introduction Semantic representations, especially embeddings, are crucial for natural language processing (NLP). In fact, the field has exploded since the success of dense word embeddings (Mikolov et al., 2013). For some tasks like finding semantic or syntactic rela- tions among words, high quality word embeddings 1https://github.com/sonalsannigrahi/ Document_Embeddings are enough. Other tasks, like question classifica- tion or paraphrase detection, benefit from sentence embeddings. Finally, lots of tasks deal with doc- uments: summarisation, document classification, question answering, etc. Document representations are difficult to be learned, especially multilingually, given the amount of available training data and the length of each training instance. For these reasons, document embeddings usu- ally resort to sentence embeddings. Since some of the state-of-the-art techniques for language mod- elling and sentence embeddings are based on self- attention architectures such as BERT (Devlin et al., 2019), and self-attention scales quadratically with the input length, one cannot afford arbitrarily long inputs. Training is usually constrained to input frag- ments up to 512 tokens (subunits). This limit goes well beyond an average sentence length and can cover several paragraphs. However, full documents can be significantly longer. The average length of a Wikipedia article in English is 647 words (not subunits) for example,2 and the average for two of the tasks that we consider in this work, document alignment and ICD code classification, is around 800 words, with …

Content truncated to 500 words