Subword Pooling Makes a Difference

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2284–2295 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2284 Subword Pooling Makes a Difference Judit ´Acs1,3 Budapest University of Technology and Economics1 judit@aut.bme.hu ´Akos K´ad´ar2 Borealis AI2 akos.kadar@ borealisai.com Andr´as Kornai1,3 Institute for Computer Science and Control3 andras@kornai.com Abstract Contextual word-representations became a standard in modern natural language process- ing systems. These models use subword tok- enization to handle large vocabularies and un- known words. Word-level usage of such sys- tems requires a way of pooling multiple sub- words that correspond to a single word. In this paper we investigate how the choice of subword pooling affects the downstream per- formance on three tasks: morphological prob- ing, POS tagging and NER, in 9 typologically diverse languages. We compare these in two massively multilingual models, mBERT and XLM-RoBERTa. For morphological tasks, the widely used ‘choose the first subword’ is the worst strategy and the best results are obtained by using attention over the subwords. For POS tagging both of these strategies perform poorly and the best choice is to use a small LSTM over the subwords. The same strat- egy works best for NER and we show that mBERT is better than XLM-RoBERTa in all 9 languages. We publicly release all code, data and the full result tables at https://github. com/juditacs/subword-choice. 1 Introduction Training of contextual language models on large training corpora generally begins with segmenting the input into subwords (Schuster and Nakajima, 2012) to reduce the vocabulary size. Since most tasks consume full words, practitioners have the freedom to decide whether to use the first, the last, or some combination of all subwords. The origi- nal paper introducing BERT, Devlin et al. (2019), suggests using the first subword for named entity recognition (NER), and did not explore different poolings. Kondratyuk and Straka (2019) also use the first subword, for dependency parsing, and re- mark in a footnote that they tried the first, last, average, and max pooling but the choice made no difference. Kitaev et al. (2019) report similar find- ings for constituency parsing, but nevertheless opt for reporting results only using the last subword. Hewitt and Manning (2019) take the average of the subword vectors for syntactic and word sense dis- ambiguation tasks. Wu et al. (2020) use attentive pooling with a trainable norm for news topic clas- sification and sentiment analysis in English. Shen et al. (2018) use hierarchical pooling for sequence classification tasks in English and Chinese. Here we show that for word-level tasks (morpho- logical, POS and NER tagging), particularly for languages where the proportion of multi-subword tokens (i.e. those word tokens that are split into more than one subword) is high, more care needs to be taken as both pooling strategy, and that the choice of language matters. We demonstrate this clearly for European languages with rich morphol- ogy, and in Chinese, Japanese and Korean (CJK). Similar to subword pooling, the choice of the low- est …

Content truncated to 500 words