Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273–291 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 273 Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation Benjamin Heinzerling†∗ and Michael Strube‡ †RIKEN AIP & Tohoku University ‡Heidelberg Institute for Theoretical Studies gGmbH benjamin.heinzerling@riken.jp | michael.strube@h-its.org Abstract Pretrained contextual and non-contextual sub- word embeddings have become available in over 250 languages, allowing massively mul- tilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it diffi- cult for practitioners to choose between them. In this work, we conduct an extensive evalu- ation comparing non-contextual subword em- beddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recogni- tion and part-of-speech tagging. We find that overall, a combination of BERT, BPEmb, and character representations works well across languages and tasks. A more detailed analysis reveals different strengths and weaknesses: Multilingual BERT performs well in medium- to high-resource languages, but is outperformed by non-contextual sub- word embeddings in a low-resource setting. 1 Introduction Rare and unknown words pose a difficult chal- lenge for embedding methods that rely on seeing a word frequently during training (Bullinaria and Levy, 2007; Luong et al., 2013). Subword seg- mentation methods avoid this problem by assum- ing a word’s meaning can be inferred from the meaning of its parts. Linguistically motivated sub- word approaches first split words into morphemes and then represent word meaning by composing morpheme embeddings (Luong et al., 2013). More recently, character-ngram approaches (Luong and Manning, 2016; Bojanowski et al., 2017) and Byte Pair Encoding (BPE) (Sennrich et al., 2016) have grown in popularity, likely due to their computa- tional simplicity and language-agnosticity.1 ∗ Work done while at HITS. 1While language-agnostic, these approaches are not language-independent. See Appendix B for a discussion. _magn us _car ls en _played M a g n u s charRNN C a r l s e n charRNN p l a y e d charRNN Magnus Carl ##sen played Multilingual BERT Magnus Carlsen played B-PER I-PER O BPEmb Figure 1: A high-performing ensemble of subword representations encodes the input using multilingual BERT (yellow, bottom left), an LSTM with BPEmb (pink, bottom middle), and a character-RNN (blue, bot- tom right). A meta-LSTM (green, center) combines the different encodings before classification (top). Hori- zontal arrows symbolize bidirectional LSTMs. Sequence tagging with subwords. Subword in- formation has long been recognized as an im- portant feature in sequence tagging tasks such as named entity recognition (NER) and part-of- speech (POS) tagging. For example, the suffix -ly often indicates adverbs in English POS tag- ging and English NER may exploit that profes- sions often end in suffixes like -ist (journalist, cyclist) or companies in suffixes like -tech or - soft. In early systems, these observations were operationalized with manually compiled lists of such word endings or with character-ngram fea- tures (Nadeau and Sekine, 2007). Since the ad- vent …

Content truncated to 500 words