Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3851–3860 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3851 Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction Danushka Bollegala∗,†, Ryuichi Kiryo†, Kosuke Tsujino†, Haruki Yukawa† University of Liverpool∗, Amazon† {danubol,kiryor,kosutu,hyukawa}@amazon.com Abstract Language-independent tokenisation (LIT) methods that do not require labelled language resources or lexicons have recently gained popularity because of their applicability in resource-poor languages. Moreover, they compactly represent a language using a fixed size vocabulary and can efficiently handle unseen or rare words. On the other hand, language-specific tokenisation (LST) methods have a long and established history, and are developed using carefully created lexicons and training resources. Unlike subtokens produced by LIT methods, LST methods produce valid morphological subwords. Despite the contrasting trade-offs between LIT vs. LST methods, their performance on downstream NLP tasks remain unclear. In this paper, we empirically compare the two approaches using semantic similarity measurement as an evaluation task across a diverse set of languages. Our experimental results covering eight languages show that LST consistently outperforms LIT when the vocabulary size is large, but LIT can produce comparable or better results than LST in many languages with comparatively smaller (i.e. less than 100K words) vocabulary sizes, encouraging the use of LIT when language-specific resources are unavailable, incomplete or a smaller model is required. Moreover, we find that smoothed inverse frequency (SIF) to be an accurate method to create word embeddings from subword embeddings for multilingual semantic similarity prediction tasks. Further analysis of the nearest neighbours of tokens show that semantically and syntactically related tokens are closely embedded in subword embedding spaces. Keywords: Subtokenisation, Byte Pair Encoding, Language Independent Tokenisation 1. Introduction One of the first steps in many NLP pipelines is tokenisation – the process of splitting a given text into a sequence of continuous lexical units for the purpose of representing the given text. Tokenisation can be performed at various gran- ularities such as at phrase-level, word-level, sub-word level or character-level, considering the level of textual represen- tation required for a particular task (Riedl and Biemann, 2018). For example, for information retrieval, we must en- sure that both documents and user queries are tokenised in a consistent manner considering the relevance of the search results. In specialised domains such as biomedical, proper tokenisation can significantly improve the retrieval accu- racy by up to 80% (Jiang and Zhai, 2007). Exceedingly finer tokenisation is likely to return many ir- relevant results with incorrect or partial matches, whereas not tokenising larger phrases will return zero results. In this paper, we use the term token to refer to both words as well as subwords, which might not necessarily be morphological units but character n-grams. For example, given the string “Hello world”, where “ ” denotes the space character, a possible sequence of subtokens could be H/el/l/o/ /world. As can be seen from this example, some of the subtokens such as H, el,, l are not valid English words, …

Content truncated to 500 words