Second-order contexts from lexical substitutes for few-shot learning of word representations

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 61–67 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 61 Second-order contexts from lexical substitutes for few-shot learning of word representations Qianchu Liu, Diana McCarthy, Anna Korhonen Language Technology Lab, University of Cambridge English Faculty Building, 9 West Road, Cambridge CB3 9DA, United Kingdom ql261@cam.ac.uk, diana@dianamccarthy.co.uk, alk23@cam.ac.uk Abstract There is a growing awareness of the need to handle rare and unseen words in word rep- resentation modelling. In this paper, we fo- cus on few-shot learning of emerging concepts that fully exploits only a few available con- texts. We introduce a substitute-based context representation technique that can be applied on an existing word embedding space. Pre- vious context-based approaches to modelling unseen words only consider bag-of-word first- order contexts, whereas our method aggre- gates contexts as second-order substitutes that are produced by a sequence-aware sentence completion model. We experimented with three tasks that aim to test the modelling of emerging concepts. We found that these tasks show different emphasis on first and second or- der contexts, and our substitute-based method achieved superior performance on naturally- occurring contexts from corpora. 1 Introduction As language vocabulary follows the zipfian dis- tribution, we expect to encounter a large number of rare and unseen words no matter how large the training corpus is. The effective handling of such words is thus crucial for Natural Language Pro- cessing (NLP). Attempts to learn rare and unseen word repre- sentations can be categorized into the following three approaches: (1) constructing target word em- beddings from the subword components (Pinter et al., 2017; Bojanowski et al., 2017), (2). lever- aging definitions or relational structures from ex- ternal resources such as Wordnet (Bahdanau et al., 2017; Pilehvar and Collier, 2017), and (3) mod- elling the target word from few available contexts. Our paper falls into the last approach. We demonstrate improvements in performance by employing an alternative context representa- tion, second-order lexical substitutes, as opposed to the traditional bag of word context representa- tions. In line with previous research in this area, we evaluate our methodology on three tasks that measure the quality of the induced unseen word representation from contexts (Lazaridou et al., 2017; Herbelot and Baroni, 2017; Khodak et al., 2018). Our results reveal that the three tasks in- volve different types of contexts which put dif- ferent emphasis on first or second order con- texts. Our second-order substitute-based method achieves the best performance for modelling rare words in natural contexts from corpora. In the tasks in which both first order and second order contexts are important, the ensemble of these two types of contexts yields superior performance. 1 2 Related work 2.1 First-order context The most naive way of inducing new word repre- sentation from contexts is to simply take the av- erage of context word embeddings that co-occur with the target word in a sentence. With stop words removed, this simple method has proven to be a strong baseline as shown …

Content truncated to 500 words