Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations

Proceedings of the 28th International Conference on Computational Linguistics, pages 3268–3284 Barcelona, Spain (Online), December 8-13, 2020 3268 Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations Simone Conia and Roberto Navigli Sapienza NLP Group Department of Computer Science Sapienza University of Rome {conia,navigli}@di.uniroma1.it Abstract To date, the most successful word, word sense, and concept modelling techniques have used large corpora and knowledge resources to produce dense vector representations that capture semantic similarities in a relatively low-dimensional space. Most current approaches, however, suffer from a monolingual bias, with their strength depending on the amount of data available across languages. In this paper we address this issue and propose Conception, a novel technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts. Our approach results in high- coverage representations that outperform the state of the art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation, proving particularly robust on low- resource languages. Conception – its software and the complete set of representations – is avail- able at https://github.com/SapienzaNLP/conception. 1 Introduction Word vector representations, in particular dense representations or word embeddings (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017), play a key role in a wide range of tasks, including Text Similarity (Kenter and de Rijke, 2015; Nguyen et al., 2019), Word Sense Disambiguation (Iacobacci et al., 2016; Raganato et al., 2017a), Semantic Role Labeling (He et al., 2017; Marcheggiani et al., 2017; Conia et al., 2020), Question Answering (Zhou et al., 2015) and Machine Translation (Mikolov et al., 2013b; Bahdanau et al., 2015). This is especially the case when they are used as the underlying input representation. Word embedding techniques map each word to a relatively low n-dimensional space where two semantically or syntactically similar words lie close together. Due to their latent nature, however, most embeddings are commonly considered to be uninterpretable (Levy and Goldberg, 2014) as the properties captured by each dimension are often unclear. More recent studies have shed some light on their interpretability (Rothe and Sch¨utze, 2016; Senel et al., 2018; Wallace et al., 2019) or included interpretability directly in the learning process (Park et al., 2017; Koc¸ et al., 2018), but the opaqueness of dense vectors is still a key reason why research has not completely given up on sparse representations (Faruqui et al., 2015; Derby et al., 2018). Moreover, most current embedding techniques rely on large corpora which are often available in few languages, such as English or Chinese, strongly limiting their robustness on low-resource languages (Speer and Lowry-Duda, 2017). In an attempt to solve this issue, researchers turned to multilingual word representations by making use of parallel vocabularies (Mikolov et al., 2013b; Ammar et al., 2016; Smith et al., 2017), exploiting multilingual knowledge graphs (Speer et al., 2017), and exploring unsupervised methods to align monolingual embeddings in a single shared distributional space (Conneau et al., 2017) or to directly learn multilingual embeddings (Chen and Cardie, 2018). Nevertheless, a well-known pitfall of …

Content truncated to 500 words