An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification

Proceedings of the 28th International Conference on Computational Linguistics, pages 3346–3356 Barcelona, Spain (Online), December 8-13, 2020 3346 An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification Robert Vacareanu,1,2 Marco A. Valenzuela-Esca´rcega,1 Rebecca Sharp,1 and Mihai Surdeanu1 1 University of Arizona, Tucson, AZ, USA 2 Technical University of Cluj-Napoca, Cluj-Napoca, Cluj, Romania {rvacareanu,marcov,bsharp,msurdeanu}@email.arizona.edu Abstract This paper explores an unsupervised approach to learning a compositional representation func- tion for multi-word expressions (MWEs), and evaluates it on the Tratz dataset, which associates two-word expressions with the semantic relation between the compound constituents (e.g. the label employer is associated with the noun compound government agency) (Tratz, 2011). The composition function is based on recurrent neural networks, and is trained using the Skip-Gram objective to predict the words in the context of MWEs. Thus our approach can naturally lever- age large unlabeled text sources. Further, our method can make use of provided MWEs when available, but can also function as a completely unsupervised algorithm, using MWE boundaries predicted by a single, domain-agnostic part-of-speech pattern. With pre-defined MWE bound- aries, our method outperforms the previous state-of-the-art performance on the coarse-grained evaluation of the Tratz dataset (Tratz, 2011), with an F1 score of 50.4%. The unsupervised ver- sion of our method approaches the performance of the supervised one, and even outperforms it in some configurations. 1 Introduction Multi-word expressions (MWEs) are fundamental to language and, as such, having a robust seman- tic representation for MWEs is important for any natural language processing task that involves text understanding such as information extraction, or question answering (e.g., da Silva and Souza, 2012; Thurmair, 2018; Subramanian et al., 2018). While MWEs have received attention in recent years, leading to considerable progress in learning MWE representations (Mitchell and Lapata, 2010; Butnariu et al., 2010; Tratz, 2011; Hendrickx et al., 2013; Dima, 2016; Shwartz and Dagan, 2018; Shwartz, 2019), we argue that the proposed methods have limitations. First, some methods require concatenating words in specified MWEs, and treating the resulting MWE phrases as atomic units. Training a set of dedicated distributional embeddings for the new multi-word terms (Shwartz and Dagan, 2018; Dima, 2016) suffers from language sparsity. For example, the MWE “red flower” is two orders of magnitude less frequent in Google search results than the noun “flower,” which is likely to affect the quality of its learned MWE representation. These methods additionally have no straightforward way of handling MWEs that are out of vocabulary. Second, other approaches require supervision for MWE boundaries (Yu and Dredze, 2015), which hinders scalability and portability to different languages. In all situations, the reliance on having determined your entities of interest ahead of time threatens to dramatically reduce the real-world utility of these approaches. Here, we propose a method that addresses both limitations by combining recent advances in language modeling (Howard and Ruder, 2018; Merity et al., 2017) with the simplicity and proven capability of the Skip-Gram training objective (Mikolov et al., 2013a). Our approach is summarized in Figure 1. Intuitively, our …

Content truncated to 500 words