300-sparsans at SemEval-2018 Task 9: Hypernymy as interaction of sparse attributes

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 928–934 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics 300-sparsans at SemEval-2018 Task 9: Hypernymy as interaction of sparse attributes Gábor Berend Department of Informatics University of Szeged Árpád tér 2, H6720 Szeged, Hungary berendg@inf.u-szeged.hu Márton Makrai Institute for Linguistics Hungarian Academy of Sciences Benczúr u. 33, H1068 Budapest, Hungary makrai.marton@nytud.mta.hu Péter Földiák Secret Sauce Partners 657 Mission Suite 410, San Francisco CA 94105 Peter.Foldiak@gmail.com Abstract This paper describes 300-sparsans’ participa- tion in SemEval-2018 Task 9: Hypernym Dis- covery, with a system based on sparse coding and a formal concept hierarchy obtained from word embeddings. Our system took first place in subtasks (1B) Italian (all and entities), (1C) Spanish entities, and (2B) music entities. 1 Introduction Natural language phenomena are extremely sparse by their nature, whereas continuous word em- beddings employ dense representations of words. Turning these dense representations into a much sparser form can help in focusing on most salient parts of word representations (Faruqui et al., 2015; Berend, 2017; Subramanian et al., 2018). Sparsity-based techniques often involve the coding of a large number of signals over the same dictionary (Rubinstein et al., 2008). Sparse, over- complete representations have been motivated in various domains as a way to increase separability and interpretability (Olshausen and Field, 1997) and stability in the presence of noise. Non-negativity has also been argued to be ad- vantageous for interpretability (Faruqui et al., 2015; Fyshe et al., 2015; Arora et al., 2016). As Subramanian et al. (2018) illustrates this in the language domain, where sparse features are inter- preted as lexical attributes, “to describe the city of Pittsburgh, one might talk about phenomena typical of the city, like erratic weather and large bridges. It is redundant and inefficient to list neg- ative properties, like the absence of the Statue of Liberty”. Berend (2018) utilizes non-negative sparse coding for word translation by training sparse word vectors for the two languages such that coding bases correspond to each other. Here we apply sparse feature pairs to hypernym extraction. The role of an attribute pair ⟨i, j⟩ ∈ φ(q) × φ(h) (where q is the query word, h is the hypernym candidate, and φ(w) is the index of a non-zero component in the sparse representations of w) is similar to interaction terms in regression, see section 2 for details. Sparse representation is related to hypernymy in various natural ways. One of them is through For- mal concept Analysis (FCA). The idea of acquir- ing concept hierarchies from a text corpus with the tools of Formal concept Analysis (FCA) is relatively new (Cimiano et al., 2005). Our sub- missions experiment with formal concept analysis tool by Endres et al. (2010). See the next section for a description of formal concept lattices, and how hypernyms can be found in them. Another natural formulation is related to hier- archical sparse coding (Zhao et al., 2009), where trees describe the order in which variables “enter the model” (i.e., …

Content truncated to 500 words