Learning Neural Word Salience Scores

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 33–42 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Learning Neural Word Salience Scores Krasen Samardzhiev Andrew Gargett Department of Computer Science, University of Liverpool Science and Technology Facilities Council. krasensam@gmail.com, danushka@liverpool.ac.uk andrew.gargett@stfc.ac.uk Danushka Bollegala Abstract Measuring the salience of a word is an essen- tial step in numerous NLP tasks. Heuristic ap- proaches such as tfidf have been used so far to estimate the salience of words. We propose Neural Word Salience (NWS) scores, unlike heuristics, are learnt from a corpus. Specifi- cally, we learn word salience scores such that, using pre-trained word embeddings as the in- put, can accurately predict the words that ap- pear in a sentence, given the words that ap- pear in the sentences preceding or succeed- ing that sentence. Experimental results on sen- tence similarity prediction show that the learnt word salience scores perform comparably or better than some of the state-of-the-art ap- proaches for representing sentences on bench- mark datasets for sentence similarity, while us- ing only a fraction of the training and predic- tion times required by prior methods. More- over, our NWS scores positively correlate with psycholinguistic measures such as concrete- ness, and imageability implying a close con- nection to the salience as perceived by hu- mans. 1 Introduction Humans can easily recognise the words that con- tribute to the meaning of a sentence (i.e. content words) from words that serve only a grammatical functionality (i.e. functional words). For example, functional words such as the, an, a etc. have lim- ited contributions towards the overall meaning of a document and are often filtered out as stop words in information retrieval systems (Salton and Buck- ley, 1983). We define the salience q(w) of a word w in a given text T as the semantic contribution made by w towards the overall meaning of T. If we can accurately compute the salience of words, then we can develop better representations of texts that can be used in downstream NLP tasks such as similarity measurement (Arora et al., 2017) or text (e.g. sentiment, entailment) classification (Socher et al., 2011). As described later in section 2, existing meth- ods for detecting word salience can be classi- fied into two groups: (a) lexicon-based filter- ing methods such as stop word lists, or (b) word frequency-based heuristics such as the pop- ular term-frequency inverse document frequency (tfidf) (Jones, 1972) measure and its variants. Un- fortunately, two main drawbacks can be identified in common to both stop words lists and frequency- based salience scores. First, such methods do not take into account the semantics associated with individual words when determining their salience. For example, consider the following two adjacent sentences extracted from a newspaper article related to the visit of the Japanese Prime Minister, Shinzo Abe, to the White House in Washington, to meet the US President Donald Trump. (a) Abe visited Washington in February and met Trump in the White House. (b) …

Content truncated to 500 words