Embedding Meta-Textual Information for Improved Learning to Rank

Proceedings of the 28th International Conference on Computational Linguistics, pages 5558–5568 Barcelona, Spain (Online), December 8-13, 2020 5558 Embedding Meta-Textual Information for Improved Learning to Rank Toshitaka Kuwa∗, Shigehiko Schamoni†,‡, Stefan Riezler†,‡ †Department of Computational Linguistics, Heidelberg, Germany. ‡Interdisciplinary Center for Scientific Computing (IWR), Heidelberg, Germany {kuwa, schamoni, riezler}@cl.uni-heidelberg.de Abstract Neural approaches to learning term embeddings have led to improved computation of similarity and ranking in information retrieval (IR). So far neural representation learning has not been ex- tended to meta-textual information that is readily available for many IR tasks, for example, patent classes in prior-art retrieval, topical information in Wikipedia articles, or product categories in e-commerce data. We present a framework that learns embeddings for meta-textual categories, and optimizes a pairwise ranking objective for improved matching based on combined embed- dings of textual and meta-textual information. We show considerable gains in an experimental evaluation on cross-lingual retrieval in the Wikipedia domain for three language pairs, and in the Patent domain for one language pair. Our results emphasize that the mode of combining different types of information is crucial for model improvement. 1 Introduction The recent success of neural methods in IR rests on bridging the gap between query and document vocabulary by learning term embeddings that allow for improved computation of similarity and ranking (Mitra and Craswell, 2018). So far vector representations in neural IR have been confined to textual information, neglecting information beyond the raw text that is potentially useful for retrieval and is readily available in many data situations. For example, meta-textual information is available in the form of patent classes in prior-art search, topic classes in search over Wikipedia articles, or product classes in retrieval in the e-commerce domain. A straightforward approach to use such meta-textual information to improve retrieval would require an exact match of meta-textual categories of queries and relevant documents. However, in the majority of cases the sets of categories assigned to queries and documents overlap only at a small percentage, for both relevant and irrelevant documents. Thus, more sophisticated techniques are needed to aid similarity computation and ranking by meta-textual category information. In this paper, we show how to apply neural embedding methods to meta-textual categories. We show that meta-textual information is an incomplete representation of queries and documents and does not suffice for a standalone computation of similarity and ranking. However, an incorporation of pre-trained embeddings of meta-textual categories into a neural learning-to-rank approach yields significant im- provements over a learning-to-rank approach that uses text-only embeddings (Sasaki et al., 2018). In our approach, enhanced embeddings are created by concatenating text embeddings with meta-textual category embeddings, and a fully connected weight layer is learned on top of the concatenation of the enhanced embeddings of a query-document pair by a deep multilayer perceptron that is optimized for pairwise ranking. We present experiments on three different language pairs for cross-lingual retrieval in the Wikipedia domain, and show improvements of up to 2 NDCG points by incorporating meta-textual embeddings in a learning-to-rank framework. Additional experiments …

Content truncated to 500 words