Cross-Lingual Document Retrieval with Smooth Learning

Proceedings of the 28th International Conference on Computational Linguistics, pages 3616–3629 Barcelona, Spain (Online), December 8-13, 2020 3616 Cross-Lingual Document Retrieval with Smooth Learning Jiapeng Liu∗and Xiao Zhang∗ Amazon.com {liujiape, zhxao}@amazon.com Dan Goldwasser and Xiao Wang Purdue University {dgoldwas, wangxiao}@purdue.edu Abstract Cross-lingual document search is an information retrieval task in which the queries’ language differs from the documents’ language. In this paper, we study the instability of neural document search models and propose a novel end-to-end robust framework that achieves improved perfor- mance in cross-lingual search with different documents’ languages. This framework includes a novel measure of the relevance, smooth cosine similarity, between queries and documents, and a novel loss function, Smooth Ordinal Search Loss, as the objective. We further provide theoretical guarantee on the generalization error bound for the proposed framework. We conduct experiments to compare our approach with other document search models, and observe significant gains under commonly used ranking metrics on the cross-lingual document retrieval task in a variety of languages. 1 Introduction In modern search engines, cross-lingual information retrieval tasks are becoming prevalent and important. For example, when searching for products on a shopping website in English, immigrants tend to use their native languages to form the queries and would like to see the most desired products which are in English. Another example is in international trading, investors might use English to describe their product to search the sentiment in other languages on online forums from different countries, in order to understand the customers’ attitude towards it. Despite that these tasks can be naturally formulated as information retrieval (IR) tasks and resolved by mono-lingual methods, the rising need of cross-lingual IR techniques also requires robust models to deal with queries and documents from different languages. Early studies on documents retrieval mostly rely on lexical matching, which is error-prone in cross- lingual tasks as vocabularies and language styles usually change across different languages, as well the contextual information is largely lost. With the recent surge of deep neural networks (DNNs), researchers are able to go beyond lexical matching by building neural architectures to represent textual information of query and documents by vector representations via non-linear transformations, which have shown great successes in many applications [Salakhutdinov and Hinton, 2009; Huang et al., 2013; Shen et al., 2014; Palangi et al., 2016]. Despite the success of those aforementioned advances, several existing difficulties have not been well addressed, e.g. exploding gradients and convergence guarantee. The most widely used optimization method for training DNNs is stochastic gradient descent, which updates model parameters by taking gradients of the weights. Meanwhile, cosine similarity, a commonly used measure of relevance between query and documents, is not stable as its gradient may go to infinity when the Euclidean norm of the representation vector is close to zero, leading to exploding gradients and resulting in irrational training. Figure 1 demonstrates the gradients of cosine similarity and our proposed smooth cosine similarity. In addition, in most of the previous works, the loss functions are either heuristically designed …

Content truncated to 500 words