TITA: A Two-stage Interaction and Topic-Aware Text Matching Model
21370Authors
nan
Abstract
AbstractIn this paper, we focus on the problem of keyword and document matching by considering different relevance levels. In our recommendation system, different people follow different hot keywords with interest. We need to attach documents to each keyword and then distribute the documents to people who follow these keywords. The ideal documents should have the same topic with the keyword, which we call topic-aware relevance. In other words, topic-aware relevance documents are better than partially-relevance ones in this application. However, previous tasks never define topic-aware relevance clearly. To tackle this problem, we define a three-level relevance in keyword-document matching task: topic-aware relevance, partially-relevance and irrelevance. To capture the relevance between the short keyword and the document at above-mentioned three levels, we should not only combine the latent topic of the document with its deep neural representation, but also model complex interactions between the keyword and the document. To this end, we propose a Two-stage Interaction and Topic-Aware text matching model (TITA). In terms of “topic-aware”, we introduce neural topic model to analyze the topic of the document and then use it to further encode the document. In terms of “two-stage interaction”, we propose two successive stages to model complex interactions between the keyword and the document. Extensive experiments reveal that TITA outperforms other well-designed baselines and shows excellent performance in our recommendation system.
Keywords
Full Content
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 499–511 June 6–11, 2021. ©2021 Association for Computational Linguistics 499 MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning Mengzhou Xia§ ∗ Guoqing Zheng‡ Subhabrata Mukherjee‡ Milad Shokouhi‡ Graham Neubig† Ahmed Hassan Awadallah‡ §Princenton University †Carnegie Mellon University ‡Microsoft Research mengzhou@princeton.edu, {zheng, submukhe, milads}@microsoft.com gneubig@cs.cmu.edu, hassanam@microsoft.com Abstract The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low- resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages (Singh et al., 2019), bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages – without access to large-scale monolingual corpora or large amounts of labeled data – for tasks like cross- lingual sentiment analysis and named entity recognition show the effectiveness of our ap- proach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL. 1 Introduction Recent advances in multilingual pre-trained repre- sentations have enabled success on a wide range of natural language processing (NLP) tasks for many languages. However, these techniques may not readily transfer onto extremely low-resource lan- guages, where: (1) large-scale monolingual cor- pora are not available for pre-training and (2) suf- ficient labeled data is lacking for effective fine- tuning for downstream tasks. For example, mul- tilingual BERT (mBERT) (Devlin et al., 2018) is pre-trained on 104 languages with many articles on ∗Most of the work was done while the first author was an intern at Microsoft Research. (a) Joint Training en tel (b) MetaXL en tel Figure 1: First two principal components of sequence representations (corresponding to [CLS] tokens) of Telugu and English examples from a jointly fine-tuned mBERT and a MetaXL model for the task of sentiment analysis. MetaXL pushes the source (EN) and target (TEL) representations closer to realize a more effective transfer. The Hausdorff distance between the source and target representations drops from 0.57 to 0.20 with F1 score improvement from 74.07 to 78.15. Wikipedia and XLM-R (Conneau et al., 2020) is pre-trained on 100 languages with CommonCrawl Corpora. However, these models still leave behind more than 200 languages with few articles avail- able in Wikipedia, not to mention the 6, 700 or so languages with no Wikipedia text at all (Artetxe et al., 2020). Cross-lingual transfer learning for these extremely low-resource languages is essen- tial for better information access but under-studied in practice (Hirschberg and Manning, 2015). Re- cent work on cross-lingual transfer learning using pre-trained representations mainly focuses on trans- ferring across languages that are already covered by existing representations (Wu and Dredze, 2019). …
Content truncated to 500 words
Labeled Text
AbstractIn this paper, we focus on the problem of keyword and document matching by considering different relevance levels. In our recommendation system, different people follow different hot keywords with interest. We need to attach documents to each keyword and then distribute the documents to people who follow these keywords. The ideal documents should have the same topic with the keyword, which we call topic-aware relevance. In other words, topic-aware relevance documents are better than partially-relevance ones in this application. However, previous tasks never define topic-aware relevance clearly. To tackle this problem, we define a three-level relevance in keyword-document matching task: topic-aware relevance, partially-relevance and irrelevance. To capture the relevance between the short keyword and the document at above-mentioned three levels, we should not only combine the latent topic of the document with its deep neural representation, but also model complex interactions between the keyword and the document. To this end, we propose a Two-stage Interaction and Topic-Aware text matching model (TITA). In terms of “topic-aware”, we introduce neural topic model to analyze the topic of the document and then use it to further encode the document. In terms of “two-stage interaction”, we propose two successive stages to model complex interactions between the keyword and the document. Extensive experiments reveal that TITA outperforms other well-designed baselines and shows excellent performance in our recommendation system.
Publication Details
- Paper ID:
- 21370
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan