Unsupervised Concept Representation Learning for Length-Varying Text Similarity

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5611–5620 June 6–11, 2021. ©2021 Association for Computational Linguistics 5611 Unsupervised Concept Representation Learning for Length-Varying Text Similarity Xuchao Zhang, Bo Zong, Wei Cheng, Jingchao Ni, Yanchi Liu, Haifeng Chen NEC Laboratories America, Princeton, NJ, USA {xuczhang, bzong, weicheng, jni, yanchi, haifeng}@nec-labs.com Abstract Measuring document similarity plays an im- portant role in natural language processing tasks. Most existing document similarity approaches suffer from the information gap caused by context and vocabulary mismatches when comparing varying-length texts. In this paper, we propose an unsupervised concept representation learning approach to address the above issues. Specifically, we propose a novel Concept Generation Network (CGNet) to learn concept representations from the per- spective of the entire text corpus. Moreover, a concept-based document matching method is proposed to leverage advances in the recog- nition of local phrase features and corpus- level concept features. Extensive experiments on real-world data sets demonstrate that new method can achieve a considerable improve- ment in comparing length-varying texts. In particular, our model achieved 6.5% better F1 Score compared to the best of the base- line models for a concept-project benchmark dataset. 1 Introduction Measuring the similarity between documents is a fundamental problem in several natural language tasks such as information retrieval (Manning et al., 2008), paraphrase identification (Yin and Schütze, 2015) and question routing (Zhang et al., 2020). A wide range of document similarity approaches (Kusner et al., 2015; Huang et al., 2016) have been proposed to handle the fundamental problem; how- ever, most of them are based on the assumption that the documents being compared have similar docu- ment length. However, varying-length document matching tasks are ubiquitous in many real-world scenarios. For instance, in the news categorization task, the news articles may include both short re- ports for breaking news or narrative reports with cumbersome details. The document matching in varying length may introduce the information gap between two doc- uments in the following two aspects: (i) context mismatch, which is caused by the long-length doc- uments usually provide more detailed context to support the key information while the short-length documents contain limited context information. The issue renders the existing pre-trained natural language representation models (Conneau et al., 2017a; Devlin et al., 2018) pay more attention to the long but less important contexts, which makes their document representations distinct from the short-length documents with little context informa- tion. (ii) vocabulary mismatch, which is usually caused by the different terms usage between short and long texts, which leads them do not share ma- jority terms. Existing document distance such as word mover’s distance (Kusner et al., 2015) focus on comparing the local features. Still, the vocabu- lary mismatch issue makes the local features hard to be matched while the majority of vocabulary is not shared. To address the above challenges, our approach proposes a concept-based document matching method that incorporates both local phrase features and corpus-level concepts in an unsupervised …

Content truncated to 500 words