Cross-sentence Pre-trained Model for Interactive QA matching

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5417–5424 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5417 Cross-sentence Pre-trained model for Interactive QA matching Jinmeng Wua,b, Yanbin Haoc aSchool of Electrical and Information Engineering, Wuhan Institute of Technology, China bThe school of Electrical Engineering, Electronics and Computer Science, The University of Liverpool, Brownlow Hill, Liverpool, UK cDepartment of Computer Science, City University of Hong Kong, Hong Kong, 999077, China � jinmeng2004910@outlook.com, haoyanbin@hotmail.com Abstract Semantic matching measures the dependencies between query and answer representations, which is an important criterion for evaluating whether the matching is successful. In fact, such matching does not examine each sentence individually, because the context information between sentences should be considered equally important to the syntactic context inside a sentence. Considering the above, we propose a novel QA matching model, built upon a cross-sentence context-aware architecture. Specifically, an interactive attention mechanism with a pre-trained language model is presented to automatically select salient positional answer representations that contribute more significantly to the answer relevance of a given question. In addition to the context information captured at each word position, we incorporate a quantity of context jump dependencies to leverage the attention weight formulation. This can capture the amount of useful information brought by the next word, is computed by modeling the joint probability between two adjacent word states. The proposed method is compared with multiple state-of-the-art methods using the TREC library, WikiQA, and the Yahoo! community question datasets. Experimental results show that the proposed method outperforms satisfactorily the competing ones. Keywords: Question answering, Interactive matching, Pre-trained language model, Context jump Dependencies 1. Introduction Question answering (QA) is the task of enabling a ma- chine to automatically answer questions posted by humans in a natural language form. The selection of the best an- swer from an existing pool of candidate answers is referred to as community QA (cQA) (Shah and Pomerantz, 2010), whereas enabling the computer to automatically generate a novel answer, through some natural language models, is also known as machine dialogue (Shen et al., 2017b). In this paper, we focus on cQA by working on the se- mantic matching between question and answer texts. In general, semantic matching requires the accurate model- ing of the relevance between two portions of text, and, in addition to QA, is widely used for tasks, such as para- phrase identification (Cheng and Kartsaklis, 2015), ma- chine translation (Bahdanau et al., 2015), image caption generation (Karpathy and Fei-Fei, 2015) and video hyper- linking (Hao et al., 2019b). In order to compute an accurate measure of relevance be- tween the question and answer, it is beneficial to take the lexical, syntactic and semantic information of the text pairs into account. Traditional matching seeks effective ways of extracting semantic features to improve a given similarity metric (Meng et al., 2013). Recent advances have managed to replace this manual feature engineering process with a model that automatically learns distributed representations of words and sentences …

Content truncated to 500 words