ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs Wenpeng Yin, Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany wenpeng@cis.lmu.de Bing Xiang, Bowen Zhou IBM Watson Yorktown Heights, NY, USA bingxia,zhou@us.ibm.com Abstract How to model a pair of sentences is a critical issue in many NLP tasks such as answer selec- tion (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence’s representation separately, rarely considering the impact of the other sentence; or (iii) re- lies fully on manually designed, task-specific linguistic features. This work presents a gen- eral Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) The ABCNN can be applied to a wide va- riety of tasks that require modeling of sen- tence pairs. (ii) We propose three attention schemes that integrate mutual influence be- tween sentences into CNNs; thus, the rep- resentation of each sentence takes into con- sideration its counterpart. These interdepen- dent sentence pair representations are more powerful than isolated sentence representa- tions. (iii) ABCNNs achieve state-of-the-art performance on AS, PI and TE tasks. We release code at: https://github.com/ yinwenpeng/Answer_Selection. 1 Introduction How to model a pair of sentences is a critical is- sue in many NLP tasks such as answer selection (AS) (Yu et al., 2014; Feng et al., 2015), paraphrase identification (PI) (Madnani et al., 2012; Yin and Sch¨utze, 2015a), textual entailment (TE) (Marelli et al., 2014a; Bowman et al., 2015a) etc. AS s0 how much did Waterboy gross? s+ 1 the movie earned $161.5 million s− 1 this was Jerry Reed’s final film appearance PI s0 she struck a deal with RH to pen a book today s+ 1 she signed a contract with RH to write a book s− 1 she denied today that she struck a deal with RH TE s0 an ice skating rink placed outdoors is full of people s+ 1 a lot of people are in an ice skating park s− 1 an ice skating rink placed indoors is full of people Figure 1: Positive (<s0, s+ 1 >) and negative (<s0, s− 1 >) examples for AS, PI and TE tasks. RH = Random House Most prior work derives each sentence’s represen- tation separately, rarely considering the impact of the other sentence. This neglects the mutual influ- ence of the two sentences in the context of the task. It also contradicts what humans do when comparing two sentences. We usually focus on key parts of one sentence by extracting parts from the other sentence that are related by identity, synonymy, antonymy and other relations. Thus, human beings model the two sentences together, using the content of one sen- tence to guide the representation of the other. Figure 1 demonstrates that each sentence of a pair partially determines which parts of the other sen- tence we must focus on. For AS, correctly answer- ing s0 requires attention on “gross”: …

Content truncated to 500 words