Alignment over Heterogeneous Embeddings for Question Answering

Proceedings of NAACL-HLT 2019, pages 2681–2691 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2681 Alignment over Heterogeneous Embeddings for Question Answering Vikas Yadav, Steven Bethard, Mihai Surdeanu University of Arizona, Tucson, AZ, USA {vikasy,bethard,msurdeanu}@email.arizona.edu Abstract We propose a simple, fast, and mostly- unsupervised approach for non-factoid ques- tion answering (QA) called Alignment over Heterogeneous Embeddings (AHE). AHE sim- ply aligns each word in the question and candi- date answer with the most similar word in the retrieved supporting paragraph, and weighs each alignment score with the inverse doc- ument frequency of the corresponding ques- tion/answer term. AHE’s similarity function operates over embeddings that model the un- derlying text at different levels of abstraction: character (FLAIR), word (BERT and GloVe), and sentence (InferSent), where the latter is the only supervised component. Despite its simplicity and lack of supervision, AHE ob- tains a new state-of-the-art performance on the “Easy” partition of the AI2 Reasoning Chal- lenge (ARC) dataset (64.6% accuracy), top- two performance on the “Challenge” partition of ARC (34.1%), and top-three performance on the WikiQA dataset (74.08% MRR), out- performing many other complex, supervised approaches. Our error analysis indicates that alignments over character, word, and sentence embeddings capture substantially different se- mantic information. We exploit this with a simple meta-classifier that learns how much to trust the predictions over each representation, which further improves the performance of un- supervised AHE1. 1 Introduction The “deep learning tsunami”(Manning, 2015) has had a major impact on important natural language processing (NLP) applications such as question an- swering (QA). Many neural approaches for QA have been proposed in the past few years, with impressive results on several QA tasks (Seo et al., 2016; Wang and Jiang, 2016; Wang et al., 2017b; 1Code: https://github.com/vikas95/AHE Question - Which sequence of energy transformations oc- curs after a battery-operated flashlight is turned on? 1. electrical → light → chemical 2. electrical →chemical → light 3. chemical → light → electrical 4. chemical → electrical → light Supporting paragraph(s): “a chemical cell converts chem- ical energy into electrical energy; a flashlight chemical energy to light energy” Figure 1: A multiple-choice question from the ARC dataset with the correct answer in bold font. This ques- tion is answered correctly by our alignment method that relies on contextualized word embeddings that capture the correct sequence, and cannot be answered correctly when relying on uncontextualized embeddings. Tymoshenko et al., 2017; Xiong et al., 2016a; Wang et al., 2018; Radford et al., 2018; Li et al., 2018, inter alia). However, an undesired effect of this focus on neural approaches was that other methods have fallen out of focus, including strong unsuper- vised benchmarks that are necessary to highlight the true gains of supervised approaches. For in- stance, alignment approaches have received consid- erably less interest recently, despite their initial suc- cesses (Echihabi and Marcu, 2003; Surdeanu et al., 2011, inter alia). While a few recent efforts have adapted these alignment methods to operate over word representations (Kenter and …

Content truncated to 500 words