MTE-NN at SemEval-2016 Task 3: Can Machine Translation Evaluation Help Community Question Answering?

Proceedings of SemEval-2016, pages 887–895, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics MTE-NN at SemEval-2016 Task 3: Can Machine Translation Evaluation Help Community Question Answering? Francisco Guzm´an, Llu´ıs M`arquez and Preslav Nakov Arabic Language Technologies Research Group Qatar Computing Research Institute, HBKU {fguzman,lmarquez,pnakov}@qf.org.qa Abstract We present a system for answer ranking (SemEval-2016 Task 3, subtask A) that is a direct adaptation of a pairwise neural net- work model for machine translation evalua- tion (MTE). In particular, the network incor- porates MTE features, as well as rich syntac- tic and semantic embeddings, and it efficiently models complex non-linear interactions be- tween them. With the addition of lightweight task-specific features, we obtained very en- couraging experimental results, with sizeable contributions from both the MTE features and from the pairwise network architecture. We also achieved good results on subtask C. 1 Introduction We present a system for SemEval-2016 Task 3 on Community Question Answering (cQA), subtask A (English). In that task, we are given a question from a community forum and a thread of associated text comments intended to answer the question, and the goal is to rank the comments according to their ap- propriateness to the question. Since cQA forum threads are noisy, as many comments are not an- swers to the question, the challenge lies in learning to rank all good comments above all bad ones.1 In this work, we approach subtask A from a novel perspective: by using notions of machine transla- tion evaluation (MTE) to decide on the quality of a comment. In particular, we extend the MTE neu- ral network framework from Guzm´an et al. (2015). 1More detail and examples can be found on the task web- site (http://alt.qcri.org/semeval2016/task3/) and in the associ- ated task description paper (Nakov et al., 2016). We believe that this neural network is interesting for the cQA problem because: (i) it works in a pairwise fashion, i.e., given two translation hypotheses and a reference translation to compare to, the network de- cides which translation hypothesis is better; this is appropriate for a ranking problem; (ii) it allows for an easy incorporation of rich syntactic and semantic embedded representations of the input texts, and it efficiently models complex non-linear relationships among them; (iii) it uses a number of MT evaluation measures that have not been explored for the cQA task (e.g., TER, Meteor and BLEU). The analogy we apply to adapt the neural MTE architecture to the cQA problem is the following: given two comments c1 and c2 from the question thread—which play the role of the two translation hypotheses—we have to decide whether c1 is a bet- ter answer than c2 to question q—which plays the role of the translation reference. The two tasks seem similar: both reason about the similarity of two competing texts against a reference text, to decide which one is better. However, there are some profound differences. In MTE, the goal is to decide whether a hypothesis translation conveys the same meaning as the reference …

Content truncated to 500 words