Orthogonality regularizer for question answering

Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM 2016), pages 142–147, Berlin, Germany, August 11-12, 2016. Orthogonality regularizer for question answering Chunyang Xiao1, Guillaume Bouchard2, Marc Dymetman1,Claire Gardent3 1Xerox Research Centre Europe, Grenoble, France 2University College London, United Kingdom 3CNRS, LORIA, Nancy, France 1chunyang.xiao, marc.dymetman@xerox.com 2g.bouchard@cs.ucl.ac.uk 3claire.gardent@loria.fr Abstract Learning embeddings of words and knowledge base elements is a promising approach for open domain question answering. Based on the remark that relations and entities are distinct object types lying in the same embedding space, we analyze the benefit of adding a regularizer favoring the embeddings of entities to be orthogonal to those of relations. The main motivation comes from the observation that modifying the embeddings using prior knowledge often helps performance. The experiments show that incorporating the regularizer yields better results on a challenging question answering benchmark. 1 Introduction Having a system which is able to answer questions based on a structured knowledge base is a chal- lenging problem. The problem has been addressed recently by researchers working on large knowl- edge bases such as Reverb (Fader et al., 2011) and Freebase (Bollacker et al., 2008). The creation of question answering (QA) benchmarks for these knowledge bases (KB) has a significant impact on the domain, as shown by the number of QA sys- tems recently proposed in the literature (Berant and Liang, 2014; Berant et al., 2013; Bordes et al., 2014a; Bordes et al., 2014b; Fader et al., 2013; Fader et al., 2014; Yao and Van Durme, 2014; Yih et al., 2014; Dong et al., 2015). We identify two types of approaches for KB- centric QA systems: parsing-based approaches and information retrieval (IR) based approaches. Parsing-based approaches (Yih et al., 2014; Be- rant et al., 2013; Berant and Liang, 2014; Reddy et al., 2014) answer factoid questions by learn- ing a structured representation for the sentences, called logical form. This logical form is then used to query the knowledge base and retrieve the an- swer. IR-based approaches try to identify the best possible match between the knowledge base and the question (Bordes et al., 2014a; Bordes et al., 2014b; Yao and Van Durme, 2014; Dong et al., 2015). In this work, we focus on the second ap- proach, using embedding models, mainly because it is robust to invalid syntax and can exploit infor- mation of the answer. We focus on the Wikianswers (Fader et al., 2013) dataset constructed for Reverb. On Wikian- swers, the underlying semantics is very simple (just one single triple). However, the task remains challenging due to the large variety of lexicaliza- tions for the same semantics. We follow the ap- proach of Bordes et .al (2014b) which learns the embeddings of words and KB elements. They model the semantics of natural language sentences and KB triples as the sum of the embeddings of the associated words and KB elements respectively. Despite its simplicity, this model performs surpris- ingly well in practice. Something even more in- teresting (Bordes et al., 2014b) is that the …

Content truncated to 500 words