Handling Noisy Queries in Cross Language FAQ Retrieval

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 87–96, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Handling Noisy Queries In Cross Language FAQ Retrieval Danish Contractor Govind Kothari Tanveer A. Faruquie L. Venkata Subramaniam IBM Research IndiaSumit Negi Vasant Kunj, Institutional Area New Delhi, India {dcontrac,govkotha,ftanveer,lvsubram,sumitneg}@in.ibm.com Abstract Recent times have seen a tremendous growth in mobile based data services that allow peo- ple to use Short Message Service (SMS) to access these data services. In a multilin- gual society it is essential that data services that were developed for a specific language be made accessible through other local lan- guages also. In this paper, we present a ser- vice that allows a user to query a Frequently- Asked-Questions (FAQ) database built in a lo- cal language (Hindi) using Noisy SMS En- glish queries. The inherent noise in the SMS queries, along with the language mismatch makes this a challenging problem. We handle these two problems by formulating the query similarity over FAQ questions as a combina- torial search problem where the search space consists of combinations of dictionary varia- tions of the noisy query and its top-N transla- tions. We demonstrate the effectiveness of our approach on a real-life dataset. 1 Introduction There has been a tremendous growth in the number of new mobile subscribers in the recent past. Most of these new subscribers are from developing coun- tries where mobile is the primary information de- vice. Even for users familiar with computers and the internet, the mobile provides unmatched portability. This has encouraged the proliferation of informa- tion services built around SMS technology. Several applications, traditionally available on Internet, are now being made available on mobile devices using SMS. Examples include SMS short code services. Short codes are numbers where a short message in a predesignated format can be sent to get specific information. For example, to get the closing stock price of a particular share, the user has to send a message IBMSTOCKPR. Other examples are search (Schusteritsch et al., 2005), access to Yellow Page services (Kopparapu et al., 2007), Email 1, Blog 2 , FAQ retrieval 3 etc. The SMS-based FAQ retrieval services use human experts to answer SMS ques- tions. Recent studies have shown that instant messag- ing is emerging as the preferred mode of commu- nication after speech and email.4 Millions of users of instant messaging (IM) services and short mes- sage service (SMS) generate electronic content in a dialect that does not adhere to conventional gram- mar, punctuation and spelling standards. Words are intentionally compressed by non-standard spellings, abbreviations and phonetic transliteration are used. Typical question answering systems are built for use with languages which are free from such errors. It is difficult to build an automated question answer- ing system around SMS technology. This is true even for questions whose answers are well docu- mented like in a Frequently-Asked-Questions (FAQ) database. Unlike other automatic question answer- ing systems that focus on searching answers from …

Content truncated to 500 words