The Query of Everything: Developing Open-Domain, Natural-Language Queries for BOLT Information Retrieval

The Query of Everything: Developing Open-Domain, Natural-Language Queries for BOLT Information Retrieval Kira Griffitt, Stephanie Strassel Linguistic Data Consortium, University of Pennsylvania 3600 Market Street, Suite 810, Philadelphia, PA. 19104 USA Email: kiragrif@ldc.upenn.edu, strassel@ldc.upenn.edu Abstract The DARPA BOLT Information Retrieval evaluations target open-domain natural-language queries over a large corpus of informal text in English, Chinese and Egyptian Arabic. We outline the goals of BOLT IR, comparing it with the prior GALE Distillation task. After discussing the properties of the BOLT IR corpus, we provide a detailed description of the query creation process, contrasting the summary query format presented to systems at run time with the full query format created by annotators. We describe the relevance criteria used to assess BOLT system responses, highlighting the evolution of the procedures used over the three evaluation phases. We provide a detailed review of the decision points model for relevance assessment introduced during Phase 2, and conclude with information about inter-assessor consistency achieved with the decision points assessment model. Keywords: language resources, information retrieval, queries, question-answering, open-domain, natural-language, informal text Introduction 1. This paper describes the resources, procedures, and adaptations developed by the Linguistic Data Consortium (LDC) in support of the Information Retrieval (IR) evaluation within DARPA's Broad Operational Language Translation (BOLT) program. Within the context of BOLT’s overarching goal of improving machine translation capabilities in informal data genres, the BOLT IR task focused on advancing the state of the art of information retrieval over these genres (DARPA, 2011). In particular, BOLT IR seeks to support development of systems which could: 1) take as input a natural language English query sentence, 2) return relevant responses to that query from a large corpus of informal documents in the three BOLT languages (Arabic, Chinese, and English), and 3) translate relevant responses into English where necessary (i.e. if those responses came from non-English documents). These objectives were chosen because they closely modeled the information retrieval needs of monolingual English intelligence analyst (NIST 2014). LDC developed queries and assessed system responses for the BOLT Phase 1, Phase 2 and Phase 3 evaluations. The National Institute for Standards and Technology (NIST) was responsible for designing the BOLT IR evaluation task and measuring system performance. The BOLT IR Corpus 2. The BOLT program focused on English, Mandarin Chinese and Egyptian Arabic covering three genres: discussion forums (Garland et al., 2012); SMS/Chat (Song et al., 2014), and Conversational Telephone Speech. While the BOLT MT evaluations covered all three genres, IR evaluations focused exclusively on discussion forum data. This genre exhibits the challenges of informal language while still containing the kind of news-focused content required for multilingual query development in BOLT. The BOLT Phase 1 IR Corpus comprised 400 million words of discussion forum data per language. The large corpus size was necessary to ensure that multiple query sets could be developed from the same data pool without exhausting all possible topics, and that systems had a sufficiently large pool of data over which to do retrieval. In Phases 2 and 3 …

Content truncated to 500 words