A first evaluation of logic form identification systems

A First Evaluation of Logic Form Identification Systems Vasile Rus Department of Computer Science Indiana University South Bend, IN 46634 vasile@cs.iusb.edu Abstract This paper presents a first experience with evalu- ating sytems that address the issue of Logic Form Identification (LFi). A Gold Standard approach was used in which experts provide solutions to test data. The expert solutions, the gold standard, are then compared against outputs from participanting sys- tems and different metrics observed. We proposed a few novel metrics, including precision and recall, that are further used to provide comparative results. The test data included 4155 arguments and 2398 predicates grouped in 300 sentences. 1 Introduction The goal of a Logic Form Identification (LFi) task is to evaluate the performance of different methods ad- dressing the issue of LFi. The Logic Form (LF) that we use is a flat, scope-free first order logic represen- tation that embeds lexical and syntactic information. Given a set of English sentences, participating sys- tems were supposed to return the sentences in Logic Form as in the example below. Input: The Earth provides the food we eat every day. Output: Earth:n (x1) provide:v (e1, x1, x2) food:n (x2) we(x3) eat:v (e2, x3, x2; x4) day:n (x4) The general approach adopted for evaluation was a gold standard approach in which the test data is first correctly mapped onto its corresponding LF by a team of experts and then this correct LF is automat- ically compared against outputs provided by partic- ipating sytems. 2 General Guidelines The Logic Form of a sentence is the conjunction of individual predicates, where the relationships among them are expressed via shared arguments. Predicates are generated for all content words such as nouns, verb, adjectives and adverbs. Pronouns are treated as independent nouns. Prepositions and conjunctions are also mapped onto predicates that capture the relation between the prepositional ob- ject and the constituent to which it is attached and the relation among the coordinated entities, respec- tively. There are two types of arguments: e - for events, x - for entities. For the sentence presented above we have two events - e1, e2 corresponding to each verb/action in the sentence and four entities - x1, x2, x3, x4 corresponding to the heads of the base noun phrases (NP). Each verb predicate has the second ar- gument set to the corresponding logical subject and the third argument to its direct object. The remain- ing slots for a verb predicate are filled with the ar- guments of their indirect and prepositional objects. In the example presented above the predicate eat has arguments after ; (semicolon) which indicates its adjuncts. The distinction between complements and adjuncts is a novelty to the LF proposed by the authors in this paper. For this first trial, we did not make the distinction between the two and thus the accepted representation would be eat:v (e2, x3, x2, x4) - see below. Output: Earth:n (x1) provide:v (e1, x1, x2) food:n (x2) we(x3) eat:v (e2, x3, x2, x4) day:n (x4) …

Content truncated to 500 words