Benchmarking Noun Compound Interpretation

Benchmarking Noun Compound Interpretation Su Nam Kim and Timothy Baldwin Department of Computer Science and Software Engineering and NICTA Victoria Lab University of Melbourne, VIC 3010 Australia {snkim,tim}@csse.unimelb.edu.au Abstract In this paper we provide benchmark results for two classes of methods used in inter- preting noun compounds (NCs): semantic similarity-based methods and their hybrids. We evaluate the methods using 7-way and binary class data from the nominal pair in- terpretation task of SEMEVAL-2007.1 We summarize and analyse our results, with the intention of providing a framework for benchmarking future research in this area. 1 Introduction This paper reviews a range of simple and hybrid approaches to noun compound (NC) interpretation. The interpretation of NCs such as computer science and paper submission involves predicting the se- mantic relation (SR) that underlies a given NC. For example, student price conventionally expresses the meaning that a student benefits from the price (SR = BENEFICIARY), while student protest conven- tionally means a student undertaking a protest (SR = AGENT).2 NCs are formed from simplex nouns with high productivity. The huge number of possible NCs and potentially large number of SRs makes NC interpre- tation a very difficult problem. In the past, much NC interpretation work has been carried out which tar- gets particular NLP applications such as information extraction, question-answering and machine trans- lation. Unfortunately, much of it has not gained 1The 4th International Workshop on Semantic Evaluation 2SRs used in the examples are taken from Barker and Sz- pakowicz (1998). traction in real-world applications as the accuracy of the methods has not been sufficiently high over open-domain data. Most prior work has been car- ried out under specific assumptions and with one- off datasets, which makes it hard to analyze perfor- mance and to build hybrid methods. Additionally, disagreement in the inventory of SRs and a lack of resource sharing has hampered comparative evalua- tion of different methods. The first step in NC interpretation is to define a set of SRs. Levi (1979), for example, proposed a system of 9 SRs, while others have proposed classifications with 20-30 SRs (Finin, 1980; Barker and Szpakow- icz, 1998; Moldovan et al., 2004). Smaller sets tend to have reduced coverage due to coarse granularity, whereas larger sets tend to be too fine grained and suffer from low inter-annotator agreement. Addi- tionally pragmatic/contextual differentiation leads to difficulties in defining and interpreting SRs (Down- ing, 1977; SparckJones, 1983). Recent attempts in the area of NC interpretation have taken two basic approaches: analogy-base in- terpretation (Rosario, 2001; Moldovan et al., 2004; Kim and Baldwin, 2005; Girju, 2007) and seman- tic disambiguation relative to an underlying predi- cate or semantically-unambiguous paraphrase (Van- derwende, 1994; Lapata, 2002; Kim and Baldwin, 2006; Nakov, 2006). Most methods employ rich on- tologies and ignore the context of use, supporting the claim by Fan (2003) that axioms and ontological distinctions are more important than detailed knowl- edge of specific nouns for NC interpretation. Addi- tionally, most approaches use supervised learning, raising questions about the generality …

Content truncated to 500 words