Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1180–1191, Osaka, Japan, December 11-17 2016. Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text? Filip Ilievski, Marten Postma, Piek Vossen Vrije Universiteit Amsterdam De Boelelaan 1105, 1081 HV Amsterdam The Netherlands {f.ilievski, m.c.postma, piek.vossen}@vu.nl Abstract Semantic text processing faces the challenge of defining the relation between lexical expressions and the world to which they make reference within a period of time. It is unclear whether the current test sets used to evaluate disambiguation tasks are representative for the full complexity considering this time-anchored relation, resulting in semantic overfitting to a specific period and the frequent phenomena within. We conceptualize and formalize a set of metrics which eval- uate this complexity of datasets. We provide evidence for their applicability on five different disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose a time-based, metric-aware method for developing datasets in a systematic and semi-automated manner, as well as an event-based QA task. 1 Introduction Semantic processing defines a relation between natural language and a representation of a world it refers to. A challenging property of natural language is the time-bound complex interaction between lexical expressions and world meanings. We use meaning in this paper as an umbrella term for both concepts and (event and entity) instances, and lexical expression as a common term for both lemmas and surface forms. We can define this interaction as a set of relations, both sense relations and referential relations, that exists within a language community in a certain period of time, e.g. one or a few generations. The people belonging to these generations share one language system that changes relatively slowly but during their lives there are many rapidly changing situations in the world that make certain meanings and expressions dominant and others not. Likewise, we expect that a generation uses a certain set of lexical expressions out of the available set in relation to a set of meanings that balances the trade-off between learning many expressions and resolving extreme ambiguity of a small set of expressions. The task of interpreting lexical expressions as meanings, known as disambiguation, has been addressed by the NLP community following a “divide & conquer” strategy that mostly ignores this complex time- bound relation. Over the years, this resulted in numerous separate disambiguation tasks each with a specific set of datasets restricted to a small bandwidth with respect to the dynamics of the world and the large scope of the possible meanings that lexical expressions can have. By dividing the problem into different tasks on relatively small datasets, researchers can focus on specific subproblems and have their efforts evaluated in a straightforward manner. Datasets have been developed independently for each task, intended as a test bench to evaluate the accuracy and applicability of the proposed systems. Official evaluation scripts have been created for most datasets to enable a fair comparison across systems. The downside of this practice is that task integration …

Content truncated to 500 words