On the Role of Supervision in Unsupervised Constituency Parsing
29703Authors
nan
Abstract
AbstractWe analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing F1 score on the Wall Street Journal (WSJ) development set (1,700 sentences). We introduce strong baselines for them, by training an existing supervised parsing model (Kitaev and Klein, 2018) on the same labeled examples they access. When training on the 1,700 examples, or even when using only 50 examples for training and 5 for development, such a few-shot parsing approach can outperform all the unsupervised parsing methods by a significant margin. Few-shot parsing can be further improved by a simple data augmentation method and self-training. This suggests that, in order to arrive at fair conclusions, we should carefully consider the amount of labeled data used for model development. We propose two protocols for future work on unsupervised parsing: (i) use fully unsupervised criteria for hyperparameter tuning and model selection; (ii) use as few labeled examples as possible for model development, and compare to few-shot parsing trained on the same labeled examples.
Keywords
Full Content
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 72–81, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 72 Statistical Power and Translationese in Machine Translation Evaluation Yvette Graham ADAPT Centre Trinity College Dublin ygraham@tcd.ie Barry Haddow School of Informatics University of Edinburgh bhaddow@inf.ed.ac.uk Philipp Koehn Dept of Computer Science Johns Hopkins University phi@jhu.edu Abstract The term translationese has been used to de- scribe features of translated text, and in this pa- per, we provide detailed analysis of potential adverse effects of translationese on machine translation evaluation. Our analysis shows differences in conclusions drawn from evalu- ations that include translationese in test data compared to experiments that tested only with text originally composed in that language. For this reason we recommend that reverse-created test data be omitted from future machine trans- lation test sets. In addition, we provide a re- evaluation of a past machine translation eval- uation claiming human-parity of MT. One im- portant issue not previously considered is sta- tistical power of significance tests applied to comparison of human and machine translation. Since the very aim of past evaluations was the investigation of ties between human and MT systems, power analysis is of particular impor- tance, to avoid, for example, claims of human parity simply corresponding to Type II error resulting from the application of a low pow- ered test. We provide detailed analysis of tests used in such evaluations to provide an indica- tion of a suitable minimum sample size for fu- ture studies. 1 Introduction Human-translated text is thought to display features that deviate to some degree from those of text orig- inally composed in that language. Baker (1993) report that translated text can: be more explicit than the original source, less ambiguous, simplified (lexical, syntactically and stylistically); display a preference for conventional grammaticality; avoid repetition; exaggerate target language features; as well as display features of the source language. The term translationese is often used to describe the presence of such phenomena in translated text. Standard evaluation protocol in Machine Trans- lation (MT) comprises system tests on a sample of human-translated text. Since creating this human- translated text is expensive, re-use of test sets for both directions of translation is commonplace, without regard to whether source or target contain features of translationese. For example, translation shared tasks at the Conference on Machine Trans- lation (WMT) (Bojar et al., 2018) generally test translation between a given language pair with two portions of data combined to make up the overall test set. Portion (a) of the test data (accounting for approximately 50% of sentences) is made up of text that originated in Chinese that was human- translated into English, while portion (b) (i.e. the remaining 50%), was translated in the opposite direction, originating in English with manual trans- lation into Chinese. The motivation for creating the test data in this way is to create test sets for both directions simultaneously (so at no extra cost).1 Although translationese has been cited as a likely confound …
Content truncated to 500 words
Labeled Text
AbstractWe analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing F1 score on the Wall Street Journal (WSJ) development set (1,700 sentences). We introduce strong baselines for them, by training an existing supervised parsing model (Kitaev and Klein, 2018) on the same labeled examples they access. When training on the 1,700 examples, or even when using only 50 examples for training and 5 for development, such a few-shot parsing approach can outperform all the unsupervised parsing methods by a significant margin. Few-shot parsing can be further improved by a simple data augmentation method and self-training. This suggests that, in order to arrive at fair conclusions, we should carefully consider the amount of labeled data used for model development. We propose two protocols for future work on unsupervised parsing: (i) use fully unsupervised criteria for hyperparameter tuning and model selection; (ii) use as few labeled examples as possible for model development, and compare to few-shot parsing trained on the same labeled examples.
Publication Details
- Paper ID:
- 29703
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan