Mechanical Turk-based Experiment vs Laboratory-based Experiment: A Case Study on the Comparison of Semantic Transparency Rating Data

Mechanical Turk-based Experiment vs Laboratory-based Experiment: A Case Study on the Comparison of Semantic Transparency Rating Data Shichang Wang, Chu-Ren Huang, Yao Yao, Angel Chan Department of Chinese and Bilingual Studies The Hong Kong Polytechnic University Hung Hom, Kowloon, Hong Kong shi-chang.wang@connect.polyu.hk {churen.huang, y.yao, angel.ws.chan}@polyu.edu.hk Abstract In this paper, we conducted semantic trans- parency rating experiments using both the traditional laboratory-based method and the crowdsourcing-based method. Then we com- pared the rating data obtained from these two experiments. We observed very strong cor- relation coefficients for both overall seman- tic transparency rating data and constituent se- mantic transparency data (rho > 0.9) which means the two experiments may yield com- parable data and crowdsourcing-based experi- ment is a feasible alternative to the laboratory- based experiment in linguistic studies. We also observed a scale shrinkage phenomenon in both experiments: the actual scale of the rat- ing results cannot cover the ideal scale [0, 1], both ends of the actual scale shrink towards the center. However, the scale shrinkage of the crowdsourcing-based experiment is stronger than that of the laboratory-based experiment, this makes the rating results obtained in these two experiments not directly comparable. In order to make the results directly compara- ble, we explored two data transformation al- gorithms, z-score transformation and adjusted normalization to unify the scales. We also in- vestigated the uncertainty of semantic trans- parency judgment among raters, we found that it had a regular relation with semantic trans- parency magnitude and this may further reveal a general cognitive mechanism of human judg- ment. 1 Introduction In experimental linguistic studies, researchers are frequently frustrated by the problem of linguistic data bottleneck which constantly limits the feasi- bility, efficiency, and reliability of various research projects. It’s caused by the practical difficulties of conducting traditional laboratory-based linguis- tic experiments. Firstly, it’s very difficult to obtain large samples using laboratory-based experiments for they are usually very time-consuming and expen- sive. In order to solve this problem, we need to find a more efficient and economic way to conduct lin- guistic experiments. Secondly, what’s more difficult is to recruit highly diverse subjects due to the diffi- culties in subject recruitment and the spacial limita- tions of laboratory-based experiments. As a result, researchers heavily and even blindly rely on rela- tively small sample size which is 30 or so (Sprouse, 2011) and the undergraduate subject pool. From the point of view of sampling, this is not a good prac- tice, since it raises the concern of external validity, i.e., the extent to which the experimental results can be generalized, because a small and homogeneous sample usually cannot be representative enough. In fact the external validity problem that results from using mainly undergraduate subjects is a typical one and has a dedicated term called the college sopho- more problem (Stanovich, 2007; Jackson, 2012). Al- though there are several responses to this criticism (Stanovich, 2007), the really convincing way to re- solve this problem is to use a more diverse subject pool. Mechanical Turk (MTurk) …

Content truncated to 500 words