# HG changeset patch # User Yoshua Bengio # Date 1294600403 18000 # Node ID 14ba0120baff11c0a4277380bf0a8a0df409e8d5 # Parent b0cdd200b2bdcdf935fcc8912116f3ebae618528 review response changes diff -r b0cdd200b2bd -r 14ba0120baff writeup/aistats_review_response.txt --- a/writeup/aistats_review_response.txt Sun Jan 09 12:13:45 2011 -0500 +++ b/writeup/aistats_review_response.txt Sun Jan 09 14:13:23 2011 -0500 @@ -51,7 +51,20 @@ true for many kinds of noises, but not for geometric transformations and deformations. -* Human labeling: +* Human labeling: We controlled noise in the labelling process by (1) +requiring AMT workers with a higher than normal average of accepted +responses (>95%) on other tasks (2) discarding responses that were not +complete (10 predictions) (3) discarding responses for which for which the +time to predict was smaller than 3 seconds for NIST (the mean response time +was 20 seconds) and 6 seconds seconds for NISTP (average response time of +45 seconds) (4) discarding responses which were obviously wrong (10 +identical ones, or "12345..."). Overall, after such filtering, we kept +approximately 95% of the AMT workers' responses. We thank the reviewer for +the suggestion about multi-stage questionnaires, we will definitely +consider this as an option next time we perform this experiment. However, +to be fair, if we were to do so, we should also consider the same +multi-stage decision process for the machine learning algorithms as well. + * Size of labeled set: