Mercurial > ift6266

comparison writeup/aistats_review_response.txt @ 618:14ba0120baff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
review response changes
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Sun, 09 Jan 2011 14:13:23 -0500
parents b0cdd200b2bd
children ea31fee25147
comparison
equal deleted inserted replaced
616:b0cdd200b2bd 618:14ba0120baff
49 about it is to consider the symmetries involved. A corruption process should 49 about it is to consider the symmetries involved. A corruption process should
50 be such that swapping input for target should be very unlikely: this is 50 be such that swapping input for target should be very unlikely: this is
51 true for many kinds of noises, but not for geometric transformations 51 true for many kinds of noises, but not for geometric transformations
52 and deformations. 52 and deformations.
53 53
54 * Human labeling: 54 * Human labeling: We controlled noise in the labelling process by (1)
55 requiring AMT workers with a higher than normal average of accepted
56 responses (>95%) on other tasks (2) discarding responses that were not
57 complete (10 predictions) (3) discarding responses for which for which the
58 time to predict was smaller than 3 seconds for NIST (the mean response time
59 was 20 seconds) and 6 seconds seconds for NISTP (average response time of
60 45 seconds) (4) discarding responses which were obviously wrong (10
61 identical ones, or "12345..."). Overall, after such filtering, we kept
62 approximately 95% of the AMT workers' responses. We thank the reviewer for
63 the suggestion about multi-stage questionnaires, we will definitely
64 consider this as an option next time we perform this experiment. However,
65 to be fair, if we were to do so, we should also consider the same
66 multi-stage decision process for the machine learning algorithms as well.
67
55 68
56 * Size of labeled set: 69 * Size of labeled set:
57 70