--- a/writeup/aistats_review_response.txt	Sun Jan 09 12:13:45 2011 -0500
+++ b/writeup/aistats_review_response.txt	Sun Jan 09 14:13:23 2011 -0500
@@ -51,7 +51,20 @@
 true for many kinds of noises, but not for geometric transformations
 and deformations.
 
-* Human labeling: 
+* Human labeling: We controlled noise in the labelling process by (1)
+requiring AMT workers with a higher than normal average of accepted
+responses (>95%) on other tasks (2) discarding responses that were not
+complete (10 predictions) (3) discarding responses for which for which the
+time to predict was smaller than 3 seconds for NIST (the mean response time
+was 20 seconds) and 6 seconds seconds for NISTP (average response time of
+45 seconds) (4) discarding responses which were obviously wrong (10
+identical ones, or "12345..."). Overall, after such filtering, we kept
+approximately 95% of the AMT workers' responses. We thank the reviewer for
+the suggestion about multi-stage questionnaires, we will definitely
+consider this as an option next time we perform this experiment. However,
+to be fair, if we were to do so, we should also consider the same
+multi-stage decision process for the machine learning algorithms as well.
+
 
 * Size of labeled set: