--- a/writeup/techreport.tex	Mon May 03 18:38:27 2010 -0400
+++ b/writeup/techreport.tex	Mon May 03 18:38:58 2010 -0400
@@ -31,7 +31,10 @@
 We find that the SDA outperforms its
 shallow counterpart, an ordinary Multi-Layer Perceptron,
 and that it is better able to take advantage of the additional
-generated data.
+generated data, as well as better able to take advantage of
+training from more classes than those of interest in the end.
+In fact, we find that the SDA reaches human performance as
+estimated by the Amazon Mechanical Turk on the NIST test characters.
 \end{abstract}
 
 \section{Introduction}
@@ -263,8 +266,25 @@
 
 \begin{itemize}
 \item {\bf NIST}
+The NIST Special Database 19 (NIST19) is a very widely used dataset for training and testing OCR systems. 
+The dataset is composed with over 800 000 digits and characters (upper and lower cases), with hand checked classifications,
+extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes 
+corresponding to "0"-"9","A"-"Z" and "a"-"z". The dataset contains 8 series of different complexity. 
+The fourth series, $hsf_4$, experimentally recognized to be the most difficult one for classification task is recommended 
+by NIST as testing set and is used in our work for that purpose.
+The performances reported by previous work on that dataset mostly use only the digits.
+Here we use the whole classes both in the training and testing phase.   
+
+
 \item {\bf Fonts}
 \item {\bf Captchas}
+The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for 
+generating characters of the same format as the NIST dataset. The core of this data source is composed with a random character
+generator and various kinds of tranformations similar to those described in the previous sections. 
+In order to increase the variability of the data generated, different fonts are used for generating the characters. 
+Transformations (slant, distorsions, rotation, translation) are applied to each randomly generated character with a complexity
+depending on the value of the complexity parameter provided by the user of the data source. Two levels of complexity are 
+allowed and can be controlled via an easy to use facade class.    
 \item {\bf OCR data}
 \end{itemize}
 
@@ -323,21 +343,45 @@
 
 \section{Experimental Results}
 
-\subsection{SDA vs MLP}
+\subsection{SDA vs MLP vs Humans}
 
+We compare here the best MLP (according to validation set error) that we found against
+the best SDA (again according to validation set error), along with a precise estimate
+of human performance obtained via Amazon's Mechanical Turk (AMT)
+service\footnote{http://mturk.com}. AMT users are paid small amounts
+of money to perform tasks for which human intelligence is required.
+Mechanical Turk has been used extensively in natural language
+processing \cite{SnowEtAl2008} and vision
+\cite{SorokinAndForsyth2008,whitehill09}. AMT users where presented
+with 10 character images and asked to type 10 corresponding ascii
+characters. Hence they were forced to make a hard choice among the
+62 character classes. Three users classified each image, allowing
+to estimate inter-human variability (shown as +/- in parenthesis below).
+
+\begin{table}
+\caption{Overall comparison of error rates on 62 character classes (10 digits +
+26 lower + 26 upper), except for last columns -- digits only, between deep architecture with pre-training
+(SDA=Stacked Denoising Autoencoder) and ordinary shallow architecture 
+(MLP=Multi-Layer Perceptron). }
+\label{tab:sda-vs-mlp-vs-humans}
 \begin{center}
-\begin{tabular}{lcc}
-      & train w/   & train w/    \\
-      & NIST       & P07 + NIST  \\ \hline 
-SDA   &            &             \\ \hline 
-MLP   &            &             \\ \hline 
+\begin{tabular}{|l|r|r|r|r|} \hline
+      & NIST test & NISTP test & P07 test  & NIST test digits   \\ \hline
+Humans&            &           &   & \\ \hline 
+SDA   &            &           &  &\\ \hline 
+MLP   &            &           &  & \\ \hline 
 \end{tabular}
 \end{center}
+\end{table}
 
 \subsection{Perturbed Training Data More Helpful for SDAE}
 
 \subsection{Training with More Classes than Necessary}
 
+As previously seen, the SDA is better able to benefit from the transformations applied to the data than the MLP. We are now training SDAs and MLPs on single classes from NIST (respectively digits, lower case characters and upper case characters), to compare the test results with those from models trained on the entire NIST database (per-class test error, with an a priori on the desired class). The goal is to find out if training the model with more classes than necessary reduces the test error on a single class, as opposed to training it only with the desired class. We use a single hidden layer MLP with 1000 hidden units, and a SDA with 3 hidden layers (1000 hidden units per layer), pre-trained and fine-tuned on NIST.
+
+Our results show that the MLP only benefits from a full NIST training on digits, and the test error is only 5\% smaller than a digits-specialized MLP. On the other hand, the SDA always gives better results when it is trained with the entire NIST database, compared to its specialized counterparts (with upper case character, the test errors is 12\% smaller, 27\% smaller on digits, and 15\% smaller on lower case characters).
+
 \section{Conclusions}
 
 \bibliography{strings,ml,aigaion,specials}