ift6266: writeup/nips2010_submission.tex comparison

comparison writeup/nips2010_submission.tex @ 535:caf7769ca19c

typo

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Tue, 01 Jun 2010 22:25:35 -0400
parents	4d6493d171f6
children	84f42fe05594

comparison

equal deleted inserted replaced

-:4d6493d171f6
+:caf7769ca19c
 with 60~000 examples, and variants involving 10~000
 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want
 to focus here on the case of much larger training sets, from 10 times to
 to 1000 times larger.
-The first step in constructing the larger datasets is to sample from
+The first step in constructing the larger datasets (called NISTP and P07) is to sample from
 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
 and {\bf OCR data} (scanned machine printed characters). Once a character
 is sampled from one of these sources (chosen randomly), the pipeline of
 the transformations and/or noise processes described in section \ref{s:perturbations}
 is applied to the image.
-We compare the best MLP against
+We compare the best MLPs against
-the best SDA (both models' hyper-parameters are selected to minimize the validation set error),
+the best SDAs (both models' hyper-parameters are selected to minimize the validation set error),
 along with a comparison against a precise estimate
 of human performance obtained via Amazon's Mechanical Turk (AMT)
 service (http://mturk.com).
 AMT users are paid small amounts
 of money to perform tasks for which human intelligence is required.
 from one of the 62 character classes.
 %\begin{itemize}
 %\item
 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
-\{651668 / 80000 / 82587\} \{training / validation / test} examples.
+\{651668 / 80000 / 82587\} \{training / validation / test\} examples.
 %\item
 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
-It has \{81920000 / 80000 / 20000\} \{training / validation / test} examples.
+It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
 %\item
 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
 except that we only apply
 transformations from slant to pinch. Therefore, the character is
 transformed but no additional noise is added to the image, giving images
 closer to the NIST dataset.
-It has \{81920000 / 80000 / 20000\} \{training / validation / test} examples.
+It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
 %\end{itemize}
 \vspace*{-1mm}
 \subsection{Models and their Hyperparameters}
 \vspace*{-1mm}
 comparable or better than RBMs in series of experiments
 \citep{VincentPLarochelleH2008}. During training, a Denoising
 Auto-Encoder is presented with a stochastically corrupted version
 of the input and trained to reconstruct the uncorrupted input,
 forcing the hidden units to represent the leading regularities in
-the data. Once it is trained, its hidden units activations can
+the data. Once it is trained, in a purely unsupervised way,
+its hidden units activations can
 be used as inputs for training a second one, etc.
 After this unsupervised pre-training stage, the parameters
 are used to initialize a deep MLP, which is fine-tuned by
 the same standard procedure used to train them (see previous section).
 The SDA hyper-parameters are the same as for the MLP, with the addition of the
 \section{Experimental Results}
 %\vspace*{-1mm}
 %\subsection{SDA vs MLP vs Humans}
 %\vspace*{-1mm}
+The models are either trained on NIST (MLP0 and SDA0),
+NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
+on either NIST, NISTP or P07, either on all 62 classes
+or only on the digits (considering only the outputs
+associated with digit classes).
 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
-comparing Humans, three MLPs (MLP0, MLP1, MLP2) and three SDAs (SDA0, SDA1,
+comparing Humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
 SDA2), along with the previous results on the digits NIST special database
 19 test set from the literature respectively based on ARTMAP neural
 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
 ~\citep{Milgram+al-2005}.  More detailed and complete numerical results
 (figures and tables, including standard errors on the error rates) can be
-found in Appendix I of the supplementary material.  The 3 kinds of model differ in the
+found in Appendix I of the supplementary material.
-training sets used: NIST only (MLP0,SDA0), NISTP (MLP1, SDA1), or P07
+The deep learner not only outperformed the shallow ones and
-(MLP2, SDA2). The deep learner not only outperformed the shallow ones and
 previously published performance (in a statistically and qualitatively
-significant way) but reaches human performance on both the 62-class task
+significant way) but when trained with perturbed data
+reaches human performance on both the 62-class task
 and the 10-class (digits) task.
 \begin{figure}[ht]
 \vspace*{-2mm}
 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}

Mercurial > ift6266

comparison writeup/nips2010_submission.tex @ 535:caf7769ca19c