# HG changeset patch # User Yoshua Bengio # Date 1275415036 14400 # Node ID b8e33d3d7f650e6e251ef0ceb6344db5504f91a2 # Parent 8bf07979b8ba1b6665e63a8b069157c40f3a8646# Parent a41a8925be70e626eba5c5edee982baf6e54ba7a merge diff -r 8bf07979b8ba -r b8e33d3d7f65 writeup/nips2010_submission.tex --- a/writeup/nips2010_submission.tex Tue Jun 01 13:56:56 2010 -0400 +++ b/writeup/nips2010_submission.tex Tue Jun 01 13:57:16 2010 -0400 @@ -20,7 +20,7 @@ Recent theoretical and empirical work in statistical machine learning has demonstrated the importance of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple - non-linear transformations. The self-taught learning (exploiting unlabeled + non-linear transformations. Self-taught learning (exploiting unlabeled examples or examples from other distributions) has already been applied to deep learners, but mostly to show the advantage of unlabeled examples. Here we explore the advantage brought by {\em out-of-distribution @@ -74,8 +74,8 @@ performed similarly or better than previously proposed Restricted Boltzmann Machines in terms of unsupervised extraction of a hierarchy of features useful for classification. The principle is that each layer starting from -the bottom is trained to encode their input (the output of the previous -layer) and try to reconstruct it from a corrupted version of it. After this +the bottom is trained to encode its input (the output of the previous +layer) and to reconstruct it from a corrupted version of it. After this unsupervised initialization, the stack of denoising auto-encoders can be converted into a deep supervised feedforward neural network and fine-tuned by stochastic gradient descent. @@ -91,6 +91,8 @@ (but see~\citep{CollobertR2008}). In particular the {\em relative advantage} of deep learning for this settings has not been evaluated. +% TODO: Explain why we care about this question. + In this paper we ask the following questions: %\begin{enumerate} @@ -115,7 +117,7 @@ a corresponding shallow and purely supervised architecture? %\end{enumerate} -The experimental results presented here provide positive evidence towards all of these questions. +Our experimental results provide evidence to support positive answers to all of these questions. \vspace*{-1mm} \section{Perturbation and Transformation of Character Images} @@ -583,13 +585,7 @@ \begin{figure}[h] \resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}\\ -\caption{Relative improvement in error rate due to self-taught learning. -Left: Improvement (or loss, when negative) -induced by out-of-distribution examples (perturbed data). -Right: Improvement (or loss, when negative) induced by multi-task -learning (training on all classes and testing only on either digits, -upper case, or lower-case). The deep learner (SDA) benefits more from -both self-taught learning scenarios, compared to the shallow MLP.} +\caption{Charts corresponding to tables 2 (left) and 3 (right), from Appendix I.} \label{fig:improvements-charts} \end{figure}