Mercurial > ift6266

comparison writeup/nips2010_submission.tex @ 482:ce69aa9204d8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
changement au titre et reecriture abstract
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Mon, 31 May 2010 13:59:11 -0400
parents 150203d2b5c3
children b9cdb464de5f
comparison
equal deleted inserted replaced
481:3e4290448eeb 482:ce69aa9204d8
5 \usepackage{algorithm,algorithmic} 5 \usepackage{algorithm,algorithmic}
6 \usepackage[utf8]{inputenc} 6 \usepackage[utf8]{inputenc}
7 \usepackage{graphicx,subfigure} 7 \usepackage{graphicx,subfigure}
8 \usepackage[numbers]{natbib} 8 \usepackage[numbers]{natbib}
9 9
10 \title{Generating and Exploiting Perturbed and Multi-Task Handwritten Training Data for Deep Architectures} 10 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
11 \author{The IFT6266 Gang} 11 \author{The IFT6266 Gang}
12 12
13 \begin{document} 13 \begin{document}
14 14
15 %\makeanontitle 15 %\makeanontitle
16 \maketitle 16 \maketitle
17 17
18 \begin{abstract} 18 \begin{abstract}
19 Recent theoretical and empirical work in statistical machine learning has 19 Recent theoretical and empirical work in statistical machine learning has
20 demonstrated the importance of learning algorithms for deep 20 demonstrated the importance of learning algorithms for deep
21 architectures, i.e., function classes obtained by composing multiple 21 architectures, i.e., function classes obtained by composing multiple
22 non-linear transformations. In the area of handwriting recognition, 22 non-linear transformations. The self-taught learning (exploitng unlabeled
23 deep learning algorithms 23 examples or examples from other distributions) has already been applied
24 had been evaluated on rather small datasets with a few tens of thousands 24 to deep learners, but mostly to show the advantage of unlabeled
25 of examples. Here we propose a powerful generator of variations 25 examples. Here we explore the advantage brought by {\em out-of-distribution
26 of examples for character images based on a pipeline of stochastic 26 examples} and show that {\em deep learners benefit more from them than a
27 transformations that include not only the usual affine transformations 27 corresponding shallow learner}, in the area
28 but also the addition of slant, local elastic deformations, changes 28 of handwritten character recognition. In fact, we show that they reach
29 in thickness, background images, color, contrast, occlusion, and 29 human-level performance on both handwritten digit classification and
30 various types of pixel and spatially correlated noise. 30 62-class handwritten character recognition. For this purpose we
31 We evaluate a deep learning algorithm (Stacked Denoising Autoencoders) 31 developed a powerful generator of stochastic variations and noise
32 on the task of learning to classify digits and letters transformed 32 processes character images, including not only affine transformations but
33 with this pipeline, using the hundreds of millions of generated examples 33 also slant, local elastic deformations, changes in thickness, background
34 and testing on the full 62-class NIST test set. 34 images, color, contrast, occlusion, and various types of pixel and
35 We find that the SDA outperforms its 35 spatially correlated noise. The out-of-distribution examples are
36 shallow counterpart, an ordinary Multi-Layer Perceptron, 36 obtained by training with these highly distorted images or
37 and that it is better able to take advantage of the additional 37 by including object classes different from those in the target test set.
38 generated data, as well as better able to take advantage of
39 the multi-task setting, i.e.,
40 training from more classes than those of interest in the end.
41 In fact, we find that the SDA reaches human performance as
42 estimated by the Amazon Mechanical Turk on the 62-class NIST test characters.
43 \end{abstract} 38 \end{abstract}
44 39
45 \section{Introduction} 40 \section{Introduction}
46 41
47 Deep Learning has emerged as a promising new area of research in 42 Deep Learning has emerged as a promising new area of research in
277 \end{figure} 272 \end{figure}
278 273
279 274
280 \begin{figure}[h] 275 \begin{figure}[h]
281 \resizebox{.99\textwidth}{!}{\includegraphics{images/transfo.png}}\\ 276 \resizebox{.99\textwidth}{!}{\includegraphics{images/transfo.png}}\\
282 \caption{Illustration of each transformation applied to the same image 277 \caption{Illustration of each transformation applied alone to the same image
283 of the upper-case h (upper-left image). first row (from left to rigth) : original image, slant, 278 of an upper-case h (top left). First row (from left to rigth) : original image, slant,
284 thickness, affine transformation, local elastic deformation; second row (from left to rigth) : 279 thickness, affine transformation, local elastic deformation; second row (from left to rigth) :
285 pinch, motion blur, occlusion, pixel permutation, gaussian noise; third row (from left to rigth) : 280 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to rigth) :
286 background image, salt and pepper noise, spatially gaussian noise, scratches, 281 background image, salt and pepper noise, spatially Gaussian noise, scratches,
287 color and contrast changes.} 282 color and contrast changes.}
288 \label{fig:transfo} 283 \label{fig:transfo}
289 \end{figure} 284 \end{figure}
290 285
291 286