Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 467:e0e57270b2af

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
refs
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Sat, 29 May 2010 16:50:03 -0400
parents 6205481bf33f
children d02d288257bf
rev   line source
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4 \usepackage{amsthm,amsmath,amssymb,bbold,bbm}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage{graphicx,subfigure}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{mlapa}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10 \title{Generating and Exploiting Perturbed and Multi-Task Handwritten Training Data for Deep Architectures}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13 \begin{document}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16 \maketitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18 \begin{abstract}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 Recent theoretical and empirical work in statistical machine learning has
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 demonstrated the importance of learning algorithms for deep
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21 architectures, i.e., function classes obtained by composing multiple
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 non-linear transformations. In the area of handwriting recognition,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 deep learning algorithms
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 had been evaluated on rather small datasets with a few tens of thousands
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 of examples. Here we propose a powerful generator of variations
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 of examples for character images based on a pipeline of stochastic
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 transformations that include not only the usual affine transformations
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 but also the addition of slant, local elastic deformations, changes
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29 in thickness, background images, color, contrast, occlusion, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 various types of pixel and spatially correlated noise.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31 We evaluate a deep learning algorithm (Stacked Denoising Autoencoders)
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 on the task of learning to classify digits and letters transformed
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 with this pipeline, using the hundreds of millions of generated examples
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 and testing on the full 62-class NIST test set.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 We find that the SDA outperforms its
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 shallow counterpart, an ordinary Multi-Layer Perceptron,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 and that it is better able to take advantage of the additional
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 generated data, as well as better able to take advantage of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 the multi-task setting, i.e.,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 training from more classes than those of interest in the end.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 In fact, we find that the SDA reaches human performance as
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 estimated by the Amazon Mechanical Turk on the 62-class NIST test characters.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 \end{abstract}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 \section{Introduction}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 Deep Learning has emerged as a promising new area of research in
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 statistical machine learning (see~\emcite{Bengio-2009} for a review).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 which consists of a chain of processing elements, each of which is associated with a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 different representation. In fact,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 those observed in the first two of these stages (in areas V1 and V2
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 of visual cortex)~\cite{HonglakL2008}.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
57 Processing images typically involves transforming the raw pixel data into
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 new {\bf representations} that can be used for analysis or classification.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59 For example, a principal component analysis representation linearly projects
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 the input image into a lower-dimensional feature space.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 Why learn a representation? Current practice in the computer vision
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 literature converts the raw pixels into a hand-crafted representation
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 (e.g.\ SIFT features~\cite{Lowe04}), but deep learning algorithms
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 tend to discover similar features in their first few
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 levels~\cite{HonglakL2008,ranzato-08,Koray-08,VincentPLarochelleH2008-very-small}.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 Learning increases the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 learning, often in an greedy layer-wise ``unsupervised pre-training''
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 stage~\cite{Bengio-2009}. One of these layer initialization techniques,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 applied here, is the Denoising
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 Auto-Encoder~(DEA)~\cite{VincentPLarochelleH2008-very-small}, which
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 useful for classification. The principle is that each layer starting from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86 the bottom is trained to encode their input (the output of the previous
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 layer) and try to reconstruct it from a corrupted version of it. After this
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88 unsupervised initialization, the stack of denoising auto-encoders can be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89 converted into a deep supervised feedforward neural network and trained by
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
92 In this paper we ask the following questions:
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
93 \begin{enumerate}
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
94 \item Do the good results previously obtained with deep architectures on the
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
95 MNIST digits generalize to the setting of a much larger and richer (but similar)
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
96 dataset, the NIST special database 19, with 62 classes and around 800k examples?
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
97 \item To what extent does the perturbation of input images (e.g. adding
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
98 noise, affine transformations, background images) make the resulting
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
99 classifier better not only on similarly perturbed images but also on
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
100 the {\em original clean examples}?
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
101 \item Do deep architectures benefit more from such {\em out-of-distribution}
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
102 examples, i.e. do they benefit more from the self-taught learning~\cite{RainaR2007} framework?
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
103 \item Similarly, does the feature learning step in deep learning algorithms benefit more
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
104 training with similar but different classes (i.e. a multi-task learning scenario) than
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
105 a corresponding shallow and purely supervised architecture?
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
106 \end{enumerate}
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
107 The experimental results presented here provide positive evidence towards all of these questions.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
109 \section{Perturbation and Transformation of Character Images}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
111 This section describes the different transformations we used to stochastically
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
112 transform source images in order to obtain data. More details can
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
113 be found in this technical report~\cite{ift6266-tr-anonymous}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
115 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
117 amount of deformation or noise introduced.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
119 There are two main parts in the pipeline. The first one,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
120 from slant to pinch below, performs transformations. The second
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
121 part, from blur to contrast, adds different kinds of noise.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
123 {\large\bf Transformations}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
124 {\bf Slant}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
125 We mimic slant by shifting each row of the image
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
126 proportionnaly to its height: $shift = round(slant \times height)$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 The $slant$ coefficient can be negative or positive with equal probability
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
128 and its value is randomly sampled according to the complexity level:
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
129 e $slant \sim U[0,complexity]$, so the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130 maximum displacement for the lowest or highest pixel line is of
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
131 $round(complexity \times 32)$.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
132 {\bf Thickness}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
133 Morpholigical operators of dilation and erosion~\cite{Haralick87,Serra82}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
134 are applied. The neighborhood of each pixel is multiplied
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
135 element-wise with a {\em structuring element} matrix.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
136 The pixel value is replaced by the maximum or the minimum of the resulting
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
137 matrix, respectively for dilation or erosion. Ten different structural elements with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
138 increasing dimensions (largest is $5\times5$) were used. For each image,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
139 randomly sample the operator type (dilation or erosion) with equal probability and one structural
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 element from a subset of the $n$ smallest structuring elements where $n$ is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
142 for erosion. A neutral element is always present in the set, and if it is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
143 chosen no transformation is applied. Erosion allows only the six
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 smallest structural elements because when the character is too thin it may
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
145 be completely erased.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
146 {\bf Affine Transformations}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
147 A $2 \times 3$ affine transform matrix (with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
148 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
149 Each pixel $(x,y)$ of the output image takes the value of the pixel
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
150 nearest to $(ax+by+c,dx+ey+f)$ in the input image. This
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
151 produces scaling, translation, rotation and shearing.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
152 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 forbid important rotations (not to confuse classes) but to give good
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
154 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
155 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
156 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
157 complexity]$.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
158 {\bf Local Elastic Deformations}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
159 This filter induces a "wiggly" effect in the image, following~\cite{SimardSP03},
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
160 which provides more details.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
161 Two "displacements" fields are generated and applied, for horizontal
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
162 and vertical displacements of pixels.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163 To generate a pixel in either field, first a value between -1 and 1 is
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
164 chosen from a uniform distribution. Then all the pixels, in both fields, are
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 multiplied by a constant $\alpha$ which controls the intensity of the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
166 displacements (larger $\alpha$ translates into larger wiggles).
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
167 Each field is convoluted with a Gaussian 2D kernel of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
168 standard deviation $\sigma$. Visually, this results in a blur.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
170 \sqrt[3]{complexity}$.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
171 {\bf Pinch}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
172 This GIMP filter is named "Whirl and
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
173 pinch", but whirl was set to 0. A pinch is ``similar to projecting the image onto an elastic
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
174 surface and pressing or pulling on the center of the surface''~\cite{GIMP-manual}.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
175 For a square input image, think of drawing a circle of
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177 that disk (region inside circle) will have its value recalculated by taking
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178 the value of another "source" pixel in the original image. The position of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 that source pixel is found on the line thats goes through $C$ and $P$, but
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
184 around the (non-integer) source position thus found.
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
185 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
187 {\large\bf Injecting Noise}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
188 {\bf Motion Blur}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
189 This GIMP filter is a ``linear motion blur'' in GIMP
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
190 terminology, with two parameters, $length$ and $angle$. The value of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
191 a pixel in the final image is the approximately mean value of the $length$ first pixels
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
192 found by moving in the $angle$ direction.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
193 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
194 {\bf Occlusion}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
195 This filter selects a random rectangle from an {\em occluder} character
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
196 images and places it over the original {\em occluded} character
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
197 image. Pixels are combined by taking the max(occluder,occluded),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
198 closer to black. The corners of the occluder The rectangle corners
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
199 are sampled so that larger complexity gives larger rectangles.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
200 The destination position in the occluded image are also sampled
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
201 according to a normal distribution (see more details in~\cite{ift6266-tr-anonymous}.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
202 It has has a probability of not being applied at all of 60\%.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
203 {\bf Pixel Permutation}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
204 This filter permutes neighbouring pixels. It selects first
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 sequentially exchanged to one other pixel in its $V4$ neighbourhood. Number
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 of exchanges to the left, right, top, bottom are equal or does not differ
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 from more than 1 if the number of selected pixels is not a multiple of 4.
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
209 It has has a probability of not being applied at all of 80\%.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
210 {\bf Gaussian Noise}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 This filter simply adds, to each pixel of the image independently, a
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
212 noise $\sim Normal(0(\frac{complexity}{10})^2)$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
213 It has has a probability of not being applied at all of 70\%.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
214 {\bf Background Images}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 Following~\cite{Larochelle-jmlr-2009}, this transformation adds a random
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 background behind the letter. The background is chosen by first selecting,
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
217 at random, an image from a set of images. Then a 32$\times$32 subregion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
218 of that image is chosen as the background image (by sampling position
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
219 uniformly while making sure not to cross image borders).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
220 To combine the original letter image and the background image, contrast
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
221 adjustments are made. We first get the maximal values (i.e. maximal
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
222 intensity) for both the original image and the background image, $maximage$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
223 and $maxbg$. We also have a parameter $contrast \sim U[complexity, 1]$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
224 Each background pixel value is multiplied by $\frac{max(maximage -
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
225 contrast, 0)}{maxbg}$ (higher contrast yield darker
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
226 background). The output image pixels are max(background,original).\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
227 {\bf Salt and Pepper Noise}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
228 This filter adds noise $\sim U[0,1]$ to random subsets of pixels.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
229 The number of selected pixels is $0.2 \times complexity$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
230 This filter has a probability of not being applied at all of 75\%.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
231 {\bf Spatially Gaussian Noise}\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
232 Different regions of the image are spatially smoothed.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
233 The image is convolved with a symmetric Gaussian kernel of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
234 size and variance choosen uniformly in the ranges $[12,12 + 20 \times
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
235 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
236 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
237 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
238 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
239 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
240 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
241 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
242 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
243 image \times mask}{mask+1}$.
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
244 This filter has a probability of not being applied at all of 75\%.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
245 {\bf Scratches}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
246 The scratches module places line-like white patches on the image. The
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
247 lines are heavily transformed images of the digit "1" (one), chosen
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
248 at random among five thousands such 1 images. The 1 image is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
249 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
250 complexity)^2$, using bicubic interpolation,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
251 Two passes of a greyscale morphological erosion filter
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
252 are applied, reducing the width of the line
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
253 by an amount controlled by $complexity$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
254 This filter is only applied only 15\% of the time. When it is applied, 50\%
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
255 of the time, only one patch image is generated and applied. In 30\% of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
256 cases, two patches are generated, and otherwise three patches are
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
257 generated. The patch is applied by taking the maximal value on any given
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
258 patch or the original image, for each of the 32x32 pixel locations.\\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
259 {\bf Color and Contrast Changes}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
260 This filter changes the constrast and may invert the image polarity (white
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
261 on black to black on white). The contrast $C$ is defined here as the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
262 difference between the maximum and the minimum pixel value of the image.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
263 Contrast $\sim U[1-0.85 \times complexity,1]$ (so constrast $\geq 0.15$).
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
264 The image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
265 polarity is inverted with $0.5$ probability.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
266
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
267
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
268 \begin{figure}[h]
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
269 \resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}\\
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
270 \caption{Illustration of the pipeline of stochastic
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
271 transformations applied to the image of a lower-case t
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
272 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
273 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
274 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
275 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
276 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
277 \end{figure}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
278
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
279
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
280 \section{Experimental Setup}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
281
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
282 \subsection{Training Datasets}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
283
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
284 \subsubsection{Data Sources}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
285
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
286 \begin{itemize}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
287 \item {\bf NIST}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
288 The NIST Special Database 19 (NIST19) is a very widely used dataset for training and testing OCR systems.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
289 The dataset is composed with 8????? digits and characters (upper and lower cases), with hand checked classifications,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
290 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
291 corresponding to "0"-"9","A"-"Z" and "a"-"z". The dataset contains 8 series of different complexity.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
292 The fourth series, $hsf_4$, experimentally recognized to be the most difficult one for classification task is recommended
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
293 by NIST as testing set and is used in our work for that purpose. It contains 82600 examples,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
294 while the training and validation sets (which have the same distribution) contain XXXXX and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
295 XXXXX examples respectively.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
296 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
297 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
298 useful to estimate the effect of a multi-task setting.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
299 Note that the distribution of the classes in the NIST training and test sets differs
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
300 substantially, with relatively many more digits in the test set, and uniform distribution
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
301 of letters in the test set, not in the training set (more like the natural distribution
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
302 of letters in text).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
303
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
304 \item {\bf Fonts} TODO!!!
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
305
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
306 \item {\bf Captchas}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
307 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
308 generating characters of the same format as the NIST dataset. The core of this data source is composed with a random character
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
309 generator and various kinds of tranformations similar to those described in the previous sections.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
310 In order to increase the variability of the data generated, different fonts are used for generating the characters.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
311 Transformations (slant, distorsions, rotation, translation) are applied to each randomly generated character with a complexity
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
312 depending on the value of the complexity parameter provided by the user of the data source. Two levels of complexity are
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
313 allowed and can be controlled via an easy to use facade class.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
314 \item {\bf OCR data}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
315 \end{itemize}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
316
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
317 \subsubsection{Data Sets}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
318 \begin{itemize}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
319 \item {\bf NIST} This is the raw NIST special database 19.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
320 \item {\bf P07}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
321 The dataset P07 is sampled with our transformation pipeline with a complexity parameter of $0.7$.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
322 For each new exemple to generate, we choose one source with the following probability: $0.1$ for the fonts,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
323 $0.25$ for the captchas, $0.25$ for OCR data and $0.4$ for NIST. We apply all the transformations in their order
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
324 and for each of them we sample uniformly a complexity in the range $[0,0.7]$.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
325 \item {\bf NISTP} NISTP is equivalent to P07 (complexity parameter of $0.7$ with the same sources proportion)
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
326 except that we only apply
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
327 transformations from slant to pinch. Therefore, the character is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
328 transformed but no additionnal noise is added to the image, giving images
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
329 closer to the NIST dataset.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
330 \end{itemize}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
331
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
332 \subsection{Models and their Hyperparameters}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
333
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
334 \subsubsection{Multi-Layer Perceptrons (MLP)}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
335
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
336 An MLP is a family of functions that are described by stacking layers of of a function similar to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
337 $$g(x) = \tanh(b+Wx)$$
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
338 The input, $x$, is a $d$-dimension vector.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
339 The output, $g(x)$, is a $m$-dimension vector.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
340 The parameter $W$ is a $m\times d$ matrix and is called the weight matrix.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
341 The parameter $b$ is a $m$-vector and is called the bias vector.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
342 The non-linearity (here $\tanh$) is applied element-wise to the output vector.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
343 Usually the input is referred to a input layer and similarly for the output.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
344 You can of course chain several such functions to obtain a more complex one.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
345 Here is a common example
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
346 $$f(x) = c + V\tanh(b+Wx)$$
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
347 In this case the intermediate layer corresponding to $\tanh(b+Wx)$ is called a hidden layer.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
348 Here the output layer does not have the same non-linearity as the hidden layer.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
349 This is a common case where some specialized non-linearity is applied to the output layer only depending on the task at hand.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
350
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
351 If you put 3 or more hidden layers in such a network you obtain what is called a deep MLP.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
352 The parameters to adapt are the weight matrix and the bias vector for each layer.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
353
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
354 \subsubsection{Stacked Denoising Auto-Encoders (SDAE)}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
355 \label{SdA}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
356
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
357 Auto-encoders are essentially a way to initialize the weights of the network to enable better generalization.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
358 This is essentially unsupervised training where the layer is made to reconstruct its input through and encoding and decoding phase.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
359 Denoising auto-encoders are a variant where the input is corrupted with random noise but the target is the uncorrupted input.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
360 The principle behind these initialization methods is that the network will learn the inherent relation between portions of the data and be able to represent them thus helping with whatever task we want to perform.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
361
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
362 An auto-encoder unit is formed of two MLP layers with the bottom one called the encoding layer and the top one the decoding layer.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
363 Usually the top and bottom weight matrices are the transpose of each other and are fixed this way.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
364 The network is trained as such and, when sufficiently trained, the MLP layer is initialized with the parameters of the encoding layer.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
365 The other parameters are discarded.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
366
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
367 The stacked version is an adaptation to deep MLPs where you initialize each layer with a denoising auto-encoder starting from the bottom.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
368 During the initialization, which is usually called pre-training, the bottom layer is treated as if it were an isolated auto-encoder.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
369 The second and following layers receive the same treatment except that they take as input the encoded version of the data that has gone through the layers before it.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
370 For additional details see \cite{vincent:icml08}.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
371
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
372 \section{Experimental Results}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
373
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
374 \subsection{SDA vs MLP vs Humans}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
375
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
376 We compare here the best MLP (according to validation set error) that we found against
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
377 the best SDA (again according to validation set error), along with a precise estimate
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
378 of human performance obtained via Amazon's Mechanical Turk (AMT)
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
379 service\footnote{http://mturk.com}. AMT users are paid small amounts
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
380 of money to perform tasks for which human intelligence is required.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
381 Mechanical Turk has been used extensively in natural language
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
382 processing \cite{SnowEtAl2008} and vision
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
383 \cite{SorokinAndForsyth2008,whitehill09}. AMT users where presented
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
384 with 10 character images and asked to type 10 corresponding ascii
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
385 characters. Hence they were forced to make a hard choice among the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
386 62 character classes. Three users classified each image, allowing
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
387 to estimate inter-human variability (shown as +/- in parenthesis below).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
388
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
389 \begin{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
390 \caption{Overall comparison of error rates ($\pm$ std.err.) on 62 character classes (10 digits +
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
391 26 lower + 26 upper), except for last columns -- digits only, between deep architecture with pre-training
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
392 (SDA=Stacked Denoising Autoencoder) and ordinary shallow architecture
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
393 (MLP=Multi-Layer Perceptron). The models shown are all trained using perturbed data (NISTP or P07)
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
394 and using a validation set to select hyper-parameters and other training choices.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
395 \{SDA,MLP\}0 are trained on NIST,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
396 \{SDA,MLP\}1 are trained on NISTP, and \{SDA,MLP\}2 are trained on P07.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
397 The human error rate on digits is a lower bound because it does not count digits that were
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
398 recognized as letters. For comparison, the results found in the literature
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
399 on NIST digits classification using the same test set are included.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
400 \label{tab:sda-vs-mlp-vs-humans}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 \begin{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
402 \begin{tabular}{|l|r|r|r|r|} \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403 & NIST test & NISTP test & P07 test & NIST test digits \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 Humans& 18.2\% $\pm$.1\% & 39.4\%$\pm$.1\% & 46.9\%$\pm$.1\% & $>1.1\%$ \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
405 SDA0 & 23.7\% $\pm$.14\% & 65.2\%$\pm$.34\% & 97.45\%$\pm$.06\% & 2.7\% $\pm$.14\%\\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
406 SDA1 & 17.1\% $\pm$.13\% & 29.7\%$\pm$.3\% & 29.7\%$\pm$.3\% & 1.4\% $\pm$.1\%\\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
407 SDA2 & 18.7\% $\pm$.13\% & 33.6\%$\pm$.3\% & 39.9\%$\pm$.17\% & 1.7\% $\pm$.1\%\\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
408 MLP0 & 24.2\% $\pm$.15\% & 68.8\%$\pm$.33\% & 78.70\%$\pm$.14\% & 3.45\% $\pm$.15\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
409 MLP1 & 23.0\% $\pm$.15\% & 41.8\%$\pm$.35\% & 90.4\%$\pm$.1\% & 3.85\% $\pm$.16\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
410 MLP2 & 24.3\% $\pm$.15\% & 46.0\%$\pm$.35\% & 54.7\%$\pm$.17\% & 4.85\% $\pm$.18\% \\ \hline
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
411 \cite{Granger+al-2007} & & & & 4.95\% $\pm$.18\% \\ \hline
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
412 \cite{Cortes+al-2000} & & & & 3.71\% $\pm$.16\% \\ \hline
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
413 \cite{Oliveira+al-2002} & & & & 2.4\% $\pm$.13\% \\ \hline
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
414 \cite{Migram+al-2005} & & & & 2.1\% $\pm$.12\% \\ \hline
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
415 \end{tabular}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
416 \end{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
417 \end{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
418
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
419 \subsection{Perturbed Training Data More Helpful for SDAE}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
420
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
421 \begin{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
422 \caption{Relative change in error rates due to the use of perturbed training data,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
423 either using NISTP, for the MLP1/SDA1 models, or using P07, for the MLP2/SDA2 models.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
424 A positive value indicates that training on the perturbed data helped for the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
425 given test set (the first 3 columns on the 62-class tasks and the last one is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
426 on the clean 10-class digits). Clearly, the deep learning models did benefit more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
427 from perturbed training data, even when testing on clean data, whereas the MLP
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
428 trained on perturbed data performed worse on the clean digits and about the same
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
429 on the clean characters. }
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
430 \label{tab:sda-vs-mlp-vs-humans}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
431 \begin{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
432 \begin{tabular}{|l|r|r|r|r|} \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
433 & NIST test & NISTP test & P07 test & NIST test digits \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
434 SDA0/SDA1-1 & 38\% & 84\% & 228\% & 93\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
435 SDA0/SDA2-1 & 27\% & 94\% & 144\% & 59\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
436 MLP0/MLP1-1 & 5.2\% & 65\% & -13\% & -10\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
437 MLP0/MLP2-1 & -0.4\% & 49\% & 44\% & -29\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
438 \end{tabular}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
439 \end{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
440 \end{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
441
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
442
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
443 \subsection{Multi-Task Learning Effects}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
444
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
445 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
446 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
447 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
448 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
449 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
450 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
451 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
452 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
453 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
454 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
455 shallow learner.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
456
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
457 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
458 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
459 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
460
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
461 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
462 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
463 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
464 On the other hand the SDA benefitted from the multi-task setting, with relative
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
465 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
466 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
467
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
468 \begin{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
469 \caption{Test error rates and relative change in error rates due to the use of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
470 a multi-task setting, i.e., training on each task in isolation vs training
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
471 for all three tasks together, for MLPs vs SDAs. The SDA benefits much
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
472 more from the multi-task setting. All experiments on only on the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
473 unperturbed NIST data, using validation error for model selection.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
474 Relative improvement is 1 - single-task error / multi-task error.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
475 \label{tab:multi-task}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
476 \begin{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
477 \begin{tabular}{|l|r|r|r|} \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
478 & single-task & multi-task & relative \\
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
479 & setting & setting & improvement \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
480 MLP-digits & 3.77\% & 3.99\% & 5.6\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
481 MLP-lower & 17.4\% & 16.8\% & -4.1\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
482 MLP-upper & 7.84\% & 7.54\% & -3.6\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
483 SDA-digits & 2.6\% & 3.56\% & 27\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
484 SDA-lower & 12.3\% & 14.4\% & 15\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
485 SDA-upper & 5.93\% & 6.78\% & 13\% \\ \hline
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
486 \end{tabular}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
487 \end{center}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
488 \end{table}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
489
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
490 \section{Conclusions}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
491
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
492 \bibliography{strings,ml,aigaion,specials}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
493 \bibliographystyle{mlapa}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
494 %\bibliographystyle{apalike}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
495
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
496 \end{document}