annotate writeup/nips2010_submission.tex @ 518:460a4e78c9a4

merging is fun, merging is fun, merging is fun
author Dumitru Erhan <dumitru.erhan@gmail.com>
date Tue, 01 Jun 2010 11:15:37 -0700
parents 0a5945249f2b 092dae9a5040
children eaa595ea2402
rev   line source
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4 \usepackage{amsthm,amsmath,amssymb,bbold,bbm}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage{graphicx,subfigure}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
8 \usepackage[numbers]{natbib}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
10 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13 \begin{document}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16 \maketitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
18 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 \begin{abstract}
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
20 Recent theoretical and empirical work in statistical machine learning has
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
21 demonstrated the importance of learning algorithms for deep
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
22 architectures, i.e., function classes obtained by composing multiple
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
23 non-linear transformations. Self-taught learning (exploiting unlabeled
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
24 examples or examples from other distributions) has already been applied
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
25 to deep learners, but mostly to show the advantage of unlabeled
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
26 examples. Here we explore the advantage brought by {\em out-of-distribution
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
27 examples} and show that {\em deep learners benefit more from them than a
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
28 corresponding shallow learner}, in the area
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
29 of handwritten character recognition. In fact, we show that they reach
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
30 human-level performance on both handwritten digit classification and
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
31 62-class handwritten character recognition. For this purpose we
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
32 developed a powerful generator of stochastic variations and noise
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
33 processes character images, including not only affine transformations but
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
34 also slant, local elastic deformations, changes in thickness, background
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
35 images, grey level changes, contrast, occlusion, and various types of pixel and
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
36 spatially correlated noise. The out-of-distribution examples are
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
37 obtained by training with these highly distorted images or
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
38 by including object classes different from those in the target test set.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 \end{abstract}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
40 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 \section{Introduction}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
43 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 Deep Learning has emerged as a promising new area of research in
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
46 statistical machine learning (see~\citet{Bengio-2009} for a review).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 which consists of a chain of processing elements, each of which is associated with a
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
51 different representation of the raw visual input. In fact,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 those observed in the first two of these stages (in areas V1 and V2
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
54 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
55 more invariant to factors of variation (such as camera movement) in
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
56 higher layers~\citep{Goodfellow2009}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
57 Learning a hierarchy of features increases the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 learning, often in an greedy layer-wise ``unsupervised pre-training''
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
71 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 applied here, is the Denoising
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
73 Auto-Encoder~(DEA)~\citep{VincentPLarochelleH2008-very-small}, which
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76 useful for classification. The principle is that each layer starting from
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
77 the bottom is trained to encode its input (the output of the previous
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
78 layer) and to reconstruct it from a corrupted version of it. After this
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 unsupervised initialization, the stack of denoising auto-encoders can be
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
80 converted into a deep supervised feedforward neural network and fine-tuned by
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
83 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
84 of semi-supervised and multi-task learning: the learner can exploit examples
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
85 that are unlabeled and/or come from a distribution different from the target
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
86 distribution, e.g., from other classes that those of interest. Whereas
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
87 it has already been shown that deep learners can clearly take advantage of
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
88 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
89 and multi-task learning, not much has been done yet to explore the impact
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
90 of {\em out-of-distribution} examples and of the multi-task setting
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
91 (but see~\citep{CollobertR2008}). In particular the {\em relative
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
92 advantage} of deep learning for this settings has not been evaluated.
512
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
93 The hypothesis explored here is that a deep hierarchy of features
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
94 may be better able to provide sharing of statistical strength
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
95 between different regions in input space or different tasks,
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
96 as discussed in the conclusion.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
97
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
98 % TODO: why we care to evaluate this relative advantage
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
99
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
100 In this paper we ask the following questions:
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
101
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
102 %\begin{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
103 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
104 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
105 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
106 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
107
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
108 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
109 To what extent does the perturbation of input images (e.g. adding
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
110 noise, affine transformations, background images) make the resulting
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
111 classifiers better not only on similarly perturbed images but also on
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
112 the {\em original clean examples}?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
113
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
114 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
115 Do deep architectures {\em benefit more from such out-of-distribution}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
116 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
117
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
118 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
119 Similarly, does the feature learning step in deep learning algorithms benefit more
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
120 training with similar but different classes (i.e. a multi-task learning scenario) than
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
121 a corresponding shallow and purely supervised architecture?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
122 %\end{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
123
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
124 Our experimental results provide positive evidence towards all of these questions.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
125
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
126 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 \section{Perturbation and Transformation of Character Images}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
128 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
129
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
130 This section describes the different transformations we used to stochastically
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
131 transform source images in order to obtain data. More details can
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
132 be found in this technical report~\citep{ift6266-tr-anonymous}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
134 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
136 amount of deformation or noise introduced.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
138 There are two main parts in the pipeline. The first one,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
139 from slant to pinch below, performs transformations. The second
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
140 part, from blur to contrast, adds different kinds of noise.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
142 \begin{figure}[h]
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
143 \resizebox{.99\textwidth}{!}{\includegraphics{images/transfo.png}}\\
506
8bf07979b8ba desiderata
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 503
diff changeset
144 % TODO: METTRE LE NOM DE LA TRANSFO A COTE DE CHAQUE IMAGE
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
145 \caption{Illustration of each transformation applied alone to the same image
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
146 of an upper-case h (top left). First row (from left to right) : original image, slant,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
147 thickness, affine transformation (translation, rotation, shear),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
148 local elastic deformation; second row (from left to right) :
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
149 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to right) :
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
150 background image, salt and pepper noise, spatially Gaussian noise, scratches,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
151 grey level and contrast changes.}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
152 \label{fig:transfo}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
153 \end{figure}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
154
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
155 {\large\bf Transformations}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
156
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
157 \vspace*{2mm}
483
b9cdb464de5f pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 482
diff changeset
158
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
159 {\bf Slant.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
160 We mimic slant by shifting each row of the image
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
161 proportionally to its height: $shift = round(slant \times height)$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162 The $slant$ coefficient can be negative or positive with equal probability
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
163 and its value is randomly sampled according to the complexity level:
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
164 $slant \sim U[0,complexity]$, so the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 maximum displacement for the lowest or highest pixel line is of
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
166 $round(complexity \times 32)$.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
167 {\bf Thickness.}
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
168 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
169 are applied. The neighborhood of each pixel is multiplied
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
170 element-wise with a {\em structuring element} matrix.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
171 The pixel value is replaced by the maximum or the minimum of the resulting
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
172 matrix, respectively for dilation or erosion. Ten different structural elements with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
173 increasing dimensions (largest is $5\times5$) were used. For each image,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
174 randomly sample the operator type (dilation or erosion) with equal probability and one structural
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175 element from a subset of the $n$ smallest structuring elements where $n$ is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
177 for erosion. A neutral element is always present in the set, and if it is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
178 chosen no transformation is applied. Erosion allows only the six
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 smallest structural elements because when the character is too thin it may
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
180 be completely erased.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
181 {\bf Affine Transformations.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
182 A $2 \times 3$ affine transform matrix (with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
183 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
184 Each pixel $(x,y)$ of the output image takes the value of the pixel
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
185 nearest to $(ax+by+c,dx+ey+f)$ in the input image. This
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
186 produces scaling, translation, rotation and shearing.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
187 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 forbid important rotations (not to confuse classes) but to give good
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
189 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
190 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
191 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
192 complexity]$.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
193 {\bf Local Elastic Deformations.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
194 This filter induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
195 which provides more details.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
196 Two ``displacements'' fields are generated and applied, for horizontal
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
197 and vertical displacements of pixels.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 To generate a pixel in either field, first a value between -1 and 1 is
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
199 chosen from a uniform distribution. Then all the pixels, in both fields, are
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
200 multiplied by a constant $\alpha$ which controls the intensity of the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
201 displacements (larger $\alpha$ translates into larger wiggles).
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
202 Each field is convoluted with a Gaussian 2D kernel of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
203 standard deviation $\sigma$. Visually, this results in a blur.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
205 \sqrt[3]{complexity}$.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
206 {\bf Pinch.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
207 This is a GIMP filter called ``Whirl and
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
208 pinch'', but whirl was set to 0. A pinch is ``similar to projecting the image onto an elastic
509
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
209 surface and pressing or pulling on the center of the surface''~\citep{GIMP-manual}.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
210 For a square input image, this is akin to drawing a circle of
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212 that disk (region inside circle) will have its value recalculated by taking
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
213 the value of another ``source'' pixel in the original image. The position of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
214 that source pixel is found on the line that goes through $C$ and $P$, but
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
217 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
218 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
219 around the (non-integer) source position thus found.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
220 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
221
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
222 \vspace*{1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
223
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
224 {\large\bf Injecting Noise}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
225
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
226 \vspace*{1mm}
483
b9cdb464de5f pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 482
diff changeset
227
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
228 {\bf Motion Blur.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
229 This is a ``linear motion blur'' in GIMP
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
230 terminology, with two parameters, $length$ and $angle$. The value of
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
231 a pixel in the final image is approximately the mean value of the $length$ first pixels
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
232 found by moving in the $angle$ direction.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
233 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
234 {\bf Occlusion.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
235 Selects a random rectangle from an {\em occluder} character
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
236 images and places it over the original {\em occluded} character
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
237 image. Pixels are combined by taking the max(occluder,occluded),
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
238 closer to black. The rectangle corners
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
239 are sampled so that larger complexity gives larger rectangles.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
240 The destination position in the occluded image are also sampled
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
241 according to a normal distribution (see more details in~\citet{ift6266-tr-anonymous}).
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
242 This filter has a probability of 60\% of not being applied.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
243 {\bf Pixel Permutation.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
244 This filter permutes neighbouring pixels. It selects first
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
245 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
246 sequentially exchanged with one other pixel in its $V4$ neighbourhood. The number
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
247 of exchanges to the left, right, top, bottom is equal or does not differ
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
248 from more than 1 if the number of selected pixels is not a multiple of 4.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
249 % TODO: The previous sentence is hard to parse
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
250 This filter has a probability of 80\% of not being applied.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
251 {\bf Gaussian Noise.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
252 This filter simply adds, to each pixel of the image independently, a
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
253 noise $\sim Normal(0(\frac{complexity}{10})^2)$.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
254 It has a probability of 70\% of not being applied.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
255 {\bf Background Images.}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
256 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
257 background behind the letter. The background is chosen by first selecting,
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
258 at random, an image from a set of images. Then a 32$\times$32 sub-region
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
259 of that image is chosen as the background image (by sampling position
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
260 uniformly while making sure not to cross image borders).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
261 To combine the original letter image and the background image, contrast
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
262 adjustments are made. We first get the maximal values (i.e. maximal
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
263 intensity) for both the original image and the background image, $maximage$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
264 and $maxbg$. We also have a parameter $contrast \sim U[complexity, 1]$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
265 Each background pixel value is multiplied by $\frac{max(maximage -
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
266 contrast, 0)}{maxbg}$ (higher contrast yield darker
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
267 background). The output image pixels are max(background,original).\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
268 {\bf Salt and Pepper Noise.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
269 This filter adds noise $\sim U[0,1]$ to random subsets of pixels.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
270 The number of selected pixels is $0.2 \times complexity$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
271 This filter has a probability of not being applied at all of 75\%.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
272 {\bf Spatially Gaussian Noise.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
273 Different regions of the image are spatially smoothed.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
274 The image is convolved with a symmetric Gaussian kernel of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
275 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
276 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
277 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
278 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
279 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
280 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
281 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
282 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
283 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
284 image \times mask}{mask+1}$.
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
285 This filter has a probability of not being applied at all of 75\%.\\
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
286 {\bf Scratches.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
287 The scratches module places line-like white patches on the image. The
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
288 lines are heavily transformed images of the digit ``1'' (one), chosen
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
289 at random among five thousands such 1 images. The 1 image is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
290 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
291 complexity)^2$, using bi-cubic interpolation,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
292 Two passes of a grey-scale morphological erosion filter
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
293 are applied, reducing the width of the line
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
294 by an amount controlled by $complexity$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
295 This filter is only applied only 15\% of the time. When it is applied, 50\%
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
296 of the time, only one patch image is generated and applied. In 30\% of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
297 cases, two patches are generated, and otherwise three patches are
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
298 generated. The patch is applied by taking the maximal value on any given
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
299 patch or the original image, for each of the 32x32 pixel locations.\\
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
300 {\bf Grey Level and Contrast Changes.}
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
301 This filter changes the contrast and may invert the image polarity (white
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
302 on black to black on white). The contrast $C$ is defined here as the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
303 difference between the maximum and the minimum pixel value of the image.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
304 Contrast $\sim U[1-0.85 \times complexity,1]$ (so contrast $\geq 0.15$).
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
305 The image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
306 polarity is inverted with $0.5$ probability.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
307
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
308 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
309 \begin{figure}[h]
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
310 \resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}\\
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
311 \caption{Illustration of the pipeline of stochastic
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
312 transformations applied to the image of a lower-case t
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
313 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
314 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
315 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
316 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
317 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
318 \end{figure}
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
319 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
320
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
321
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
322 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
323 \section{Experimental Setup}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
324 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
325
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
326 Whereas much previous work on deep learning algorithms had been performed on
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
327 the MNIST digits classification task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
328 with 60~000 examples, and variants involving 10~000
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
329 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
330 to focus here on the case of much larger training sets, from 10 times to
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
331 to 1000 times larger. The larger datasets are obtained by first sampling from
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
332 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
333 and {\bf OCR data} (scanned machine printed characters). Once a character
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
334 is sampled from one of these sources (chosen randomly), a pipeline of
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
335 the above transformations and/or noise processes is applied to the
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
336 image.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
337
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
338 We compare the best MLP (according to validation set error) that we found against
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
339 the best SDA (again according to validation set error), along with a precise estimate
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
340 of human performance obtained via Amazon's Mechanical Turk (AMT)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
341 service\footnote{http://mturk.com}.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
342 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
343 of money to perform tasks for which human intelligence is required.
509
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
344 Mechanical Turk has been used extensively in natural language
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
345 processing \citep{SnowEtAl2008} and vision
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
346 \citep{SorokinAndForsyth2008,whitehill09}.
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
347 AMT users where presented
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
348 with 10 character images and asked to type 10 corresponding ASCII
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
349 characters. They were forced to make a hard choice among the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
350 62 or 10 character classes (all classes or digits only).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
351 Three users classified each image, allowing
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
352 to estimate inter-human variability.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
353
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
354 \vspace*{-1mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
355 \subsection{Data Sources}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
356 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
357
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
358 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
359 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
360 {\bf NIST.}
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
361 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
362 widely used for training and testing character
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
363 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
364 The dataset is composed with 814255 digits and characters (upper and lower cases), with hand checked classifications,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
365 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
366 corresponding to "0"-"9","A"-"Z" and "a"-"z". The dataset contains 8 series of different complexity.
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
367 The fourth series, $hsf_4$, experimentally recognized to be the most difficult one is recommended
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
368 by NIST as testing set and is used in our work and some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
369 for that purpose. We randomly split the remainder into a training set and a validation set for
480
150203d2b5c3 added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 479
diff changeset
370 model selection. The sizes of these data sets are: 651668 for training, 80000 for validation,
150203d2b5c3 added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 479
diff changeset
371 and 82587 for testing.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
372 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
373 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
374 useful to estimate the effect of a multi-task setting.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
375 Note that the distribution of the classes in the NIST training and test sets differs
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
376 substantially, with relatively many more digits in the test set, and uniform distribution
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
377 of letters in the test set, not in the training set (more like the natural distribution
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
378 of letters in text).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
379
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
380 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
381 {\bf Fonts.}
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
382 In order to have a good variety of sources we downloaded an important number of free fonts from: {\tt http://anonymous.url.net}
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
383 %real adress {\tt http://cg.scs.carleton.ca/~luc/freefonts.html}
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
384 in addition to Windows 7's, this adds up to a total of $9817$ different fonts that we can choose uniformly.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
385 The {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
386 directly as input to our models.
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
387
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
388 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
389 {\bf Captchas.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
390 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
391 generating characters of the same format as the NIST dataset. This software is based on
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
392 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
393 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
394 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
395 depending on the value of the complexity parameter provided by the user of the data source. Two levels of complexity are
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
396 allowed and can be controlled via an easy to use facade class.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
397
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
398 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
399 {\bf OCR data.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
400 A large set (2 million) of scanned, OCRed and manually verified machine-printed
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
401 characters (from various documents and books) where included as an
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
402 additional source. This set is part of a larger corpus being collected by the Image Understanding
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
403 Pattern Recognition Research group lead by Thomas Breuel at University of Kaiserslautern
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
404 ({\tt http://www.iupr.com}), and which will be publicly released.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
405 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
406
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
407 \vspace*{-1mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
408 \subsection{Data Sets}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
409 \vspace*{-1mm}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
410
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
411 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
412 from one of the 62 character classes.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
413 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
414
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
415 %\item
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
416 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
417
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
418 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
419 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
420 and sending them through the above transformation pipeline.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
421 For each new example to generate, a source is selected with probability $10\%$ from the fonts,
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
422 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
423 order given above, and for each of them we sample uniformly a complexity in the range $[0,0.7]$.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
424
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
425 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
426 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same sources proportion)
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
427 except that we only apply
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
428 transformations from slant to pinch. Therefore, the character is
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
429 transformed but no additional noise is added to the image, giving images
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
430 closer to the NIST dataset.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
431 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
432
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
433 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
434 \subsection{Models and their Hyperparameters}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
435 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
436
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
437 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
438 hidden layer and with Stacked Denoising Auto-Encoders (SDA).
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
439 All hyper-parameters are selected based on performance on the NISTP validation set.
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
440
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
441 {\bf Multi-Layer Perceptrons (MLP).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
442 Whereas previous work had compared deep architectures to both shallow MLPs and
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
443 SVMs, we only compared to MLPs here because of the very large datasets used
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
444 (making the use of SVMs computationally inconvenient because of their quadratic
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
445 scaling behavior).
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
446 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
447 exponentials) on the output layer for estimating P(class | image).
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
448 The hyper-parameters are the following: number of hidden units, taken in
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
449 $\{300,500,800,1000,1500\}$. The optimization procedure is as follows. Training
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
450 examples are presented in minibatches of size 20. A constant learning
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
451 rate is chosen in $10^{-3},0.01, 0.025, 0.075, 0.1, 0.5\}$
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
452 through preliminary experiments, and 0.1 was selected.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
453
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
454 {\bf Stacked Denoising Auto-Encoders (SDA).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
455 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
456 can be used to initialize the weights of each layer of a deep MLP (with many hidden
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
457 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
458 enabling better generalization, apparently setting parameters in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
459 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
460 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
461 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
462 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
463 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
464 $P(y|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
465 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
466 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
467 compositions of simpler ones through a deep hierarchy).
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
468 Here we chose to use the Denoising
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
469 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
470 % AJOUTER UNE IMAGE?
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
471 these deep hierarchies of features, as it is very simple to train and
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
472 teach (see tutorial and code there: {\tt http://deeplearning.net/tutorial}),
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
473 provides immediate and efficient inference, and yielded results
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
474 comparable or better than RBMs in series of experiments
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
475 \citep{VincentPLarochelleH2008}. During training of a Denoising
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
476 Auto-Encoder, it is presented with a stochastically corrupted version
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
477 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
478 forcing the hidden units to represent the leading regularities in
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
479 the data. Once it is trained, its hidden units activations can
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
480 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
481 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
482 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
483 the same standard procedure used to train them (see previous section).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
484 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
485 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
486 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
487 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
488 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
489 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
490 of hidden layers but it was fixed to 3 based on previous work with
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
491 stacked denoising auto-encoders on MNIST~\citep{VincentPLarochelleH2008}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
492
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
493 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
494 \section{Experimental Results}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
495
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
496 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
497 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
498 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
499
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
500 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
501 comparing Humans, three MLPs (MLP0, MLP1, MLP2) and three SDAs (SDA0, SDA1,
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
502 SDA2), along with the previous results on the digits NIST special database
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
503 19 test set from the literature respectively based on ARTMAP neural
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
504 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
505 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
506 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
507 (figures and tables, including standard errors on the error rates) can be
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
508 found in the supplementary material. The 3 kinds of model differ in the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
509 training sets used: NIST only (MLP0,SDA0), NISTP (MLP1, SDA1), or P07
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
510 (MLP2, SDA2). The deep learner not only outperformed the shallow ones and
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
511 previously published performance (in a statistically and qualitatively
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
512 significant way) but reaches human performance on both the 62-class task
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
513 and the 10-class (digits) task. In addition, as shown in the left of
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
514 Figure~\ref{fig:fig:improvements-charts}, the relative improvement in error
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
515 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
516 differences with the MLP are statistically and qualitatively
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
517 significant.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
518 The left side of the figure shows the improvement to the clean
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
519 NIST test set error brought by the use of out-of-distribution examples
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
520 (i.e. the perturbed examples examples from NISTP or P07).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
521 Relative change is measured by taking
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
522 (original model's error / perturbed-data model's error - 1).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
523 The right side of
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
524 Figure~\ref{fig:fig:improvements-charts} shows the relative improvement
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
525 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
526 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
527 with all 62 classes when the target classes are respectively the digits,
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
528 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
529 multi-task setting is marginal or negative for the MLP, it is substantial
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
530 for the SDA. Note that for these multi-task experiment, only the original
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
531 NIST dataset is used. For example, the MLP-digits bar shows the relative
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
532 improvement in MLP error rate on the NIST digits test set (1 - single-task
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
533 model's error / multi-task model's error). The single-task model is
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
534 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
535 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
536 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
537 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
538 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
539 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
540 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
541 and upper case characters).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
542
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
543 \begin{figure}[h]
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
544 \resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}\\
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
545 \caption{Error bars indicate a 95\% confidence interval. 0 indicates training
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
546 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
547 of all models, on 3 different test sets corresponding to the three
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
548 datasets.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
549 Right: error rates on NIST test digits only, along with the previous results from
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
550 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
551 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
552
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
553 \label{fig:error-rates-charts}
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
554 \end{figure}
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
555
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
556 %\vspace*{-1mm}
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
557 %\subsection{Perturbed Training Data More Helpful for SDA}
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
558 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
559
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
560 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
561 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
562 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
563
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
564 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
565 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
566 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
567 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
568 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
569 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
570 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
571 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
572 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
573 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
574 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
575 shallow learner.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
576
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
577 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
578 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
579 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
580
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
581 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
582 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
583 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
584 On the other hand the SDA benefited from the multi-task setting, with relative
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
585 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
586 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
587 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
588
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
589
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
590 \begin{figure}[h]
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
591 \resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}\\
509
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
592 \caption{Relative improvement in error rate due to self-taught learning.
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
593 Left: Improvement (or loss, when negative)
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
594 induced by out-of-distribution examples (perturbed data).
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
595 Right: Improvement (or loss, when negative) induced by multi-task
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
596 learning (training on all classes and testing only on either digits,
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
597 upper case, or lower-case). The deep learner (SDA) benefits more from
860c755ddcff argh, sorry about that
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 505
diff changeset
598 both self-taught learning scenarios, compared to the shallow MLP.}
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
599 \label{fig:improvements-charts}
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
600 \end{figure}
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
601
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
602 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
603 \section{Conclusions}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
604 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
605
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
606 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
607 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
608 supervised learner. More precisely,
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
609 the conclusions are positive for all the questions asked in the introduction.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
610 %\begin{itemize}
487
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 486
diff changeset
611
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
612 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
613 Do the good results previously obtained with deep architectures on the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
614 MNIST digits generalize to the setting of a much larger and richer (but similar)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
615 dataset, the NIST special database 19, with 62 classes and around 800k examples?
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
616 Yes, the SDA {\bf systematically outperformed the MLP and all the previously
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
617 published results on this dataset (as far as we know), in fact reaching human-level
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
618 performance} at round 17\% error on the 62-class task and 1.4\% on the digits.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
619
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
620 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
621 To what extent does the perturbation of input images (e.g. adding
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
622 noise, affine transformations, background images) make the resulting
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
623 classifier better not only on similarly perturbed images but also on
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
624 the {\em original clean examples}? Do deep architectures benefit more from such {\em out-of-distribution}
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
625 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
626 MLPs were helped by perturbed training examples when tested on perturbed input
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
627 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
628 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
629 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
630 with respect to clean examples . On the other hand, the deep SDAs
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
631 were very significantly boosted by these out-of-distribution examples.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
632
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
633 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
634 Similarly, does the feature learning step in deep learning algorithms benefit more
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
635 training with similar but different classes (i.e. a multi-task learning scenario) than
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
636 a corresponding shallow and purely supervised architecture?
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
637 Whereas the improvement due to the multi-task setting was marginal or
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
638 negative for the MLP (from +5.6\% to -3.6\% relative change),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
639 it was very significant for the SDA (from +13\% to +27\% relative change).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
640 %\end{itemize}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
641
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
642 Why would deep learners benefit more from the self-taught learning framework?
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
643 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
644 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
645 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
646 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
647 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
648 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
649 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
650 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
651 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
652 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
653 experiments could attempt at teasing apart these factors.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
654
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
655 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
656 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
657
498
7ff00c27c976 add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents: 496
diff changeset
658 \newpage
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
659 {
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
660 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
661 %\bibliographystyle{plainnat}
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
662 \bibliographystyle{unsrtnat}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
663 %\bibliographystyle{apalike}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
664 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
665
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
666
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
667 \end{document}