Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 626:75dbbe409578

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
author fsavard
date Wed, 16 Mar 2011 13:43:32 -0400
parents 9ebb335ca904
children
rev   line source
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
3 \usepackage{wrapfig}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
5 \usepackage[psamsfonts]{amssymb}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{graphicx,subfigure}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
9 \usepackage[numbers]{natbib}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
12
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
581
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
14 \author{
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
15 Frédéric Bastien ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
16 Yoshua Bengio ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
17 Arnaud Bergeron ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
18 Nicolas Boulanger-Lewandowski ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
19 Thomas Breuel ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
20 Youssouf Chherawala ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
21 Moustapha Cisse ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
22 Myriam Côté ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
23 Dumitru Erhan ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
24 Jeremy Eustache ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
25 Xavier Glorot ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
26 Xavier Muller ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
27 Sylvain Pannetier Lebeuf ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
28 Razvan Pascanu ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
29 Salah Rifai ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
30 Francois Savard ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
31 Guillaume Sicard
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
32 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 \begin{document}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 \maketitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
39 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 \begin{abstract}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
41 Recent theoretical and empirical work in statistical machine learning has
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
42 demonstrated the importance of learning algorithms for deep
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
43 architectures, i.e., function classes obtained by composing multiple
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
44 non-linear transformations. Self-taught learning (exploiting unlabeled
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
45 examples or examples from other distributions) has already been applied
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
46 to deep learners, but mostly to show the advantage of unlabeled
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
47 examples. Here we explore the advantage brought by {\em out-of-distribution examples}.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
48 For this purpose we
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
49 developed a powerful generator of stochastic variations and noise
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
50 processes for character images, including not only affine transformations
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
51 but also slant, local elastic deformations, changes in thickness,
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
52 background images, grey level changes, contrast, occlusion, and various
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
53 types of noise. The out-of-distribution examples are obtained from these
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
54 highly distorted images or by including examples of object classes
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
55 different from those in the target test set.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
56 We show that {\em deep learners benefit
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
57 more from them than a corresponding shallow learner}, at least in the area of
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
58 handwritten character recognition. In fact, we show that they reach
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
59 human-level performance on both handwritten digit classification and
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
60 62-class handwritten character recognition.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 \end{abstract}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
62 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 \section{Introduction}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
65 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
67 {\bf Deep Learning} has emerged as a promising new area of research in
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
68 statistical machine learning (see~\citet{Bengio-2009} for a review).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 of useful representations of data, which are better suited to the task at hand.
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
71 This is in part inspired by observations of the mammalian visual cortex,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 which consists of a chain of processing elements, each of which is associated with a
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
73 different representation of the raw visual input. In fact,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 those observed in the first two of these stages (in areas V1 and V2
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
76 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
77 more invariant to factors of variation (such as camera movement) in
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
78 higher layers~\citep{Goodfellow2009}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
79 Learning a hierarchy of features increases the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
87 {\bf Self-taught learning}~\citep{RainaR2007} is a paradigm that combines principles
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
88 of semi-supervised and multi-task learning: the learner can exploit examples
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
89 that are unlabeled and possibly come from a distribution different from the target
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
90 distribution, e.g., from other classes than those of interest.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
91 It has already been shown that deep learners can clearly take advantage of
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
92 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
93 but more needs to be done to explore the impact
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
94 of {\em out-of-distribution} examples and of the multi-task setting
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
95 (one exception is~\citep{CollobertR2008}, which uses a different kind
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
96 of learning algorithm). In particular the {\em relative
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
97 advantage} of deep learning for these settings has not been evaluated.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
98 The hypothesis discussed in the conclusion is that a deep hierarchy of features
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
99 may be better able to provide sharing of statistical strength
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
100 between different regions in input space or different tasks.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
101
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
102 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108 learning, often in an greedy layer-wise ``unsupervised pre-training''
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
109 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110 applied here, is the Denoising
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
111 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see Figure~\ref{fig:da}),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
112 which
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 Machines in terms of unsupervised extraction of a hierarchy of features
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
115 useful for classification. Each layer is trained to denoise its
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
116 input, creating a layer of features that can be used as input for the next layer.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
117 \fi
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
118 %The principle is that each layer starting from
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
119 %the bottom is trained to encode its input (the output of the previous
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
120 %layer) and to reconstruct it from a corrupted version. After this
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
121 %unsupervised initialization, the stack of DAs can be
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
122 %converted into a deep supervised feedforward neural network and fine-tuned by
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
123 %stochastic gradient descent.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
125 %
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
126 In this paper we ask the following questions:
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
127
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
128 %\begin{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
129 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
130 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
131 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
132 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
133
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
134 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
135 To what extent does the perturbation of input images (e.g. adding
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
136 noise, affine transformations, background images) make the resulting
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
137 classifiers better not only on similarly perturbed images but also on
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
138 the {\em original clean examples}? We study this question in the
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
139 context of the 62-class and 10-class tasks of the NIST special database 19.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
140
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
141 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
142 Do deep architectures {\em benefit more from such out-of-distribution}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
143 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
144 We use highly perturbed examples to generate out-of-distribution examples.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
145
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
146 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
147 Similarly, does the feature learning step in deep learning algorithms benefit more
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
148 from training with moderately different classes (i.e. a multi-task learning scenario) than
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
149 a corresponding shallow and purely supervised architecture?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
150 We train on 62 classes and test on 10 (digits) or 26 (upper case or lower case)
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
151 to answer this question.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
152 %\end{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
153
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
154 Our experimental results provide positive evidence towards all of these questions.
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
155 To achieve these results, we introduce in the next section a sophisticated system
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
156 for stochastically transforming character images and then explain the methodology,
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
157 which is based on training with or without these transformed images and testing on
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
158 clean ones. We measure the relative advantage of out-of-distribution examples
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
159 for a deep learner vs a supervised shallow one.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
160 Code for generating these transformations as well as for the deep learning
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
161 algorithms are made available.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
162 We also estimate the relative advantage for deep learners of training with
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
163 other classes than those of interest, by comparing learners trained with
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
164 62 classes with learners trained with only a subset (on which they
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
165 are then tested).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
166 The conclusion discusses
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
167 the more general question of why deep learners may benefit so much from
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
168 the self-taught learning framework.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
170 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 \section{Perturbation and Transformation of Character Images}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
172 \label{s:perturbations}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
173 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
175 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
176 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
177 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
178 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
179 \includegraphics[scale=.4]{images/Original.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
180 {\bf Original}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
181 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
182 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
183 %\vspace{0.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
184 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
185 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
186 This section describes the different transformations we used to stochastically
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
187 transform $32 \times 32$ source images (such as the one on the left)
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
188 in order to obtain data from a larger distribution which
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
189 covers a domain substantially larger than the clean characters distribution from
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
190 which we start.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
191 Although character transformations have been used before to
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
192 improve character recognizers, this effort is on a large scale both
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
193 in number of classes and in the complexity of the transformations, hence
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
194 in the complexity of the learning task.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
195 More details can
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
196 be found in this technical report~\citep{ift6266-tr-anonymous}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
200 amount of deformation or noise introduced.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
201 There are two main parts in the pipeline. The first one,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
202 from slant to pinch below, performs transformations. The second
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
203 part, from blur to contrast, adds different kinds of noise.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
204 %\end{minipage}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
206 \vspace*{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
207 %\subsection{Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
208 {\large\bf 2.1 Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
209 \vspace*{1mm}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
210
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
211
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
212 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
213 %\begin{minipage}[b]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
214 %\centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
215 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
216 \vspace*{-5mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
217 \includegraphics[scale=.4]{images/Thick_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
218 {\bf Thickness}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
219 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
220 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
221 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
222 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
223 \end{wrapfigure}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
224 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
225 are applied. The neighborhood of each pixel is multiplied
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
226 element-wise with a {\em structuring element} matrix.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
227 The pixel value is replaced by the maximum or the minimum of the resulting
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
228 matrix, respectively for dilation or erosion. Ten different structural elements with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
229 increasing dimensions (largest is $5\times5$) were used. For each image,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
230 randomly sample the operator type (dilation or erosion) with equal probability and one structural
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
231 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
232 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
233 A neutral element (no transformation)
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
234 is always present in the set.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
235 %\vspace{.4cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
236 %\end{minipage}
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
237 \vspace*{3mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
238
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
239 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
240 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
241 \includegraphics[scale=.4]{images/Slant_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
242 {\bf Slant}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
243 \end{minipage}%
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
244 \hspace{0.3cm}
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
245 \begin{minipage}[b]{0.83\linewidth}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
246 %\centering
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
247 To produce {\bf slant}, each row of the image is shifted
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
248 proportionally to its height: $shift = round(slant \times height)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
249 $slant \sim U[-complexity,complexity]$.
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
250 The shift is randomly chosen to be either to the left or to the right.
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
251 \vspace{8mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
252 \end{minipage}
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
253 \vspace*{3mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
254
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
255 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
256 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
257 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
258 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
259 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
260 \includegraphics[scale=.4]{images/Affine_only.png}\\
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
261 {\small {\bf Affine \mbox{Transformation}}}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
262 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
263 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
264 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
265 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
266 A $2 \times 3$ {\bf affine transform} matrix (with
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
267 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
268 Output pixel $(x,y)$ takes the value of input pixel
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
269 nearest to $(ax+by+c,dx+ey+f)$,
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
270 producing scaling, translation, rotation and shearing.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
271 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
272 forbid large rotations (to avoid confusing classes) but to give good
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
273 variability of the transformation: $a$ and $d$ $\sim U[1-3
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
274 complexity,1+3\,complexity]$, $b$ and $e$ $\sim U[-3 \,complexity,3\,
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
275 complexity]$, and $c$ and $f \sim U[-4 \,complexity, 4 \,
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
276 complexity]$.\\
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
277 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
278
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
279 \vspace*{-4.5mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
280
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
281 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
282 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
283 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
284 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
285 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
286 \vspace*{-4mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
287 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.png}\\
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
288 {\bf Local Elastic Deformation}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
289 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
290 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
291 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
292 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
293 %\vspace*{-20mm}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
294 The {\bf local elastic deformation}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
295 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
296 which provides more details.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
297 The intensity of the displacement fields is given by
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
298 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
299 convolved with a Gaussian 2D kernel (resulting in a blur) of
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
300 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
301 %\vspace{.9cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
302 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
303
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
304 \vspace*{7mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
305
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
306 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
307 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
308 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
309 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
310 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
311 \includegraphics[scale=.4]{images/Pinch_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
312 {\bf Pinch}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
313 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
314 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
315 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
316 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
317 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
318 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl set to 0.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
319 A pinch is ``similar to projecting the image onto an elastic
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
320 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
321 For a square input image, draw a radius-$r$ disk
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
322 around its center $C$. Any pixel $P$ belonging to
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
323 that disk has its value replaced by
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
324 the value of a ``source'' pixel in the original image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
325 on the line that goes through $C$ and $P$, but
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
326 at some other distance $d_2$. Define $d_1=distance(P,C)$
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
327 and $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
328 d_1$, where $pinch$ is a parameter of the filter.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
329 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
330 around the (non-integer) source position thus found.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
331 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
332 %\vspace{1.5cm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
333 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
334
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
335 \vspace{1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
336
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
337 {\large\bf 2.2 Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
338 %\subsection{Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
339 \vspace{2mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
340
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
341 %\vspace*{-.2cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
342 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
343 \centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
344 \vspace*{-2mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
345 \includegraphics[scale=.4]{images/Motionblur_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
346 {\bf Motion Blur}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
347 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
348 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
349 %\vspace*{.5mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
350 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
351 has parameters $length$ and $angle$. The value of
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
352 a pixel in the final image is approximately the mean of the first $length$ pixels
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
353 found by moving in the $angle$ direction,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
354 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
355 \vspace{5mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
356 \end{minipage}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
357
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
358 \vspace*{1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
359
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
360 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
361 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
362 \includegraphics[scale=.4]{images/occlusion_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
363 {\bf Occlusion}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
364 %\vspace{.5cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
365 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
366 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
367 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
368 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
369 image and places it over the original {\em occluded}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
370 image. Pixels are combined by taking the max(occluder, occluded),
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
371 i.e. keeping the lighter ones.
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
372 The rectangle corners
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
373 are sampled so that larger complexity gives larger rectangles.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
374 The destination position in the occluded image are also sampled
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
375 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
376 This module is skipped with probability 60\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
377 %\vspace{7mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
378 \end{minipage}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
379
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
380 \vspace*{1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
381
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
382 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
383 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
384 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
385 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
386 %\centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
387 \includegraphics[scale=.4]{images/Bruitgauss_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
388 {\bf Gaussian Smoothing}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
389 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
390 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
391 %\vspace{.5cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
392 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
393 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
394 With the {\bf Gaussian smoothing} module,
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
395 different regions of the image are spatially smoothed.
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
396 This is achieved by first convolving
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
397 the image with an isotropic Gaussian kernel of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
398 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
399 complexity]$ and $[2,2 + 6 \times complexity]$. This filtered image is normalized
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
400 between $0$ and $1$. We also create an isotropic weighted averaging window, of the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
402 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 initialize to zero a mask matrix of the image size. For each selected pixel
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
405 we add to the mask the averaging window centered on it. The final image is
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
406 computed from the following element-wise operation: $\frac{image + filtered\_image
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
407 \times mask}{mask+1}$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
408 This module is skipped with probability 75\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
409 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
410
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
411 \newpage
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
412
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
413 \vspace*{-9mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
414
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
415 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
416 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
417 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
418 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
419 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
420 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
421 \includegraphics[scale=.4]{images/Permutpixel_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
422 {\small\bf Permute Pixels}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
423 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
424 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
425 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
426 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
427 %\vspace*{-20mm}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
428 This module {\bf permutes neighbouring pixels}. It first selects a
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
429 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
430 of these pixels is then sequentially exchanged with a random pixel
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
431 among its four nearest neighbors (on its left, right, top or bottom).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
432 This module is skipped with probability 80\%.\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
433 \vspace*{1mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
434 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
435
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
436 \vspace{-3mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
437
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
438 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
439 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
440 %\vspace*{-3mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
441 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
442 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
443 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
444 \vspace*{-5mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
445 \includegraphics[scale=.4]{images/Distorsiongauss_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
446 {\small \bf Gauss. Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
447 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
448 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
449 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
450 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
451 \vspace*{12mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
452 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
453 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
454 This module is skipped with probability 70\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
455 %\vspace{1.1cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
456 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
457
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
458 \vspace*{1.2cm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
459
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
460 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
461 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
462 \centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
463 \includegraphics[scale=.4]{images/background_other_only.png}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
464 {\small \bf Bg Image}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
465 \end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
466 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
467 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
468 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
469 background image behind the letter, from a randomly chosen natural image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
470 with contrast adjustments depending on $complexity$, to preserve
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
471 more or less of the original character image.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
472 %\vspace{.8cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
473 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
474 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
475 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
476
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
477 \begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
478 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
479 \includegraphics[scale=.4]{images/Poivresel_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
480 {\small \bf Salt \& Pepper}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
481 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
482 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
483 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
484 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
485 The number of selected pixels is $0.2 \times complexity$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
486 This module is skipped with probability 75\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
487 %\vspace{.9cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
488 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
489 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
490
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
491 \vspace{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
492
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
493 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
494 \begin{wrapfigure}[7]{l}{0.14\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
495 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
496 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
497 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
498 \vspace*{-4mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
499 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
500 {\bf Scratches}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
501 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
502 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
503 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
504 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
505 %\vspace{.4cm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
506 The {\bf scratches} module places line-like white patches on the image. The
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
507 lines are heavily transformed images of the digit ``1'' (one), chosen
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
508 at random among 500 such 1 images,
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
509 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
554
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 553
diff changeset
510 complexity)^2$ (in degrees), using bi-cubic interpolation.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
511 Two passes of a grey-scale morphological erosion filter
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
512 are applied, reducing the width of the line
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
513 by an amount controlled by $complexity$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
514 This module is skipped with probability 85\%. The probabilities
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
515 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
516 \end{minipage}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
517
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
518 \vspace*{1mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
519
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
520 \begin{minipage}[t]{0.25\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
521 \centering
559
Olivier Delalleau <delallea@iro>
parents: 558 557
diff changeset
522 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.png}\\
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
523 {\bf Grey Level \& Contrast}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
524 \end{minipage}%
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
525 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
526 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
527 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
528 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
529 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
530 polarity is inverted with probability 50\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
531 %\vspace{.7cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
532 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
533 \vspace{2mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
534
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
535
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
536 \iffalse
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
537 \begin{figure}[ht]
538
f0ee2212ea7c typos and stuff
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 537 534
diff changeset
538 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
539 \caption{Illustration of the pipeline of stochastic
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
540 transformations applied to the image of a lower-case \emph{t}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
541 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
542 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
543 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
544 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
545 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
546 \end{figure}
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
547 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
548
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
549 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
550 \section{Experimental Setup}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
551 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
552
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
553 Much previous work on deep learning had been performed on
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
554 the MNIST digits task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
555 with 60~000 examples, and variants involving 10~000
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
556 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}.
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
557 The focus here is on much larger training sets, from 10 times to
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
558 to 1000 times larger, and 62 classes.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
559
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
560 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
561 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
562 and {\bf OCR data} (scanned machine printed characters). Once a character
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
563 is sampled from one of these sources (chosen randomly), the second step is to
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
564 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
565
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
566 To provide a baseline of error rate comparison we also estimate human performance
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
567 on both the 62-class task and the 10-class digits task.
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
568 We compare the best Multi-Layer Perceptrons (MLP) against
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
569 the best Stacked Denoising Auto-encoders (SDA), when
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
570 both models' hyper-parameters are selected to minimize the validation set error.
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
571 We also provide a comparison against a precise estimate
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
572 of human performance obtained via Amazon's Mechanical Turk (AMT)
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
573 service (http://mturk.com).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
574 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
575 of money to perform tasks for which human intelligence is required.
522
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
576 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
577 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
578 %\citep{SorokinAndForsyth2008,whitehill09}.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
579 AMT users were presented
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
580 with 10 character images (from a test set) and asked to choose 10 corresponding ASCII
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
581 characters. They were forced to choose a single character class (either among the
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
582 62 or 10 character classes) for each image.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
583 80 subjects classified 2500 images per (dataset,task) pair,
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
584 with the guarantee that 3 different subjects classified each image, allowing
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
585 us to estimate inter-human variability (e.g a standard error of 0.1\%
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
586 on the average 18.2\% error done by humans on the 62-class task NIST test set).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
587
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
588 \vspace*{-3mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
589 \subsection{Data Sources}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
590 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
591
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
592 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
593 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
594 {\bf NIST.}
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
595 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
596 widely used for training and testing character
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
597 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
598 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
599 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
600 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
601 The fourth partition (called $hsf_4$, 82587 examples),
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
602 experimentally recognized to be the most difficult one, is the one recommended
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
603 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
604 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
605 model selection.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
606 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
607 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
608 useful to estimate the effect of a multi-task setting.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
609 The distribution of the classes in the NIST training and test sets differs
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
610 substantially, with relatively many more digits in the test set, and a more uniform distribution
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
611 of letters in the test set (whereas in the training set they are distributed
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
612 more like in natural text).
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
613 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
614
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
615 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
616 {\bf Fonts.}
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
617 In order to have a good variety of sources we downloaded an important number of free fonts from:
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
618 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
619 % TODO: pointless to anonymize, it's not pointing to our work
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
620 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
621 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
622 directly as input to our models.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
623 \vspace*{-1mm}
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
624
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
625 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
626 {\bf Captchas.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
627 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
628 generating characters of the same format as the NIST dataset. This software is based on
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
629 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
630 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
631 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
632 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
633 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
634 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
635
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
636 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
637 {\bf OCR data.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
638 A large set (2 million) of scanned, OCRed and manually verified machine-printed
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
639 characters where included as an
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
640 additional source. This set is part of a larger corpus being collected by the Image Understanding
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
641 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
642 ({\tt http://www.iupr.com}), and which will be publicly released.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
643 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
644 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
645
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
646 \vspace*{-3mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
647 \subsection{Data Sets}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
648 \vspace*{-2mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
649
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
650 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
651 from one of the 62 character classes.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
652 %\begin{itemize}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
653 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
654
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
655 %\item
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
656 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
657 \{651668 / 80000 / 82587\} \{training / validation / test\} examples.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
658 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
659
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
660 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
661 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
662 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
663 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
664 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
665 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
666 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
667 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
668
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
669 %\item
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
670 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
671 except that we only apply
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
672 transformations from slant to pinch. Therefore, the character is
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
673 transformed but no additional noise is added to the image, giving images
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
674 closer to the NIST dataset.
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
675 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
676 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
677
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
678 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
679 \subsection{Models and their Hyperparameters}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
680 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
681
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
682 The experiments are performed using MLPs (with a single
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
683 hidden layer) and SDAs.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
684 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
685
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
686 {\bf Multi-Layer Perceptrons (MLP).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
687 Whereas previous work had compared deep architectures to both shallow MLPs and
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
688 SVMs, we only compared to MLPs here because of the very large datasets used
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
689 (making the use of SVMs computationally challenging because of their quadratic
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
690 scaling behavior).
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
691 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
692 exponentials) on the output layer for estimating $P(class | image)$.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
693 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
694 Training examples are presented in minibatches of size 20. A constant learning
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
695 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
696 %through preliminary experiments (measuring performance on a validation set),
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
697 %and $0.1$ (which was found to work best) was then selected for optimizing on
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
698 %the whole training sets.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
699 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
700
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
701
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
702 {\bf Stacked Denoising Auto-Encoders (SDA).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
703 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
704 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
705 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
706 apparently setting parameters in the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
707 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
708 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
709 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
710 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
711 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
712 $P(y|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
713 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
714 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
715 compositions of simpler ones through a deep hierarchy).
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
716
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
717 \begin{figure}[ht]
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
718 \vspace*{-2mm}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
719 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
720 \vspace*{-2mm}
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
721 \caption{Illustration of the computations and training criterion for the denoising
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
722 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
723 the layer (i.e. raw input or output of previous layer)
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
724 s corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
725 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
726 is compared to the uncorrupted input $x$ through the loss function
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
727 $L_H(x,z)$, whose expected value is approximately minimized during training
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
728 by tuning $\theta$ and $\theta'$.}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
729 \label{fig:da}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
730 \vspace*{-2mm}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
731 \end{figure}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
732
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
733 Here we chose to use the Denoising
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
734 Auto-encoder~\citep{VincentPLarochelleH2008} as the building block for
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
735 these deep hierarchies of features, as it is simple to train and
532
2e33885730cf changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 529
diff changeset
736 explain (see Figure~\ref{fig:da}, as well as
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
737 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
738 provides efficient inference, and yielded results
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
739 comparable or better than RBMs in series of experiments
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
740 \citep{VincentPLarochelleH2008}. During training, a Denoising
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
741 Auto-encoder is presented with a stochastically corrupted version
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
742 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
743 forcing the hidden units to represent the leading regularities in
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
744 the data. Here we use the random binary masking corruption
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
745 (which sets to 0 a random subset of the inputs).
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
746 Once it is trained, in a purely unsupervised way,
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
747 its hidden units' activations can
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
748 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
749 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
750 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
751 the same standard procedure used to train them (see previous section).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
752 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
753 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
754 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
755 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
756 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
757 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
758 of hidden layers but it was fixed to 3 based on previous work with
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
759 SDAs on MNIST~\citep{VincentPLarochelleH2008}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
760
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
761 \vspace*{-1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
762
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
763 \begin{figure}[ht]
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
764 \vspace*{-2mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
765 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
766 \vspace*{-3mm}
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
767 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
768 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
548
34cb28249de0 suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 547
diff changeset
769 of all models, on NIST and NISTP test sets.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
770 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
771 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
772 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
773 \label{fig:error-rates-charts}
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
774 \vspace*{-2mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
775 \end{figure}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
776
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
777
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
778 \begin{figure}[ht]
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
779 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
780 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
781 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
782 \caption{Relative improvement in error rate due to self-taught learning.
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
783 Left: Improvement (or loss, when negative)
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
784 induced by out-of-distribution examples (perturbed data).
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
785 Right: Improvement (or loss, when negative) induced by multi-task
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
786 learning (training on all classes and testing only on either digits,
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
787 upper case, or lower-case). The deep learner (SDA) benefits more from
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
788 both self-taught learning scenarios, compared to the shallow MLP.}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
789 \label{fig:improvements-charts}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
790 \vspace*{-2mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
791 \end{figure}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
792
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
793 \section{Experimental Results}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
794 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
795
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
796 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
797 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
798 %\vspace*{-1mm}
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
799 The models are either trained on NIST (MLP0 and SDA0),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
800 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
801 on either NIST, NISTP or P07, either on the 62-class task
568
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
802 or on the 10-digits task. Training (including about half
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
803 for unsupervised pre-training, for DAs) on the larger
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
804 datasets takes around one day on a GPU-285.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
805 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
806 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
807 SDA2), along with the previous results on the digits NIST special database
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
808 19 test set from the literature, respectively based on ARTMAP neural
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
809 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
810 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
811 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
812 (figures and tables, including standard errors on the error rates) can be
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
813 found in Appendix I of the supplementary material.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
814 The deep learner not only outperformed the shallow ones and
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
815 previously published performance (in a statistically and qualitatively
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
816 significant way) but when trained with perturbed data
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
817 reaches human performance on both the 62-class task
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
818 and the 10-class (digits) task.
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
819 17\% error (SDA1) or 18\% error (humans) may seem large but a large
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
820 majority of the errors from humans and from SDA1 are from out-of-context
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
821 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
822 ``c'' and a ``C'' are often indistinguishible).
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
823
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
824 In addition, as shown in the left of
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
825 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
826 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
827 differences with the MLP are statistically and qualitatively
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
828 significant.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
829 The left side of the figure shows the improvement to the clean
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
830 NIST test set error brought by the use of out-of-distribution examples
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
831 (i.e. the perturbed examples examples from NISTP or P07).
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
832 Relative percent change is measured by taking
548
34cb28249de0 suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 547
diff changeset
833 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
834 The right side of
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
835 Figure~\ref{fig:improvements-charts} shows the relative improvement
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
836 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
837 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
838 with all 62 classes when the target classes are respectively the digits,
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
839 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
840 multi-task setting is marginal or negative for the MLP, it is substantial
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
841 for the SDA. Note that to simplify these multi-task experiments, only the original
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
842 NIST dataset is used. For example, the MLP-digits bar shows the relative
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
843 percent improvement in MLP error rate on the NIST digits test set
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
844 is $100\% \times$ (single-task
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
845 model's error / multi-task model's error - 1). The single-task model is
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
846 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
847 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
848 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
849 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
850 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
851 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
852 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
853 and upper case characters).
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
854 %\vspace*{-1mm}
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
855 %\subsection{Perturbed Training Data More Helpful for SDA}
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
856 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
857
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
858 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
859 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
860 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
861
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
862 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
863 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
864 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
865 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
866 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
867 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
868 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
869 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
870 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
871 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
872 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
873 shallow learner.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
874
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
875 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
876 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
877 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
878
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
879 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
880 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
881 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
882 On the other hand the SDA benefited from the multi-task setting, with relative
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
883 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
884 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
885 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
886
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
887
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
888 \vspace*{-2mm}
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
889 \section{Conclusions and Discussion}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
890 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
891
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
892 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
893 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
894 supervised learner. More precisely,
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
895 the answers are positive for all the questions asked in the introduction.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
896 %\begin{itemize}
487
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 486
diff changeset
897
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
898 $\bullet$ %\item
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
899 {\bf Do the good results previously obtained with deep architectures on the
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
900 MNIST digits generalize to a much larger and richer (but similar)
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
901 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
902 Yes, the SDA {\em systematically outperformed the MLP and all the previously
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
903 published results on this dataset} (the ones that we are aware of), {\em in fact reaching human-level
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
904 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
905
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
906 $\bullet$ %\item
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
907 {\bf To what extent do self-taught learning scenarios help deep learners,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
908 and do they help them more than shallow supervised ones}?
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
909 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
910 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
911 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
912 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
913 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
914 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
915 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
916 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
917 with respect to clean examples . On the other hand, the deep SDAs
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
918 were significantly boosted by these out-of-distribution examples.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
919 Similarly, whereas the improvement due to the multi-task setting was marginal or
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
920 negative for the MLP (from +5.6\% to -3.6\% relative change),
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
921 it was quite significant for the SDA (from +13\% to +27\% relative change),
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
922 which may be explained by the arguments below.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
923 %\end{itemize}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
924
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
925 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
926 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
927 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
928 scenario. However, many of the results by \citet{RainaR2007} (who used a
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
929 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
930 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
931 We note instead that, for deep
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
932 architectures, our experiments show that such a positive effect is accomplished
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
933 even in a scenario with a \emph{large number of labeled examples},
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
934 i.e., here, the relative gain of self-taught learning is probably preserved
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
935 in the asymptotic regime.
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
936
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
937 {\bf Why would deep learners benefit more from the self-taught learning framework}?
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
938 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
939 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
940 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
941 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
942 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
943 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
944 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
945 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
946 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
947 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
948 experiments could attempt at teasing apart these factors.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
949 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
950 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
951 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
952 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
953 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
954 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
955 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
956 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
957 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
958 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
959 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
960 (with or without self-taught settings), and more labeled examples
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
961 does not allow the model to go from the poorer basins of attraction discovered
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
962 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
963 with deep learning and self-taught learning.
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
964
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
965 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
966 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
967
498
7ff00c27c976 add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents: 496
diff changeset
968 \newpage
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
969 {
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
970 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
971 %\bibliographystyle{plainnat}
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
972 \bibliographystyle{unsrtnat}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
973 %\bibliographystyle{apalike}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
974 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
975
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
976
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
977 \end{document}