Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 612:21d53fd07f6e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
reviews AISTATS
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Mon, 20 Dec 2010 11:54:35 -0500
parents 9ebb335ca904
children
rev   line source
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
3 \usepackage{wrapfig}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
5 \usepackage[psamsfonts]{amssymb}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{graphicx,subfigure}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
9 \usepackage[numbers]{natbib}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
12
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
581
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
14 \author{
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
15 Frédéric Bastien ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
16 Yoshua Bengio ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
17 Arnaud Bergeron ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
18 Nicolas Boulanger-Lewandowski ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
19 Thomas Breuel ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
20 Youssouf Chherawala ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
21 Moustapha Cisse ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
22 Myriam Côté ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
23 Dumitru Erhan ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
24 Jeremy Eustache ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
25 Xavier Glorot ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
26 Xavier Muller ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
27 Sylvain Pannetier Lebeuf ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
28 Razvan Pascanu ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
29 Salah Rifai ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
30 Francois Savard ,
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
31 Guillaume Sicard
df749e70f637 ajoute les auteurs
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 569
diff changeset
32 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 \begin{document}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 \maketitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
39 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 \begin{abstract}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
41 Recent theoretical and empirical work in statistical machine learning has
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
42 demonstrated the importance of learning algorithms for deep
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
43 architectures, i.e., function classes obtained by composing multiple
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
44 non-linear transformations. Self-taught learning (exploiting unlabeled
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
45 examples or examples from other distributions) has already been applied
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
46 to deep learners, but mostly to show the advantage of unlabeled
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
47 examples. Here we explore the advantage brought by {\em out-of-distribution examples}.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
48 For this purpose we
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
49 developed a powerful generator of stochastic variations and noise
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
50 processes for character images, including not only affine transformations
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
51 but also slant, local elastic deformations, changes in thickness,
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
52 background images, grey level changes, contrast, occlusion, and various
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
53 types of noise. The out-of-distribution examples are obtained from these
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
54 highly distorted images or by including examples of object classes
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
55 different from those in the target test set.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
56 We show that {\em deep learners benefit
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
57 more from them than a corresponding shallow learner}, at least in the area of
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
58 handwritten character recognition. In fact, we show that they reach
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
59 human-level performance on both handwritten digit classification and
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
60 62-class handwritten character recognition.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 \end{abstract}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
62 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 \section{Introduction}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
65 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
67 {\bf Deep Learning} has emerged as a promising new area of research in
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
68 statistical machine learning (see~\citet{Bengio-2009} for a review).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 of useful representations of data, which are better suited to the task at hand.
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
71 This is in part inspired by observations of the mammalian visual cortex,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 which consists of a chain of processing elements, each of which is associated with a
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
73 different representation of the raw visual input. In fact,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 those observed in the first two of these stages (in areas V1 and V2
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
76 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
77 more invariant to factors of variation (such as camera movement) in
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
78 higher layers~\citep{Goodfellow2009}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
79 Learning a hierarchy of features increases the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
87 {\bf Self-taught learning}~\citep{RainaR2007} is a paradigm that combines principles
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
88 of semi-supervised and multi-task learning: the learner can exploit examples
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
89 that are unlabeled and possibly come from a distribution different from the target
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
90 distribution, e.g., from other classes than those of interest.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
91 It has already been shown that deep learners can clearly take advantage of
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
92 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
93 but more needs to be done to explore the impact
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
94 of {\em out-of-distribution} examples and of the multi-task setting
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
95 (one exception is~\citep{CollobertR2008}, which uses a different kind
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
96 of learning algorithm). In particular the {\em relative
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
97 advantage} of deep learning for these settings has not been evaluated.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
98 The hypothesis discussed in the conclusion is that a deep hierarchy of features
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
99 may be better able to provide sharing of statistical strength
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
100 between different regions in input space or different tasks.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
101
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
102 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108 learning, often in an greedy layer-wise ``unsupervised pre-training''
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
109 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110 applied here, is the Denoising
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
111 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see Figure~\ref{fig:da}),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
112 which
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 Machines in terms of unsupervised extraction of a hierarchy of features
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
115 useful for classification. Each layer is trained to denoise its
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
116 input, creating a layer of features that can be used as input for the next layer.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
117 \fi
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
118 %The principle is that each layer starting from
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
119 %the bottom is trained to encode its input (the output of the previous
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
120 %layer) and to reconstruct it from a corrupted version. After this
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
121 %unsupervised initialization, the stack of DAs can be
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
122 %converted into a deep supervised feedforward neural network and fine-tuned by
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
123 %stochastic gradient descent.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
125 %
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
126 In this paper we ask the following questions:
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
127
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
128 %\begin{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
129 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
130 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
131 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
132 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
133
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
134 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
135 To what extent does the perturbation of input images (e.g. adding
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
136 noise, affine transformations, background images) make the resulting
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
137 classifiers better not only on similarly perturbed images but also on
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
138 the {\em original clean examples}? We study this question in the
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
139 context of the 62-class and 10-class tasks of the NIST special database 19.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
140
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
141 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
142 Do deep architectures {\em benefit more from such out-of-distribution}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
143 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
144 We use highly perturbed examples to generate out-of-distribution examples.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
145
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
146 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
147 Similarly, does the feature learning step in deep learning algorithms benefit more
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
148 from training with moderately different classes (i.e. a multi-task learning scenario) than
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
149 a corresponding shallow and purely supervised architecture?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
150 We train on 62 classes and test on 10 (digits) or 26 (upper case or lower case)
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
151 to answer this question.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
152 %\end{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
153
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
154 Our experimental results provide positive evidence towards all of these questions.
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
155 To achieve these results, we introduce in the next section a sophisticated system
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
156 for stochastically transforming character images and then explain the methodology,
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
157 which is based on training with or without these transformed images and testing on
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
158 clean ones. We measure the relative advantage of out-of-distribution examples
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
159 for a deep learner vs a supervised shallow one.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
160 Code for generating these transformations as well as for the deep learning
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
161 algorithms are made available.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
162 We also estimate the relative advantage for deep learners of training with
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
163 other classes than those of interest, by comparing learners trained with
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
164 62 classes with learners trained with only a subset (on which they
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
165 are then tested).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
166 The conclusion discusses
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
167 the more general question of why deep learners may benefit so much from
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
168 the self-taught learning framework.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
170 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 \section{Perturbation and Transformation of Character Images}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
172 \label{s:perturbations}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
173 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
175 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
176 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
177 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
178 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
179 \includegraphics[scale=.4]{images/Original.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
180 {\bf Original}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
181 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
182 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
183 %\vspace{0.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
184 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
185 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
186 This section describes the different transformations we used to stochastically
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
187 transform $32 \times 32$ source images (such as the one on the left)
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
188 in order to obtain data from a larger distribution which
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
189 covers a domain substantially larger than the clean characters distribution from
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
190 which we start.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
191 Although character transformations have been used before to
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
192 improve character recognizers, this effort is on a large scale both
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
193 in number of classes and in the complexity of the transformations, hence
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
194 in the complexity of the learning task.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
195 More details can
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
196 be found in this technical report~\citep{ift6266-tr-anonymous}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
200 amount of deformation or noise introduced.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
201 There are two main parts in the pipeline. The first one,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
202 from slant to pinch below, performs transformations. The second
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
203 part, from blur to contrast, adds different kinds of noise.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
204 %\end{minipage}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
206 \vspace*{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
207 %\subsection{Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
208 {\large\bf 2.1 Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
209 \vspace*{1mm}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
210
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
211
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
212 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
213 %\begin{minipage}[b]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
214 %\centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
215 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
216 \vspace*{-5mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
217 \includegraphics[scale=.4]{images/Thick_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
218 {\bf Thickness}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
219 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
220 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
221 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
222 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
223 \end{wrapfigure}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
224 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
225 are applied. The neighborhood of each pixel is multiplied
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
226 element-wise with a {\em structuring element} matrix.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
227 The pixel value is replaced by the maximum or the minimum of the resulting
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
228 matrix, respectively for dilation or erosion. Ten different structural elements with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
229 increasing dimensions (largest is $5\times5$) were used. For each image,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
230 randomly sample the operator type (dilation or erosion) with equal probability and one structural
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
231 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
232 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
233 A neutral element (no transformation)
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
234 is always present in the set.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
235 %\vspace{.4cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
236 %\end{minipage}
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
237 \vspace*{3mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
238
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
239 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
240 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
241 \includegraphics[scale=.4]{images/Slant_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
242 {\bf Slant}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
243 \end{minipage}%
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
244 \hspace{0.3cm}
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
245 \begin{minipage}[b]{0.83\linewidth}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
246 %\centering
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
247 To produce {\bf slant}, each row of the image is shifted
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
248 proportionally to its height: $shift = round(slant \times height)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
249 $slant \sim U[-complexity,complexity]$.
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
250 The shift is randomly chosen to be either to the left or to the right.
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
251 \vspace{8mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
252 \end{minipage}
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
253 \vspace*{3mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
254
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
255 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
256 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
257 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
258 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
259 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
260 \includegraphics[scale=.4]{images/Affine_only.png}\\
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
261 {\small {\bf Affine \mbox{Transformation}}}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
262 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
263 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
264 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
265 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
266 A $2 \times 3$ {\bf affine transform} matrix (with
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
267 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
268 Output pixel $(x,y)$ takes the value of input pixel
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
269 nearest to $(ax+by+c,dx+ey+f)$,
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
270 producing scaling, translation, rotation and shearing.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
271 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
272 forbid large rotations (to avoid confusing classes) but to give good
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
273 variability of the transformation: $a$ and $d$ $\sim U[1-3
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
274 complexity,1+3\,complexity]$, $b$ and $e$ $\sim U[-3 \,complexity,3\,
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
275 complexity]$, and $c$ and $f \sim U[-4 \,complexity, 4 \,
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
276 complexity]$.\\
582
9ebb335ca904 techreport.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 581
diff changeset
277 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
278
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
279 \vspace*{-4.5mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
280
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
281 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
282 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
283 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
284 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
285 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
286 \vspace*{-4mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
287 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.png}\\
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
288 {\bf Local Elastic Deformation}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
289 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
290 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
291 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
292 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
293 %\vspace*{-20mm}
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
294 The {\bf local elastic deformation}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
295 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
296 which provides more details.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
297 The intensity of the displacement fields is given by
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
298 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
299 convolved with a Gaussian 2D kernel (resulting in a blur) of
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
300 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
301 %\vspace{.9cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
302 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
303
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
304 \vspace*{7mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
305
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
306 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
307 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
308 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
309 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
310 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
311 \includegraphics[scale=.4]{images/Pinch_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
312 {\bf Pinch}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
313 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
314 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
315 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
316 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
317 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
318 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl set to 0.
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
319 A pinch is ``similar to projecting the image onto an elastic
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
320 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
321 For a square input image, draw a radius-$r$ disk
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
322 around its center $C$. Any pixel $P$ belonging to
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
323 that disk has its value replaced by
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
324 the value of a ``source'' pixel in the original image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
325 on the line that goes through $C$ and $P$, but
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
326 at some other distance $d_2$. Define $d_1=distance(P,C)$
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
327 and $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
328 d_1$, where $pinch$ is a parameter of the filter.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
329 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
330 around the (non-integer) source position thus found.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
331 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
332 %\vspace{1.5cm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
333 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
334
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
335 \vspace{1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
336
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
337 {\large\bf 2.2 Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
338 %\subsection{Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
339 \vspace{2mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
340
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
341 %\vspace*{-.2cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
342 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
343 \centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
344 \vspace*{-2mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
345 \includegraphics[scale=.4]{images/Motionblur_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
346 {\bf Motion Blur}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
347 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
348 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
349 %\vspace*{.5mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
350 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
351 has parameters $length$ and $angle$. The value of
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
352 a pixel in the final image is approximately the mean of the first $length$ pixels
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
353 found by moving in the $angle$ direction,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
354 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
355 \vspace{5mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
356 \end{minipage}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
357
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
358 \vspace*{1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
359
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
360 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
361 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
362 \includegraphics[scale=.4]{images/occlusion_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
363 {\bf Occlusion}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
364 %\vspace{.5cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
365 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
366 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
367 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
368 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
369 image and places it over the original {\em occluded}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
370 image. Pixels are combined by taking the max(occluder, occluded),
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
371 i.e. keeping the lighter ones.
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
372 The rectangle corners
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
373 are sampled so that larger complexity gives larger rectangles.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
374 The destination position in the occluded image are also sampled
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
375 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
376 This module is skipped with probability 60\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
377 %\vspace{7mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
378 \end{minipage}
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
379
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
380 \vspace*{1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
381
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
382 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
383 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
384 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
385 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
386 %\centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
387 \includegraphics[scale=.4]{images/Bruitgauss_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
388 {\bf Gaussian Smoothing}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
389 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
390 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
391 %\vspace{.5cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
392 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
393 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
394 With the {\bf Gaussian smoothing} module,
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
395 different regions of the image are spatially smoothed.
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
396 This is achieved by first convolving
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
397 the image with an isotropic Gaussian kernel of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
398 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
399 complexity]$ and $[2,2 + 6 \times complexity]$. This filtered image is normalized
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
400 between $0$ and $1$. We also create an isotropic weighted averaging window, of the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
402 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 initialize to zero a mask matrix of the image size. For each selected pixel
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
405 we add to the mask the averaging window centered on it. The final image is
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
406 computed from the following element-wise operation: $\frac{image + filtered\_image
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
407 \times mask}{mask+1}$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
408 This module is skipped with probability 75\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
409 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
410
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
411 \newpage
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
412
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
413 \vspace*{-9mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
414
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
415 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
416 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
417 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
418 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
419 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
420 \begin{center}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
421 \includegraphics[scale=.4]{images/Permutpixel_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
422 {\small\bf Permute Pixels}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
423 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
424 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
425 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
426 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
427 %\vspace*{-20mm}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
428 This module {\bf permutes neighbouring pixels}. It first selects a
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
429 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
430 of these pixels is then sequentially exchanged with a random pixel
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
431 among its four nearest neighbors (on its left, right, top or bottom).
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
432 This module is skipped with probability 80\%.\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
433 \vspace*{1mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
434 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
435
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
436 \vspace{-3mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
437
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
438 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
439 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
440 %\vspace*{-3mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
441 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
442 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
443 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
444 \vspace*{-5mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
445 \includegraphics[scale=.4]{images/Distorsiongauss_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
446 {\small \bf Gauss. Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
447 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
448 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
449 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
450 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
451 \vspace*{12mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
452 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
453 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
454 This module is skipped with probability 70\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
455 %\vspace{1.1cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
456 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
457
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
458 \vspace*{1.2cm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
459
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
460 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
461 \begin{minipage}[t]{0.14\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
462 \centering
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
463 \includegraphics[scale=.4]{images/background_other_only.png}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
464 {\small \bf Bg Image}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
465 \end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
466 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
467 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
468 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
469 background image behind the letter, from a randomly chosen natural image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
470 with contrast adjustments depending on $complexity$, to preserve
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
471 more or less of the original character image.
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
472 %\vspace{.8cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
473 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
474 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
475 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
476
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
477 \begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
478 \centering
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
479 \includegraphics[scale=.4]{images/Poivresel_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
480 {\small \bf Salt \& Pepper}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
481 \end{minipage}%
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
482 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
483 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
484 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
485 The number of selected pixels is $0.2 \times complexity$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
486 This module is skipped with probability 75\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
487 %\vspace{.9cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
488 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
489 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
490
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
491 \vspace{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
492
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
493 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
494 \begin{wrapfigure}[7]{l}{0.14\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
495 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
496 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
497 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
498 \vspace*{-4mm}
558
143a1467f157 Fixed compilation: .PNG -> .png
Olivier Delalleau <delallea@iro>
parents: 555
diff changeset
499 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.png}\\
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
500 {\bf Scratches}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
501 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
502 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
503 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
504 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
505 %\vspace{.4cm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
506 The {\bf scratches} module places line-like white patches on the image. The
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
507 lines are heavily transformed images of the digit ``1'' (one), chosen
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
508 at random among 500 such 1 images,
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
509 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
554
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 553
diff changeset
510 complexity)^2$ (in degrees), using bi-cubic interpolation.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
511 Two passes of a grey-scale morphological erosion filter
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
512 are applied, reducing the width of the line
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
513 by an amount controlled by $complexity$.
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
514 This module is skipped with probability 85\%. The probabilities
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
515 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
516 \end{minipage}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
517
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
518 \vspace*{1mm}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
519
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
520 \begin{minipage}[t]{0.25\linewidth}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
521 \centering
559
Olivier Delalleau <delallea@iro>
parents: 558 557
diff changeset
522 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.png}\\
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
523 {\bf Grey Level \& Contrast}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
524 \end{minipage}%
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
525 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
526 \vspace*{-18mm}
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
527 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
544
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
528 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
529 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
1cdfc17e890f ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 541
diff changeset
530 polarity is inverted with probability 50\%.
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
531 %\vspace{.7cm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
532 \end{minipage}
555
b6dfba0a110c ameliorer l'aspect visuel, Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 554
diff changeset
533 \vspace{2mm}
551
8f365abf171d separete the transmo image
Frederic Bastien <nouiz@nouiz.org>
parents: 550
diff changeset
534
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
535
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
536 \iffalse
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
537 \begin{figure}[ht]
538
f0ee2212ea7c typos and stuff
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 537 534
diff changeset
538 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
539 \caption{Illustration of the pipeline of stochastic
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
540 transformations applied to the image of a lower-case \emph{t}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
541 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
542 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
543 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
544 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
545 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
546 \end{figure}
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
547 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
548
560
dc5c3f538a05 Small fixes (typos / precisions)
Olivier Delalleau <delallea@iro>
parents: 559
diff changeset
549 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
550 \section{Experimental Setup}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
551 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
552
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
553 Much previous work on deep learning had been performed on
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
554 the MNIST digits task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
555 with 60~000 examples, and variants involving 10~000
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
556 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}.
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
557 The focus here is on much larger training sets, from 10 times to
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
558 to 1000 times larger, and 62 classes.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
559
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
560 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
561 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
562 and {\bf OCR data} (scanned machine printed characters). Once a character
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
563 is sampled from one of these sources (chosen randomly), the second step is to
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
564 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
565
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
566 To provide a baseline of error rate comparison we also estimate human performance
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
567 on both the 62-class task and the 10-class digits task.
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
568 We compare the best Multi-Layer Perceptrons (MLP) against
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
569 the best Stacked Denoising Auto-encoders (SDA), when
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
570 both models' hyper-parameters are selected to minimize the validation set error.
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
571 We also provide a comparison against a precise estimate
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
572 of human performance obtained via Amazon's Mechanical Turk (AMT)
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
573 service (http://mturk.com).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
574 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
575 of money to perform tasks for which human intelligence is required.
522
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
576 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
577 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
578 %\citep{SorokinAndForsyth2008,whitehill09}.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
579 AMT users were presented
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
580 with 10 character images (from a test set) and asked to choose 10 corresponding ASCII
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
581 characters. They were forced to choose a single character class (either among the
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
582 62 or 10 character classes) for each image.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
583 80 subjects classified 2500 images per (dataset,task) pair,
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
584 with the guarantee that 3 different subjects classified each image, allowing
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
585 us to estimate inter-human variability (e.g a standard error of 0.1\%
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
586 on the average 18.2\% error done by humans on the 62-class task NIST test set).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
587
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
588 \vspace*{-3mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
589 \subsection{Data Sources}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
590 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
591
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
592 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
593 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
594 {\bf NIST.}
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
595 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
596 widely used for training and testing character
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
597 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
598 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
599 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
600 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
601 The fourth partition (called $hsf_4$, 82587 examples),
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
602 experimentally recognized to be the most difficult one, is the one recommended
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
603 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
604 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
605 model selection.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
606 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
607 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
608 useful to estimate the effect of a multi-task setting.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
609 The distribution of the classes in the NIST training and test sets differs
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
610 substantially, with relatively many more digits in the test set, and a more uniform distribution
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
611 of letters in the test set (whereas in the training set they are distributed
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
612 more like in natural text).
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
613 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
614
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
615 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
616 {\bf Fonts.}
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
617 In order to have a good variety of sources we downloaded an important number of free fonts from:
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
618 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
619 % TODO: pointless to anonymize, it's not pointing to our work
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
620 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
621 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
622 directly as input to our models.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
623 \vspace*{-1mm}
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
624
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
625 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
626 {\bf Captchas.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
627 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
628 generating characters of the same format as the NIST dataset. This software is based on
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
629 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
630 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
631 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
632 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
633 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
634 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
635
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
636 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
637 {\bf OCR data.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
638 A large set (2 million) of scanned, OCRed and manually verified machine-printed
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
639 characters where included as an
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
640 additional source. This set is part of a larger corpus being collected by the Image Understanding
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
641 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
642 ({\tt http://www.iupr.com}), and which will be publicly released.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
643 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
644 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
645
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
646 \vspace*{-3mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
647 \subsection{Data Sets}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
648 \vspace*{-2mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
649
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
650 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
651 from one of the 62 character classes.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
652 %\begin{itemize}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
653 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
654
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
655 %\item
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
656 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
657 \{651668 / 80000 / 82587\} \{training / validation / test\} examples.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
658 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
659
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
660 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
661 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
662 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
663 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
664 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
665 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
666 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
667 \vspace*{-1mm}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
668
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
669 %\item
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
670 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
671 except that we only apply
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
672 transformations from slant to pinch. Therefore, the character is
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
673 transformed but no additional noise is added to the image, giving images
534
4d6493d171f6 added all sizes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 533
diff changeset
674 closer to the NIST dataset.
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
675 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
676 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
677
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
678 \vspace*{-3mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
679 \subsection{Models and their Hyperparameters}
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
680 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
681
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
682 The experiments are performed using MLPs (with a single
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
683 hidden layer) and SDAs.
553
8f6c09d1140f ca fitte de nouveau
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 552
diff changeset
684 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
685
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
686 {\bf Multi-Layer Perceptrons (MLP).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
687 Whereas previous work had compared deep architectures to both shallow MLPs and
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
688 SVMs, we only compared to MLPs here because of the very large datasets used
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
689 (making the use of SVMs computationally challenging because of their quadratic
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
690 scaling behavior).
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
691 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
692 exponentials) on the output layer for estimating $P(class | image)$.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
693 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
694 Training examples are presented in minibatches of size 20. A constant learning
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
695 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$.
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
696 %through preliminary experiments (measuring performance on a validation set),
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
697 %and $0.1$ (which was found to work best) was then selected for optimizing on
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
698 %the whole training sets.
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
699 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
700
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
701
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
702 {\bf Stacked Denoising Auto-Encoders (SDA).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
703 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
704 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
705 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
706 apparently setting parameters in the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
707 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
708 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
709 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
710 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
711 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
712 $P(y|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
713 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
714 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
715 compositions of simpler ones through a deep hierarchy).
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
716
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
717 \begin{figure}[ht]
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
718 \vspace*{-2mm}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
719 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
720 \vspace*{-2mm}
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
721 \caption{Illustration of the computations and training criterion for the denoising
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
722 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
723 the layer (i.e. raw input or output of previous layer)
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
724 s corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
725 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
726 is compared to the uncorrupted input $x$ through the loss function
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
727 $L_H(x,z)$, whose expected value is approximately minimized during training
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
728 by tuning $\theta$ and $\theta'$.}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
729 \label{fig:da}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
730 \vspace*{-2mm}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
731 \end{figure}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
732
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
733 Here we chose to use the Denoising
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
734 Auto-encoder~\citep{VincentPLarochelleH2008} as the building block for
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
735 these deep hierarchies of features, as it is simple to train and
532
2e33885730cf changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 529
diff changeset
736 explain (see Figure~\ref{fig:da}, as well as
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
737 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
738 provides efficient inference, and yielded results
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
739 comparable or better than RBMs in series of experiments
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
740 \citep{VincentPLarochelleH2008}. During training, a Denoising
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
741 Auto-encoder is presented with a stochastically corrupted version
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
742 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
743 forcing the hidden units to represent the leading regularities in
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
744 the data. Here we use the random binary masking corruption
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
745 (which sets to 0 a random subset of the inputs).
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
746 Once it is trained, in a purely unsupervised way,
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
747 its hidden units' activations can
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
748 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
749 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
750 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
751 the same standard procedure used to train them (see previous section).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
752 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
753 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
754 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
755 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
756 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
757 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
758 of hidden layers but it was fixed to 3 based on previous work with
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
759 SDAs on MNIST~\citep{VincentPLarochelleH2008}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
760
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
761 \vspace*{-1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
762
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
763 \begin{figure}[ht]
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
764 \vspace*{-2mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
765 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
766 \vspace*{-3mm}
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
767 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
768 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
548
34cb28249de0 suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 547
diff changeset
769 of all models, on NIST and NISTP test sets.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
770 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
771 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
772 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
773 \label{fig:error-rates-charts}
541
8aad1c6ec39a reduction espace
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 539
diff changeset
774 \vspace*{-2mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
775 \end{figure}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
776
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
777
557
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
778 \begin{figure}[ht]
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
779 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
780 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
781 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
782 \caption{Relative improvement in error rate due to self-taught learning.
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
783 Left: Improvement (or loss, when negative)
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
784 induced by out-of-distribution examples (perturbed data).
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
785 Right: Improvement (or loss, when negative) induced by multi-task
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
786 learning (training on all classes and testing only on either digits,
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
787 upper case, or lower-case). The deep learner (SDA) benefits more from
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
788 both self-taught learning scenarios, compared to the shallow MLP.}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
789 \label{fig:improvements-charts}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
790 \vspace*{-2mm}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
791 \end{figure}
17d16700e0c8 encore du visuel de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 555
diff changeset
792
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
793 \section{Experimental Results}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
794 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
795
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
796 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
797 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
798 %\vspace*{-1mm}
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
799 The models are either trained on NIST (MLP0 and SDA0),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
800 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
801 on either NIST, NISTP or P07, either on the 62-class task
568
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
802 or on the 10-digits task. Training (including about half
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
803 for unsupervised pre-training, for DAs) on the larger
ae6ba0309bf9 nouveaux graphes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 566
diff changeset
804 datasets takes around one day on a GPU-285.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
805 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
806 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
807 SDA2), along with the previous results on the digits NIST special database
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
808 19 test set from the literature, respectively based on ARTMAP neural
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
809 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
810 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
811 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
812 (figures and tables, including standard errors on the error rates) can be
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
813 found in Appendix I of the supplementary material.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
814 The deep learner not only outperformed the shallow ones and
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
815 previously published performance (in a statistically and qualitatively
535
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
816 significant way) but when trained with perturbed data
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 534
diff changeset
817 reaches human performance on both the 62-class task
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
818 and the 10-class (digits) task.
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
819 17\% error (SDA1) or 18\% error (humans) may seem large but a large
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
820 majority of the errors from humans and from SDA1 are from out-of-context
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
821 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
822 ``c'' and a ``C'' are often indistinguishible).
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
823
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
824 In addition, as shown in the left of
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
825 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
826 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
827 differences with the MLP are statistically and qualitatively
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
828 significant.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
829 The left side of the figure shows the improvement to the clean
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
830 NIST test set error brought by the use of out-of-distribution examples
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
831 (i.e. the perturbed examples examples from NISTP or P07).
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
832 Relative percent change is measured by taking
548
34cb28249de0 suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 547
diff changeset
833 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
834 The right side of
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
835 Figure~\ref{fig:improvements-charts} shows the relative improvement
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
836 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
837 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
838 with all 62 classes when the target classes are respectively the digits,
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
839 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
840 multi-task setting is marginal or negative for the MLP, it is substantial
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
841 for the SDA. Note that to simplify these multi-task experiments, only the original
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
842 NIST dataset is used. For example, the MLP-digits bar shows the relative
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
843 percent improvement in MLP error rate on the NIST digits test set
566
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
844 is $100\% \times$ (single-task
b9b811e886ae Small fixes
Olivier Delalleau <delallea@iro>
parents: 560
diff changeset
845 model's error / multi-task model's error - 1). The single-task model is
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
846 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
847 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
848 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
849 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
850 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
851 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
852 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
853 and upper case characters).
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
854 %\vspace*{-1mm}
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
855 %\subsection{Perturbed Training Data More Helpful for SDA}
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
856 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
857
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
858 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
859 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
860 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
861
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
862 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
863 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
864 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
865 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
866 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
867 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
868 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
869 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
870 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
871 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
872 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
873 shallow learner.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
874
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
875 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
876 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
877 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
878
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
879 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
880 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
881 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
882 On the other hand the SDA benefited from the multi-task setting, with relative
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
883 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
884 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
885 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
886
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
887
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
888 \vspace*{-2mm}
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
889 \section{Conclusions and Discussion}
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
890 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
891
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
892 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
893 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
894 supervised learner. More precisely,
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
895 the answers are positive for all the questions asked in the introduction.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
896 %\begin{itemize}
487
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 486
diff changeset
897
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
898 $\bullet$ %\item
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
899 {\bf Do the good results previously obtained with deep architectures on the
549
ef172f4a322a ca fitte
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 548
diff changeset
900 MNIST digits generalize to a much larger and richer (but similar)
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
901 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
902 Yes, the SDA {\em systematically outperformed the MLP and all the previously
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
903 published results on this dataset} (the ones that we are aware of), {\em in fact reaching human-level
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
904 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
905
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
906 $\bullet$ %\item
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
907 {\bf To what extent do self-taught learning scenarios help deep learners,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
908 and do they help them more than shallow supervised ones}?
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
909 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
910 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
911 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
912 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
913 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
914 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
915 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
916 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
917 with respect to clean examples . On the other hand, the deep SDAs
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
918 were significantly boosted by these out-of-distribution examples.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
919 Similarly, whereas the improvement due to the multi-task setting was marginal or
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
920 negative for the MLP (from +5.6\% to -3.6\% relative change),
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
921 it was quite significant for the SDA (from +13\% to +27\% relative change),
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
922 which may be explained by the arguments below.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
923 %\end{itemize}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
924
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
925 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
926 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
927 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
928 scenario. However, many of the results by \citet{RainaR2007} (who used a
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
929 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
930 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
931 We note instead that, for deep
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
932 architectures, our experiments show that such a positive effect is accomplished
569
9d01280ff1c1 commentaires de Joseph Turian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 568
diff changeset
933 even in a scenario with a \emph{large number of labeled examples},
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
934 i.e., here, the relative gain of self-taught learning is probably preserved
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
935 in the asymptotic regime.
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
936
547
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 544
diff changeset
937 {\bf Why would deep learners benefit more from the self-taught learning framework}?
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
938 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
939 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
940 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
941 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
942 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
943 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
944 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
945 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
946 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
947 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
948 experiments could attempt at teasing apart these factors.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
949 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
950 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
951 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
952 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
953 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
954 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
955 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
956 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
957 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
958 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
959 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
960 (with or without self-taught settings), and more labeled examples
550
662299f265ab suggestions from Ian
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 549
diff changeset
961 does not allow the model to go from the poorer basins of attraction discovered
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
962 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
963 with deep learning and self-taught learning.
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
964
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
965 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
966 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
967
498
7ff00c27c976 add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents: 496
diff changeset
968 \newpage
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
969 {
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
970 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
971 %\bibliographystyle{plainnat}
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
972 \bibliographystyle{unsrtnat}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
973 %\bibliographystyle{apalike}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
974 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
975
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
976
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
977 \end{document}