annotate writeup/nips2010_submission.tex @ 536:5157a5830125

One comma
author Dumitru Erhan <dumitru.erhan@gmail.com>
date Tue, 01 Jun 2010 18:28:09 -0700
parents 85f2337d47d2
children 47894d0ecbde
rev   line source
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
5 \usepackage[psamsfonts]{amssymb}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{graphicx,subfigure}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
9 \usepackage[numbers]{natbib}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
12
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16 \begin{document}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 \maketitle
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
21 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 \begin{abstract}
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
23 Recent theoretical and empirical work in statistical machine learning has
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
24 demonstrated the importance of learning algorithms for deep
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
25 architectures, i.e., function classes obtained by composing multiple
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
26 non-linear transformations. Self-taught learning (exploiting unlabeled
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
27 examples or examples from other distributions) has already been applied
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
28 to deep learners, but mostly to show the advantage of unlabeled
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
29 examples. Here we explore the advantage brought by {\em out-of-distribution
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
30 examples} and show that {\em deep learners benefit more from them than a
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
31 corresponding shallow learner}, in the area
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
32 of handwritten character recognition. In fact, we show that they reach
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
33 human-level performance on both handwritten digit classification and
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
34 62-class handwritten character recognition. For this purpose we
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
35 developed a powerful generator of stochastic variations and noise
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
36 processes character images, including not only affine transformations but
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
37 also slant, local elastic deformations, changes in thickness, background
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
38 images, grey level changes, contrast, occlusion, and various types of pixel and
482
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
39 spatially correlated noise. The out-of-distribution examples are
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
40 obtained by training with these highly distorted images or
ce69aa9204d8 changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 480
diff changeset
41 by including object classes different from those in the target test set.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 \end{abstract}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
43 \vspace*{-2mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 \section{Introduction}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
46 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 Deep Learning has emerged as a promising new area of research in
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
49 statistical machine learning (see~\citet{Bengio-2009} for a review).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 which consists of a chain of processing elements, each of which is associated with a
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
54 different representation of the raw visual input. In fact,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 those observed in the first two of these stages (in areas V1 and V2
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
57 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
58 more invariant to factors of variation (such as camera movement) in
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
59 higher layers~\citep{Goodfellow2009}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
60 Learning a hierarchy of features increases the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73 learning, often in an greedy layer-wise ``unsupervised pre-training''
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
74 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 applied here, is the Denoising
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
76 Auto-Encoder~(DEA)~\citep{VincentPLarochelleH2008-very-small}, which
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 useful for classification. The principle is that each layer starting from
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
80 the bottom is trained to encode its input (the output of the previous
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
81 layer) and to reconstruct it from a corrupted version. After this
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 unsupervised initialization, the stack of denoising auto-encoders can be
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
83 converted into a deep supervised feedforward neural network and fine-tuned by
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
86 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
87 of semi-supervised and multi-task learning: the learner can exploit examples
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
88 that are unlabeled and/or come from a distribution different from the target
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
89 distribution, e.g., from other classes that those of interest. Whereas
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
90 it has already been shown that deep learners can clearly take advantage of
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
91 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
92 and multi-task learning, not much has been done yet to explore the impact
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
93 of {\em out-of-distribution} examples and of the multi-task setting
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
94 (but see~\citep{CollobertR2008}). In particular the {\em relative
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
95 advantage} of deep learning for this settings has not been evaluated.
512
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
96 The hypothesis explored here is that a deep hierarchy of features
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
97 may be better able to provide sharing of statistical strength
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
98 between different regions in input space or different tasks,
6f042a71be23 todo done
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 507
diff changeset
99 as discussed in the conclusion.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
100
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
101 In this paper we ask the following questions:
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
102
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
103 %\begin{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
104 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
105 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
106 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
107 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
108
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
109 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
110 To what extent does the perturbation of input images (e.g. adding
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
111 noise, affine transformations, background images) make the resulting
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
112 classifiers better not only on similarly perturbed images but also on
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
113 the {\em original clean examples}?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
114
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
115 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
116 Do deep architectures {\em benefit more from such out-of-distribution}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
117 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
118
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
119 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
120 Similarly, does the feature learning step in deep learning algorithms benefit more
466
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
121 training with similar but different classes (i.e. a multi-task learning scenario) than
6205481bf33f asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 464
diff changeset
122 a corresponding shallow and purely supervised architecture?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
123 %\end{enumerate}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
124
511
d057941417ed a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 510
diff changeset
125 Our experimental results provide positive evidence towards all of these questions.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
127 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
128 \section{Perturbation and Transformation of Character Images}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
129 \label{s:perturbations}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
130 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
131
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
132 This section describes the different transformations we used to stochastically
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
133 transform source images in order to obtain data. More details can
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
134 be found in this technical report~\citep{ift6266-tr-anonymous}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
138 amount of deformation or noise introduced.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
139
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
140 There are two main parts in the pipeline. The first one,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
141 from slant to pinch below, performs transformations. The second
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
142 part, from blur to contrast, adds different kinds of noise.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
144 \begin{figure}[ht]
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
145 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/transfo.png}}}
506
8bf07979b8ba desiderata
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 503
diff changeset
146 % TODO: METTRE LE NOM DE LA TRANSFO A COTE DE CHAQUE IMAGE
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
147 \caption{Illustration of each transformation applied alone to the same image
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
148 of an upper-case h (top left). First row (from left to right) : original image, slant,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
149 thickness, affine transformation (translation, rotation, shear),
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
150 local elastic deformation; second row (from left to right) :
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
151 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to right) :
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
152 background image, salt and pepper noise, spatially Gaussian noise, scratches,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
153 grey level and contrast changes.}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
154 \label{fig:transfo}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
155 \end{figure}
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
156
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
157 {\large\bf Transformations}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
158
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
159 \vspace*{2mm}
483
b9cdb464de5f pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 482
diff changeset
160
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
161 {\bf Slant.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
162 We mimic slant by shifting each row of the image
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
163 proportionally to its height: $shift = round(slant \times height)$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 The $slant$ coefficient can be negative or positive with equal probability
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
165 and its value is randomly sampled according to the complexity level:
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
166 $slant \sim U[0,complexity]$, so the
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 maximum displacement for the lowest or highest pixel line is of
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
168 $round(complexity \times 32)$.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
169 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
170
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
171 {\bf Thickness.}
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
172 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
173 are applied. The neighborhood of each pixel is multiplied
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
174 element-wise with a {\em structuring element} matrix.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
175 The pixel value is replaced by the maximum or the minimum of the resulting
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
176 matrix, respectively for dilation or erosion. Ten different structural elements with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
177 increasing dimensions (largest is $5\times5$) were used. For each image,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
178 randomly sample the operator type (dilation or erosion) with equal probability and one structural
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 element from a subset of the $n$ smallest structuring elements where $n$ is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
181 for erosion. A neutral element is always present in the set, and if it is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
182 chosen no transformation is applied. Erosion allows only the six
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 smallest structural elements because when the character is too thin it may
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
184 be completely erased.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
185 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
186
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
187 {\bf Affine Transformations.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
188 A $2 \times 3$ affine transform matrix (with
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
189 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
190 Each pixel $(x,y)$ of the output image takes the value of the pixel
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
191 nearest to $(ax+by+c,dx+ey+f)$ in the input image. This
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
192 produces scaling, translation, rotation and shearing.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
193 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 forbid important rotations (not to confuse classes) but to give good
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
195 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
196 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
197 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
198 complexity]$.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
199 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
200
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
201 {\bf Local Elastic Deformations.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
202 This filter induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
203 which provides more details.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
204 Two ``displacements'' fields are generated and applied, for horizontal
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
205 and vertical displacements of pixels.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 To generate a pixel in either field, first a value between -1 and 1 is
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
207 chosen from a uniform distribution. Then all the pixels, in both fields, are
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 multiplied by a constant $\alpha$ which controls the intensity of the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
209 displacements (larger $\alpha$ translates into larger wiggles).
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
210 Each field is convoluted with a Gaussian 2D kernel of
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
211 standard deviation $\sigma$. Visually, this results in a blur.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
213 \sqrt[3]{complexity}$.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
214 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
215
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
216 {\bf Pinch.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
217 This is a GIMP filter called ``Whirl and
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
218 pinch'', but whirl was set to 0. A pinch is ``similar to projecting the image onto an elastic
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
219 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
220 For a square input image, this is akin to drawing a circle of
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
221 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
222 that disk (region inside circle) will have its value recalculated by taking
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
223 the value of another ``source'' pixel in the original image. The position of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
224 that source pixel is found on the line that goes through $C$ and $P$, but
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
225 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
226 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
227 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
228 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
229 around the (non-integer) source position thus found.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
230 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
231
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
232 \vspace*{1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
233
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
234 {\large\bf Injecting Noise}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
235
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
236 \vspace*{1mm}
483
b9cdb464de5f pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 482
diff changeset
237
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
238 {\bf Motion Blur.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
239 This is a ``linear motion blur'' in GIMP
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
240 terminology, with two parameters, $length$ and $angle$. The value of
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
241 a pixel in the final image is approximately the mean value of the $length$ first pixels
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
242 found by moving in the $angle$ direction.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
243 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
244 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
245
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
246 {\bf Occlusion.}
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
247 Selects a random rectangle from an {\em occluder} character
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
248 images and places it over the original {\em occluded} character
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
249 image. Pixels are combined by taking the max(occluder,occluded),
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
250 closer to black. The rectangle corners
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
251 are sampled so that larger complexity gives larger rectangles.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
252 The destination position in the occluded image are also sampled
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
253 according to a normal distribution (see more details in~\citet{ift6266-tr-anonymous}).
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
254 This filter has a probability of 60\% of not being applied.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
255 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
256
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
257 {\bf Pixel Permutation.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
258 This filter permutes neighbouring pixels. It selects first
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
259 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
260 sequentially exchanged with one other pixel in its $V4$ neighbourhood. The number
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
261 of exchanges to the left, right, top, bottom is equal or does not differ
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
262 from more than 1 if the number of selected pixels is not a multiple of 4.
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
263 % TODO: The previous sentence is hard to parse
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
264 This filter has a probability of 80\% of not being applied.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
265 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
266
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
267 {\bf Gaussian Noise.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
268 This filter simply adds, to each pixel of the image independently, a
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
269 noise $\sim Normal(0(\frac{complexity}{10})^2)$.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
270 It has a probability of 70\% of not being applied.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
271 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
272
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
273 {\bf Background Images.}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
274 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
275 background behind the letter. The background is chosen by first selecting,
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
276 at random, an image from a set of images. Then a 32$\times$32 sub-region
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
277 of that image is chosen as the background image (by sampling position
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
278 uniformly while making sure not to cross image borders).
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
279 To combine the original letter image and the background image, contrast
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
280 adjustments are made. We first get the maximal values (i.e. maximal
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
281 intensity) for both the original image and the background image, $maximage$
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
282 and $maxbg$. We also have a parameter $contrast \sim U[complexity, 1]$.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
283 Each background pixel value is multiplied by $\frac{max(maximage -
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
284 contrast, 0)}{maxbg}$ (higher contrast yield darker
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
285 background). The output image pixels are max(background,original).
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
286 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
287
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
288 {\bf Salt and Pepper Noise.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
289 This filter adds noise $\sim U[0,1]$ to random subsets of pixels.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
290 The number of selected pixels is $0.2 \times complexity$.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
291 This filter has a probability of not being applied at all of 75\%.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
292 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
293
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
294 {\bf Spatially Gaussian Noise.}
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
295 Different regions of the image are spatially smoothed.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
296 The image is convolved with a symmetric Gaussian kernel of
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
297 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
298 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
299 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
300 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
301 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
302 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
303 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
304 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
305 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
306 image \times mask}{mask+1}$.
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
307 This filter has a probability of not being applied at all of 75\%.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
308 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
309
474
bcf024e6ab23 fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 472
diff changeset
310 {\bf Scratches.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
311 The scratches module places line-like white patches on the image. The
517
0a5945249f2b section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 511
diff changeset
312 lines are heavily transformed images of the digit ``1'' (one), chosen
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
313 at random among five thousands such 1 images. The 1 image is
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
314 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
315 complexity)^2$, using bi-cubic interpolation,
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
316 Two passes of a grey-scale morphological erosion filter
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
317 are applied, reducing the width of the line
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
318 by an amount controlled by $complexity$.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
319 This filter is only applied only 15\% of the time. When it is applied, 50\%
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
320 of the time, only one patch image is generated and applied. In 30\% of
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
321 cases, two patches are generated, and otherwise three patches are
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
322 generated. The patch is applied by taking the maximal value on any given
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
323 patch or the original image, for each of the 32x32 pixel locations.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
324 \vspace*{0mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
325
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
326 {\bf Grey Level and Contrast Changes.}
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
327 This filter changes the contrast and may invert the image polarity (white
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
328 on black to black on white). The contrast $C$ is defined here as the
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
329 difference between the maximum and the minimum pixel value of the image.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
330 Contrast $\sim U[1-0.85 \times complexity,1]$ (so contrast $\geq 0.15$).
467
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 466
diff changeset
331 The image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
332 polarity is inverted with $0.5$ probability.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
333
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
334 \iffalse
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
335 \begin{figure}[ht]
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
336 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
337 \caption{Illustration of the pipeline of stochastic
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
338 transformations applied to the image of a lower-case \emph{t}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
339 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
340 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
341 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
342 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
343 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
344 \end{figure}
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
345 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
346
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
347
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
348 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
349 \section{Experimental Setup}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
350 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
351
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
352 Whereas much previous work on deep learning algorithms had been performed on
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
353 the MNIST digits classification task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
354 with 60~000 examples, and variants involving 10~000
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
355 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
356 to focus here on the case of much larger training sets, from 10 times to
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
357 to 1000 times larger.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
358
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
359 The first step in constructing the larger datasets is to sample from
499
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
360 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 495
diff changeset
361 and {\bf OCR data} (scanned machine printed characters). Once a character
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
362 is sampled from one of these sources (chosen randomly), the pipeline of
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
363 the transformations and/or noise processes described in section \ref{s:perturbations}
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
364 is applied to the image.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
365
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
366 We compare the best MLP against
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
367 the best SDA (both models' hyper-parameters are selected to minimize the validation set error),
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
368 along with a comparison against a precise estimate
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
369 of human performance obtained via Amazon's Mechanical Turk (AMT)
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
370 service (http://mturk.com).
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
371 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
372 of money to perform tasks for which human intelligence is required.
522
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
373 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
374 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 521
diff changeset
375 %\citep{SorokinAndForsyth2008,whitehill09}.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
376 AMT users were presented
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
377 with 10 character images and asked to choose 10 corresponding ASCII
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
378 characters. They were forced to make a hard choice among the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
379 62 or 10 character classes (all classes or digits only).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
380 Three users classified each image, allowing
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
381 to estimate inter-human variability. A total 2500 images/dataset were classified.
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
382
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
383 \vspace*{-1mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
384 \subsection{Data Sources}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
385 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
386
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
387 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
388 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
389 {\bf NIST.}
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
390 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
391 widely used for training and testing character
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
392 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
393 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
394 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
395 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
396 The fourth partition (called $hsf_4$), experimentally recognized to be the most difficult one, is the one recommended
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
397 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
398 for that purpose. We randomly split the remainder into a training set and a validation set for
480
150203d2b5c3 added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 479
diff changeset
399 model selection. The sizes of these data sets are: 651668 for training, 80000 for validation,
150203d2b5c3 added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 479
diff changeset
400 and 82587 for testing.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
402 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403 useful to estimate the effect of a multi-task setting.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 Note that the distribution of the classes in the NIST training and test sets differs
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
405 substantially, with relatively many more digits in the test set, and more uniform distribution
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
406 of letters in the test set, compared to the training set (in the latter, the letters are distributed
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
407 more like the natural distribution of letters in text).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
408
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
409 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
410 {\bf Fonts.}
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
411 In order to have a good variety of sources we downloaded an important number of free fonts from:
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
412 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
413 % TODO: pointless to anonymize, it's not pointing to our work
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
414 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
415 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
416 directly as input to our models.
6593e67381a3 Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 476
diff changeset
417
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
418 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
419 {\bf Captchas.}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
420 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
421 generating characters of the same format as the NIST dataset. This software is based on
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
422 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
423 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
424 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
425 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
426 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
427
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
428 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
429 {\bf OCR data.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
430 A large set (2 million) of scanned, OCRed and manually verified machine-printed
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
431 characters (from various documents and books) where included as an
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
432 additional source. This set is part of a larger corpus being collected by the Image Understanding
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
433 Pattern Recognition Research group lead by Thomas Breuel at University of Kaiserslautern
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
434 ({\tt http://www.iupr.com}), and which will be publicly released.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
435 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
436 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
437
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
438 \vspace*{-1mm}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
439 \subsection{Data Sets}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
440 \vspace*{-1mm}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
441
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
442 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
443 from one of the 62 character classes.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
444 %\begin{itemize}
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
445
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
446 %\item
501
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 500
diff changeset
447 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
448
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
449 %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
450 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
451 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
452 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
453 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
454 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
455
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
456 %\item
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
457 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
458 except that we only apply
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
459 transformations from slant to pinch. Therefore, the character is
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
460 transformed but no additional noise is added to the image, giving images
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
461 closer to the NIST dataset.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
462 %\end{itemize}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
463
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
464 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
465 \subsection{Models and their Hyperparameters}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
466 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
467
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
468 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
469 hidden layer and with Stacked Denoising Auto-Encoders (SDA).
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
470 \emph{Note that all hyper-parameters are selected based on performance on the {\bf NISTP} validation set.}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
471
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
472 {\bf Multi-Layer Perceptrons (MLP).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
473 Whereas previous work had compared deep architectures to both shallow MLPs and
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
474 SVMs, we only compared to MLPs here because of the very large datasets used
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
475 (making the use of SVMs computationally challenging because of their quadratic
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
476 scaling behavior).
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
477 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
478 exponentials) on the output layer for estimating $P(class | image)$.
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
479 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
480 Training examples are presented in minibatches of size 20. A constant learning
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
481 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
482 through preliminary experiments (measuring performance on a validation set),
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
483 and $0.1$ was then selected for optimizing on the whole training sets.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
484
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
485 \begin{figure}[ht]
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
486 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
487 \caption{Illustration of the computations and training criterion for the denoising
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
488 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
489 the layer (i.e. raw input or output of previous layer)
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
490 is corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
491 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
492 is compared to the uncorrupted input $x$ through the loss function
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
493 $L_H(x,z)$, whose expected value is approximately minimized during training
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
494 by tuning $\theta$ and $\theta'$.}
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
495 \label{fig:da}
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
496 \end{figure}
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
497
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
498 {\bf Stacked Denoising Auto-Encoders (SDA).}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
499 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
500 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
501 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
502 apparently setting parameters in the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
503 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
504 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
505 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
506 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
507 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
508 $P(y|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
509 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
510 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
511 compositions of simpler ones through a deep hierarchy).
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
512
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
513 Here we chose to use the Denoising
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
514 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
515 these deep hierarchies of features, as it is very simple to train and
521
13816dbef6ed des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 520
diff changeset
516 teach (see Figure~\ref{fig:da}, as well as
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
517 tutorial and code at {\tt http://deeplearning.net/tutorial}),
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
518 provides immediate and efficient inference, and yielded results
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
519 comparable or better than RBMs in series of experiments
519
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
520 \citep{VincentPLarochelleH2008}. During training, a Denoising
eaa595ea2402 section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 518
diff changeset
521 Auto-Encoder is presented with a stochastically corrupted version
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
522 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
523 forcing the hidden units to represent the leading regularities in
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
524 the data. Once it is trained, its hidden units activations can
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
525 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
526 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
527 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
528 the same standard procedure used to train them (see previous section).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
529 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
530 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
531 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
532 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
533 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
534 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
535 of hidden layers but it was fixed to 3 based on previous work with
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
536 stacked denoising auto-encoders on MNIST~\citep{VincentPLarochelleH2008}.
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
537
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
538 \vspace*{-1mm}
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
539
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
540 \begin{figure}[ht]
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
541 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
542 \caption{Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
543 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
530
8fe77eac344f Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 524
diff changeset
544 of all models, on 3 different test sets (NIST, NISTP, P07).
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
545 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
546 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
547 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
548
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
549 \label{fig:error-rates-charts}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
550 \vspace*{-1mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
551 \end{figure}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
552
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
553
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
554 \section{Experimental Results}
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
555
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
556 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
557 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
558 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
559
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
560 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
561 comparing Humans, three MLPs (MLP0, MLP1, MLP2) and three SDAs (SDA0, SDA1,
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
562 SDA2), along with the previous results on the digits NIST special database
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
563 19 test set from the literature respectively based on ARTMAP neural
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
564 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516
092dae9a5040 make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents: 514
diff changeset
565 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
566 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
567 (figures and tables, including standard errors on the error rates) can be
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
568 found in Appendix I of the supplementary material. The 3 kinds of model differ in the
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
569 training sets used: NIST only (MLP0,SDA0), NISTP (MLP1, SDA1), or P07
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
570 (MLP2, SDA2). The deep learner not only outperformed the shallow ones and
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
571 previously published performance (in a statistically and qualitatively
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
572 significant way) but reaches human performance on both the 62-class task
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
573 and the 10-class (digits) task.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
574
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
575 \begin{figure}[ht]
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
576 \vspace*{-2mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
577 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
578 \caption{Relative improvement in error rate due to self-taught learning.
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
579 Left: Improvement (or loss, when negative)
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
580 induced by out-of-distribution examples (perturbed data).
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
581 Right: Improvement (or loss, when negative) induced by multi-task
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
582 learning (training on all classes and testing only on either digits,
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
583 upper case, or lower-case). The deep learner (SDA) benefits more from
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
584 both self-taught learning scenarios, compared to the shallow MLP.}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
585 \label{fig:improvements-charts}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
586 \vspace*{-2mm}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
587 \end{figure}
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
588
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
589 In addition, as shown in the left of
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
590 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
591 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
592 differences with the MLP are statistically and qualitatively
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
593 significant.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
594 The left side of the figure shows the improvement to the clean
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
595 NIST test set error brought by the use of out-of-distribution examples
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
596 (i.e. the perturbed examples examples from NISTP or P07).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
597 Relative change is measured by taking
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
598 (original model's error / perturbed-data model's error - 1).
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
599 The right side of
523
c778d20ab6f8 space adjustments
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 522
diff changeset
600 Figure~\ref{fig:improvements-charts} shows the relative improvement
486
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
601 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
602 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 485
diff changeset
603 with all 62 classes when the target classes are respectively the digits,
493
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
604 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
605 multi-task setting is marginal or negative for the MLP, it is substantial
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
606 for the SDA. Note that for these multi-task experiment, only the original
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
607 NIST dataset is used. For example, the MLP-digits bar shows the relative
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
608 improvement in MLP error rate on the NIST digits test set (1 - single-task
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
609 model's error / multi-task model's error). The single-task model is
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
610 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
611 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
612 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
613 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
614 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
615 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
616 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 491
diff changeset
617 and upper case characters).
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
618
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
619 %\vspace*{-1mm}
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
620 %\subsection{Perturbed Training Data More Helpful for SDA}
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
621 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
622
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
623 %\vspace*{-1mm}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
624 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
625 %\vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
626
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
627 \iffalse
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
628 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
629 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
630 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
631 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
632 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
633 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
634 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
635 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
636 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
637 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
638 shallow learner.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
639
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
640 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
641 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
642 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
643
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
644 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
645 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
646 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 493
diff changeset
647 On the other hand the SDA benefited from the multi-task setting, with relative
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
648 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
649 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
650 \fi
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
651
475
ead3085c1c66 Added charts to nips2010_submission.tex
fsavard
parents: 469
diff changeset
652
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
653 \vspace*{-1mm}
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
654 \section{Conclusions and Discussion}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
655 \vspace*{-1mm}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
656
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
657 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
658 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
659 supervised learner. More precisely,
520
18a6379999fd more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 519
diff changeset
660 the answers are positive for all the questions asked in the introduction.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
661 %\begin{itemize}
487
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 486
diff changeset
662
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
663 $\bullet$ %\item
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
664 Do the good results previously obtained with deep architectures on the
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
665 MNIST digits generalize to the setting of a much larger and richer (but similar)
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
666 dataset, the NIST special database 19, with 62 classes and around 800k examples?
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
667 Yes, the SDA {\bf systematically outperformed the MLP and all the previously
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
668 published results on this dataset} (the ones that we are aware of), {\bf in fact reaching human-level
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
669 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
670
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
671 $\bullet$ %\item
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
672 To what extent do self-taught learning scenarios help deep learners,
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
673 and do they help them more than shallow supervised ones?
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
674 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
675 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
676 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
677 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
678 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
679 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
680 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
681 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
682 with respect to clean examples . On the other hand, the deep SDAs
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
683 were very significantly boosted by these out-of-distribution examples.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
684 Similarly, whereas the improvement due to the multi-task setting was marginal or
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
685 negative for the MLP (from +5.6\% to -3.6\% relative change),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
686 it was very significant for the SDA (from +13\% to +27\% relative change).
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
687 %\end{itemize}
472
2dd6e8962df1 conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 469
diff changeset
688
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
689 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
690 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
691 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
692 scenario. However, many of the results by \citet{RainaR2007} (who used a
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
693 shallow, sparse coding approach) suggest that the relative gain of self-taught
536
5157a5830125 One comma
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 531
diff changeset
694 learning diminishes as the number of labeled examples increases (essentially,
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
695 a ``diminishing returns'' scenario occurs). We note instead that, for deep
524
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
696 architectures, our experiments show that such a positive effect is accomplished
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
697 even in a scenario with a \emph{very large number of labeled examples}.
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents: 523
diff changeset
698
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
699 Why would deep learners benefit more from the self-taught learning framework?
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
700 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
701 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
702 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
703 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
704 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
705 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
706 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
707 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
708 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
709 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
710 experiments could attempt at teasing apart these factors.
529
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
711 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
712 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
713 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
714 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
715 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
716 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
717 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
718 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
719 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
720 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
721 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
722 (with or without self-taught settings), and more labeled examples
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
723 does not allow to go from the poorer basins of attraction discovered
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
724 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 524
diff changeset
725 with deep learning and self-taught learning.
502
2b35a6e5ece4 changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 501
diff changeset
726
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
727 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
728 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
729
498
7ff00c27c976 add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents: 496
diff changeset
730 \newpage
496
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
731 {
e41007dd40e9 make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents: 495
diff changeset
732 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
733 %\bibliographystyle{plainnat}
d02d288257bf redone bib style
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 467
diff changeset
734 \bibliographystyle{unsrtnat}
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
735 %\bibliographystyle{apalike}
484
9a757d565e46 reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 483
diff changeset
736 }
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
737
485
6beaf3328521 les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 484
diff changeset
738
464
24f4a8b53fcc nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
739 \end{document}