Mercurial > ift6266
annotate writeup/nips2010_submission.tex @ 512:6f042a71be23
todo done
| author | Yoshua Bengio <bengioy@iro.umontreal.ca> |
|---|---|
| date | Tue, 01 Jun 2010 14:02:04 -0400 |
| parents | b8e33d3d7f65 |
| children | 66a905508e34 |
| rev | line source |
|---|---|
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
1 \documentclass{article} % For LaTeX2e |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
2 \usepackage{nips10submit_e,times} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
3 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
4 \usepackage{amsthm,amsmath,amssymb,bbold,bbm} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
5 \usepackage{algorithm,algorithmic} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
6 \usepackage[utf8]{inputenc} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
7 \usepackage{graphicx,subfigure} |
| 469 | 8 \usepackage[numbers]{natbib} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
9 |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
10 \title{Deep Self-Taught Learning for Handwritten Character Recognition} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
11 \author{The IFT6266 Gang} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
12 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
13 \begin{document} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
14 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
15 %\makeanontitle |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
16 \maketitle |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
17 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
18 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
19 \begin{abstract} |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
20 Recent theoretical and empirical work in statistical machine learning has |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
21 demonstrated the importance of learning algorithms for deep |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
22 architectures, i.e., function classes obtained by composing multiple |
|
504
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
23 non-linear transformations. Self-taught learning (exploiting unlabeled |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
24 examples or examples from other distributions) has already been applied |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
25 to deep learners, but mostly to show the advantage of unlabeled |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
26 examples. Here we explore the advantage brought by {\em out-of-distribution |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
27 examples} and show that {\em deep learners benefit more from them than a |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
28 corresponding shallow learner}, in the area |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
29 of handwritten character recognition. In fact, we show that they reach |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
30 human-level performance on both handwritten digit classification and |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
31 62-class handwritten character recognition. For this purpose we |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
32 developed a powerful generator of stochastic variations and noise |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
33 processes character images, including not only affine transformations but |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
34 also slant, local elastic deformations, changes in thickness, background |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
35 images, grey level changes, contrast, occlusion, and various types of pixel and |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
36 spatially correlated noise. The out-of-distribution examples are |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
37 obtained by training with these highly distorted images or |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
38 by including object classes different from those in the target test set. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
39 \end{abstract} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
40 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
41 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
42 \section{Introduction} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
43 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
44 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
45 Deep Learning has emerged as a promising new area of research in |
| 469 | 46 statistical machine learning (see~\citet{Bengio-2009} for a review). |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
47 Learning algorithms for deep architectures are centered on the learning |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
48 of useful representations of data, which are better suited to the task at hand. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
49 This is in great part inspired by observations of the mammalian visual cortex, |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
50 which consists of a chain of processing elements, each of which is associated with a |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
51 different representation of the raw visual input. In fact, |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
52 it was found recently that the features learnt in deep architectures resemble |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
53 those observed in the first two of these stages (in areas V1 and V2 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
54 of visual cortex)~\citep{HonglakL2008}, and that they become more and |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
55 more invariant to factors of variation (such as camera movement) in |
| 501 | 56 higher layers~\citep{Goodfellow2009}. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
57 Learning a hierarchy of features increases the |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
58 ease and practicality of developing representations that are at once |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
59 tailored to specific tasks, yet are able to borrow statistical strength |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
60 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
61 feature representation can lead to higher-level (more abstract, more |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
62 general) features that are more robust to unanticipated sources of |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
63 variance extant in real data. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
64 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
65 Whereas a deep architecture can in principle be more powerful than a |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
66 shallow one in terms of representation, depth appears to render the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
67 training problem more difficult in terms of optimization and local minima. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
68 It is also only recently that successful algorithms were proposed to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
69 overcome some of these difficulties. All are based on unsupervised |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
70 learning, often in an greedy layer-wise ``unsupervised pre-training'' |
| 469 | 71 stage~\citep{Bengio-2009}. One of these layer initialization techniques, |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
72 applied here, is the Denoising |
| 469 | 73 Auto-Encoder~(DEA)~\citep{VincentPLarochelleH2008-very-small}, which |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
74 performed similarly or better than previously proposed Restricted Boltzmann |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
75 Machines in terms of unsupervised extraction of a hierarchy of features |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
76 useful for classification. The principle is that each layer starting from |
|
504
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
77 the bottom is trained to encode its input (the output of the previous |
|
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
78 layer) and to reconstruct it from a corrupted version of it. After this |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
79 unsupervised initialization, the stack of denoising auto-encoders can be |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
80 converted into a deep supervised feedforward neural network and fine-tuned by |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
81 stochastic gradient descent. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
82 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
83 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
84 of semi-supervised and multi-task learning: the learner can exploit examples |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
85 that are unlabeled and/or come from a distribution different from the target |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
86 distribution, e.g., from other classes that those of interest. Whereas |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
87 it has already been shown that deep learners can clearly take advantage of |
|
496
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
88 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
89 and multi-task learning, not much has been done yet to explore the impact |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
90 of {\em out-of-distribution} examples and of the multi-task setting |
|
496
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
91 (but see~\citep{CollobertR2008}). In particular the {\em relative |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
92 advantage} of deep learning for this settings has not been evaluated. |
| 512 | 93 The hypothesis explored here is that a deep hierarchy of features |
| 94 may be better able to provide sharing of statistical strength | |
| 95 between different regions in input space or different tasks, | |
| 96 as discussed in the conclusion. | |
|
504
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
97 |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
98 In this paper we ask the following questions: |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
99 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
100 %\begin{enumerate} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
101 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
102 Do the good results previously obtained with deep architectures on the |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
103 MNIST digit images generalize to the setting of a much larger and richer (but similar) |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
104 dataset, the NIST special database 19, with 62 classes and around 800k examples? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
105 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
106 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
107 To what extent does the perturbation of input images (e.g. adding |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
108 noise, affine transformations, background images) make the resulting |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
109 classifiers better not only on similarly perturbed images but also on |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
110 the {\em original clean examples}? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
111 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
112 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
113 Do deep architectures {\em benefit more from such out-of-distribution} |
| 469 | 114 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
115 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
116 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
117 Similarly, does the feature learning step in deep learning algorithms benefit more |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
118 training with similar but different classes (i.e. a multi-task learning scenario) than |
|
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
119 a corresponding shallow and purely supervised architecture? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
120 %\end{enumerate} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
121 |
|
504
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
122 Our experimental results provide evidence to support positive answers to all of these questions. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
123 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
124 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
125 \section{Perturbation and Transformation of Character Images} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
126 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
127 |
| 467 | 128 This section describes the different transformations we used to stochastically |
| 129 transform source images in order to obtain data. More details can | |
| 469 | 130 be found in this technical report~\citep{ift6266-tr-anonymous}. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
131 The code for these transformations (mostly python) is available at |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
132 {\tt http://anonymous.url.net}. All the modules in the pipeline share |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
133 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the |
| 467 | 134 amount of deformation or noise introduced. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
135 |
| 467 | 136 There are two main parts in the pipeline. The first one, |
| 137 from slant to pinch below, performs transformations. The second | |
| 138 part, from blur to contrast, adds different kinds of noise. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
139 |
| 501 | 140 \begin{figure}[h] |
| 141 \resizebox{.99\textwidth}{!}{\includegraphics{images/transfo.png}}\\ | |
| 506 | 142 % TODO: METTRE LE NOM DE LA TRANSFO A COTE DE CHAQUE IMAGE |
| 501 | 143 \caption{Illustration of each transformation applied alone to the same image |
| 144 of an upper-case h (top left). First row (from left to right) : original image, slant, | |
| 145 thickness, affine transformation (translation, rotation, shear), | |
| 146 local elastic deformation; second row (from left to right) : | |
| 147 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to right) : | |
| 148 background image, salt and pepper noise, spatially Gaussian noise, scratches, | |
| 149 grey level and contrast changes.} | |
| 150 \label{fig:transfo} | |
| 151 \end{figure} | |
| 152 | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
153 {\large\bf Transformations} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
154 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
155 \vspace*{2mm} |
|
483
b9cdb464de5f
pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
482
diff
changeset
|
156 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
157 {\bf Slant.} |
| 467 | 158 We mimic slant by shifting each row of the image |
| 495 | 159 proportionally to its height: $shift = round(slant \times height)$. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
160 The $slant$ coefficient can be negative or positive with equal probability |
| 467 | 161 and its value is randomly sampled according to the complexity level: |
| 162 e $slant \sim U[0,complexity]$, so the | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
163 maximum displacement for the lowest or highest pixel line is of |
| 467 | 164 $round(complexity \times 32)$.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
165 {\bf Thickness.} |
| 495 | 166 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82} |
| 467 | 167 are applied. The neighborhood of each pixel is multiplied |
| 168 element-wise with a {\em structuring element} matrix. | |
| 169 The pixel value is replaced by the maximum or the minimum of the resulting | |
| 170 matrix, respectively for dilation or erosion. Ten different structural elements with | |
| 171 increasing dimensions (largest is $5\times5$) were used. For each image, | |
| 172 randomly sample the operator type (dilation or erosion) with equal probability and one structural | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
173 element from a subset of the $n$ smallest structuring elements where $n$ is |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
174 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$ |
| 467 | 175 for erosion. A neutral element is always present in the set, and if it is |
| 176 chosen no transformation is applied. Erosion allows only the six | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
177 smallest structural elements because when the character is too thin it may |
| 467 | 178 be completely erased.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
179 {\bf Affine Transformations.} |
| 467 | 180 A $2 \times 3$ affine transform matrix (with |
| 181 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level. | |
| 182 Each pixel $(x,y)$ of the output image takes the value of the pixel | |
| 183 nearest to $(ax+by+c,dx+ey+f)$ in the input image. This | |
| 184 produces scaling, translation, rotation and shearing. | |
| 185 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
186 forbid important rotations (not to confuse classes) but to give good |
| 467 | 187 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times |
| 188 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3 | |
| 189 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times | |
| 190 complexity]$.\\ | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
191 {\bf Local Elastic Deformations.} |
|
496
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
192 This filter induces a "wiggly" effect in the image, following~\citet{SimardSP03-short}, |
| 467 | 193 which provides more details. |
| 194 Two "displacements" fields are generated and applied, for horizontal | |
| 195 and vertical displacements of pixels. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
196 To generate a pixel in either field, first a value between -1 and 1 is |
| 467 | 197 chosen from a uniform distribution. Then all the pixels, in both fields, are |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
198 multiplied by a constant $\alpha$ which controls the intensity of the |
| 467 | 199 displacements (larger $\alpha$ translates into larger wiggles). |
| 200 Each field is convoluted with a Gaussian 2D kernel of | |
| 201 standard deviation $\sigma$. Visually, this results in a blur. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
202 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times |
| 467 | 203 \sqrt[3]{complexity}$.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
204 {\bf Pinch.} |
| 467 | 205 This GIMP filter is named "Whirl and |
| 206 pinch", but whirl was set to 0. A pinch is ``similar to projecting the image onto an elastic | |
| 503 | 207 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual). |
| 467 | 208 For a square input image, think of drawing a circle of |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
209 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
210 that disk (region inside circle) will have its value recalculated by taking |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
211 the value of another "source" pixel in the original image. The position of |
| 495 | 212 that source pixel is found on the line that goes through $C$ and $P$, but |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
213 at some other distance $d_2$. Define $d_1$ to be the distance between $P$ |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
214 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
215 d_1$, where $pinch$ is a parameter to the filter. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
216 The actual value is given by bilinear interpolation considering the pixels |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
217 around the (non-integer) source position thus found. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
218 Here $pinch \sim U[-complexity, 0.7 \times complexity]$. |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
219 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
220 \vspace*{1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
221 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
222 {\large\bf Injecting Noise} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
223 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
224 \vspace*{1mm} |
|
483
b9cdb464de5f
pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
482
diff
changeset
|
225 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
226 {\bf Motion Blur.} |
| 467 | 227 This GIMP filter is a ``linear motion blur'' in GIMP |
| 228 terminology, with two parameters, $length$ and $angle$. The value of | |
| 229 a pixel in the final image is the approximately mean value of the $length$ first pixels | |
| 230 found by moving in the $angle$ direction. | |
| 231 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.\\ | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
232 {\bf Occlusion.} |
| 467 | 233 This filter selects a random rectangle from an {\em occluder} character |
| 234 images and places it over the original {\em occluded} character | |
| 235 image. Pixels are combined by taking the max(occluder,occluded), | |
| 236 closer to black. The corners of the occluder The rectangle corners | |
| 237 are sampled so that larger complexity gives larger rectangles. | |
| 238 The destination position in the occluded image are also sampled | |
| 469 | 239 according to a normal distribution (see more details in~\citet{ift6266-tr-anonymous}). |
| 467 | 240 It has has a probability of not being applied at all of 60\%.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
241 {\bf Pixel Permutation.} |
| 467 | 242 This filter permutes neighbouring pixels. It selects first |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
243 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
244 sequentially exchanged to one other pixel in its $V4$ neighbourhood. Number |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
245 of exchanges to the left, right, top, bottom are equal or does not differ |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
246 from more than 1 if the number of selected pixels is not a multiple of 4. |
| 467 | 247 It has has a probability of not being applied at all of 80\%.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
248 {\bf Gaussian Noise.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
249 This filter simply adds, to each pixel of the image independently, a |
| 467 | 250 noise $\sim Normal(0(\frac{complexity}{10})^2)$. |
| 251 It has has a probability of not being applied at all of 70\%.\\ | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
252 {\bf Background Images.} |
| 469 | 253 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
254 background behind the letter. The background is chosen by first selecting, |
| 495 | 255 at random, an image from a set of images. Then a 32$\times$32 sub-region |
| 467 | 256 of that image is chosen as the background image (by sampling position |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
257 uniformly while making sure not to cross image borders). |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
258 To combine the original letter image and the background image, contrast |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
259 adjustments are made. We first get the maximal values (i.e. maximal |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
260 intensity) for both the original image and the background image, $maximage$ |
| 467 | 261 and $maxbg$. We also have a parameter $contrast \sim U[complexity, 1]$. |
| 262 Each background pixel value is multiplied by $\frac{max(maximage - | |
| 263 contrast, 0)}{maxbg}$ (higher contrast yield darker | |
| 264 background). The output image pixels are max(background,original).\\ | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
265 {\bf Salt and Pepper Noise.} |
| 467 | 266 This filter adds noise $\sim U[0,1]$ to random subsets of pixels. |
| 267 The number of selected pixels is $0.2 \times complexity$. | |
| 268 This filter has a probability of not being applied at all of 75\%.\\ | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
269 {\bf Spatially Gaussian Noise.} |
| 467 | 270 Different regions of the image are spatially smoothed. |
| 271 The image is convolved with a symmetric Gaussian kernel of | |
| 495 | 272 size and variance chosen uniformly in the ranges $[12,12 + 20 \times |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
273 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
274 between $0$ and $1$. We also create a symmetric averaging window, of the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
275 kernel size, with maximum value at the center. For each image we sample |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
276 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
277 averaging centers between the original image and the filtered one. We |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
278 initialize to zero a mask matrix of the image size. For each selected pixel |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
279 we add to the mask the averaging window centered to it. The final image is |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
280 computed from the following element-wise operation: $\frac{image + filtered |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
281 image \times mask}{mask+1}$. |
| 467 | 282 This filter has a probability of not being applied at all of 75\%.\\ |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
283 {\bf Scratches.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
284 The scratches module places line-like white patches on the image. The |
| 467 | 285 lines are heavily transformed images of the digit "1" (one), chosen |
| 286 at random among five thousands such 1 images. The 1 image is | |
| 287 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times | |
| 495 | 288 complexity)^2$, using bi-cubic interpolation, |
| 289 Two passes of a grey-scale morphological erosion filter | |
| 467 | 290 are applied, reducing the width of the line |
| 291 by an amount controlled by $complexity$. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
292 This filter is only applied only 15\% of the time. When it is applied, 50\% |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
293 of the time, only one patch image is generated and applied. In 30\% of |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
294 cases, two patches are generated, and otherwise three patches are |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
295 generated. The patch is applied by taking the maximal value on any given |
| 467 | 296 patch or the original image, for each of the 32x32 pixel locations.\\ |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
297 {\bf Grey Level and Contrast Changes.} |
| 495 | 298 This filter changes the contrast and may invert the image polarity (white |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
299 on black to black on white). The contrast $C$ is defined here as the |
| 467 | 300 difference between the maximum and the minimum pixel value of the image. |
| 495 | 301 Contrast $\sim U[1-0.85 \times complexity,1]$ (so contrast $\geq 0.15$). |
| 467 | 302 The image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
303 polarity is inverted with $0.5$ probability. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
304 |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
305 \iffalse |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
306 \begin{figure}[h] |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
307 \resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}\\ |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
308 \caption{Illustration of the pipeline of stochastic |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
309 transformations applied to the image of a lower-case t |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
310 (the upper left image). Each image in the pipeline (going from |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
311 left to right, first top line, then bottom line) shows the result |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
312 of applying one of the modules in the pipeline. The last image |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
313 (bottom right) is used as training example.} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
314 \label{fig:pipeline} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
315 \end{figure} |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
316 \fi |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
317 |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
318 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
319 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
320 \section{Experimental Setup} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
321 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
322 |
| 472 | 323 Whereas much previous work on deep learning algorithms had been performed on |
| 324 the MNIST digits classification task~\citep{Hinton06,ranzato-07,Bengio-nips-2006,Salakhutdinov+Hinton-2009}, | |
| 325 with 60~000 examples, and variants involving 10~000 | |
| 501 | 326 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want |
| 472 | 327 to focus here on the case of much larger training sets, from 10 times to |
| 328 to 1000 times larger. The larger datasets are obtained by first sampling from | |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
329 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas}, |
|
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
330 and {\bf OCR data} (scanned machine printed characters). Once a character |
|
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
331 is sampled from one of these sources (chosen randomly), a pipeline of |
|
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
332 the above transformations and/or noise processes is applied to the |
|
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
333 image. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
334 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
335 We compare the best MLP (according to validation set error) that we found against |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
336 the best SDA (again according to validation set error), along with a precise estimate |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
337 of human performance obtained via Amazon's Mechanical Turk (AMT) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
338 service\footnote{http://mturk.com}. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
339 AMT users are paid small amounts |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
340 of money to perform tasks for which human intelligence is required. |
| 503 | 341 Mechanical Turk has been used extensively in natural language processing and vision. |
| 342 %processing \citep{SnowEtAl2008} and vision | |
| 343 %\citep{SorokinAndForsyth2008,whitehill09}. | |
| 344 %\citep{SorokinAndForsyth2008,whitehill09}. | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
345 AMT users where presented |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
346 with 10 character images and asked to type 10 corresponding ASCII |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
347 characters. They were forced to make a hard choice among the |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
348 62 or 10 character classes (all classes or digits only). |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
349 Three users classified each image, allowing |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
350 to estimate inter-human variability. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
351 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
352 \vspace*{-1mm} |
| 472 | 353 \subsection{Data Sources} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
354 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
355 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
356 %\begin{itemize} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
357 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
358 {\bf NIST.} |
| 501 | 359 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995}, |
| 472 | 360 widely used for training and testing character |
| 501 | 361 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002,Milgram+al-2005}. |
| 362 The dataset is composed with 814255 digits and characters (upper and lower cases), with hand checked classifications, | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
363 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
364 corresponding to "0"-"9","A"-"Z" and "a"-"z". The dataset contains 8 series of different complexity. |
| 472 | 365 The fourth series, $hsf_4$, experimentally recognized to be the most difficult one is recommended |
| 501 | 366 by NIST as testing set and is used in our work and some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002,Milgram+al-2005} |
| 472 | 367 for that purpose. We randomly split the remainder into a training set and a validation set for |
|
480
150203d2b5c3
added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
479
diff
changeset
|
368 model selection. The sizes of these data sets are: 651668 for training, 80000 for validation, |
|
150203d2b5c3
added number of train test and valid for NIST
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
479
diff
changeset
|
369 and 82587 for testing. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
370 The performances reported by previous work on that dataset mostly use only the digits. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
371 Here we use all the classes both in the training and testing phase. This is especially |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
372 useful to estimate the effect of a multi-task setting. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
373 Note that the distribution of the classes in the NIST training and test sets differs |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
374 substantially, with relatively many more digits in the test set, and uniform distribution |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
375 of letters in the test set, not in the training set (more like the natural distribution |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
376 of letters in text). |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
377 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
378 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
379 {\bf Fonts.} |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
380 In order to have a good variety of sources we downloaded an important number of free fonts from: {\tt http://anonymous.url.net} |
|
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
381 %real adress {\tt http://cg.scs.carleton.ca/~luc/freefonts.html} |
|
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
382 in addition to Windows 7's, this adds up to a total of $9817$ different fonts that we can choose uniformly. |
| 495 | 383 The {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image, |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
384 directly as input to our models. |
|
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
385 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
386 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
387 {\bf Captchas.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
388 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for |
| 472 | 389 generating characters of the same format as the NIST dataset. This software is based on |
| 495 | 390 a random character class generator and various kinds of transformations similar to those described in the previous sections. |
| 472 | 391 In order to increase the variability of the data generated, many different fonts are used for generating the characters. |
| 495 | 392 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
393 depending on the value of the complexity parameter provided by the user of the data source. Two levels of complexity are |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
394 allowed and can be controlled via an easy to use facade class. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
395 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
396 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
397 {\bf OCR data.} |
| 472 | 398 A large set (2 million) of scanned, OCRed and manually verified machine-printed |
| 399 characters (from various documents and books) where included as an | |
| 400 additional source. This set is part of a larger corpus being collected by the Image Understanding | |
| 401 Pattern Recognition Research group lead by Thomas Breuel at University of Kaiserslautern | |
| 495 | 402 ({\tt http://www.iupr.com}), and which will be publicly released. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
403 %\end{itemize} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
404 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
405 \vspace*{-1mm} |
| 472 | 406 \subsection{Data Sets} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
407 \vspace*{-1mm} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
408 |
| 472 | 409 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label |
| 410 from one of the 62 character classes. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
411 %\begin{itemize} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
412 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
413 %\item |
| 501 | 414 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
415 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
416 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
417 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources |
| 472 | 418 and sending them through the above transformation pipeline. |
| 495 | 419 For each new example to generate, a source is selected with probability $10\%$ from the fonts, |
| 472 | 420 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the |
| 421 order given above, and for each of them we sample uniformly a complexity in the range $[0,0.7]$. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
422 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
423 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
424 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same sources proportion) |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
425 except that we only apply |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
426 transformations from slant to pinch. Therefore, the character is |
| 495 | 427 transformed but no additional noise is added to the image, giving images |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
428 closer to the NIST dataset. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
429 %\end{itemize} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
430 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
431 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
432 \subsection{Models and their Hyperparameters} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
433 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
434 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
435 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
436 hidden layer and with Stacked Denoising Auto-Encoders (SDA). |
| 472 | 437 All hyper-parameters are selected based on performance on the NISTP validation set. |
| 438 | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
439 {\bf Multi-Layer Perceptrons (MLP).} |
| 472 | 440 Whereas previous work had compared deep architectures to both shallow MLPs and |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
441 SVMs, we only compared to MLPs here because of the very large datasets used |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
442 (making the use of SVMs computationally inconvenient because of their quadratic |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
443 scaling behavior). |
| 472 | 444 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized |
| 445 exponentials) on the output layer for estimating P(class | image). | |
| 446 The hyper-parameters are the following: number of hidden units, taken in | |
| 447 $\{300,500,800,1000,1500\}$. The optimization procedure is as follows. Training | |
| 448 examples are presented in minibatches of size 20. A constant learning | |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
449 rate is chosen in $10^{-3},0.01, 0.025, 0.075, 0.1, 0.5\}$ |
| 472 | 450 through preliminary experiments, and 0.1 was selected. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
451 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
452 {\bf Stacked Denoising Auto-Encoders (SDA).} |
| 472 | 453 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs) |
| 454 can be used to initialize the weights of each layer of a deep MLP (with many hidden | |
| 455 layers)~\citep{Hinton06,ranzato-07,Bengio-nips-2006} | |
| 456 enabling better generalization, apparently setting parameters in the | |
| 457 basin of attraction of supervised gradient descent yielding better | |
| 458 generalization~\citep{Erhan+al-2010}. It is hypothesized that the | |
| 459 advantage brought by this procedure stems from a better prior, | |
| 460 on the one hand taking advantage of the link between the input | |
| 461 distribution $P(x)$ and the conditional distribution of interest | |
| 462 $P(y|x)$ (like in semi-supervised learning), and on the other hand | |
| 463 taking advantage of the expressive power and bias implicit in the | |
| 464 deep architecture (whereby complex concepts are expressed as | |
| 465 compositions of simpler ones through a deep hierarchy). | |
| 466 Here we chose to use the Denoising | |
| 467 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
468 % AJOUTER UNE IMAGE? |
| 472 | 469 these deep hierarchies of features, as it is very simple to train and |
| 470 teach (see tutorial and code there: {\tt http://deeplearning.net/tutorial}), | |
| 471 provides immediate and efficient inference, and yielded results | |
| 472 comparable or better than RBMs in series of experiments | |
| 473 \citep{VincentPLarochelleH2008}. During training of a Denoising | |
| 474 Auto-Encoder, it is presented with a stochastically corrupted version | |
| 475 of the input and trained to reconstruct the uncorrupted input, | |
| 476 forcing the hidden units to represent the leading regularities in | |
| 477 the data. Once it is trained, its hidden units activations can | |
| 478 be used as inputs for training a second one, etc. | |
| 479 After this unsupervised pre-training stage, the parameters | |
| 480 are used to initialize a deep MLP, which is fine-tuned by | |
| 481 the same standard procedure used to train them (see previous section). | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
482 The SDA hyper-parameters are the same as for the MLP, with the addition of the |
| 472 | 483 amount of corruption noise (we used the masking noise process, whereby a |
| 484 fixed proportion of the input values, randomly selected, are zeroed), and a | |
| 485 separate learning rate for the unsupervised pre-training stage (selected | |
| 486 from the same above set). The fraction of inputs corrupted was selected | |
| 487 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number | |
| 488 of hidden layers but it was fixed to 3 based on previous work with | |
| 489 stacked denoising auto-encoders on MNIST~\citep{VincentPLarochelleH2008}. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
490 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
491 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
492 \section{Experimental Results} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
493 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
494 %\vspace*{-1mm} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
495 %\subsection{SDA vs MLP vs Humans} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
496 %\vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
497 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
498 Figure~\ref{fig:error-rates-charts} summarizes the results obtained, |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
499 comparing Humans, three MLPs (MLP0, MLP1, MLP2) and three SDAs (SDA0, SDA1, |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
500 SDA2), along with the previous results on the digits NIST special database |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
501 19 test set from the literature respectively based on ARTMAP neural |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
502 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
503 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002}, and SVMs |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
504 ~\citep{Milgram+al-2005}. More detailed and complete numerical results |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
505 (figures and tables, including standard errors on the error rates) can be |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
506 found in the supplementary material. The 3 kinds of model differ in the |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
507 training sets used: NIST only (MLP0,SDA0), NISTP (MLP1, SDA1), or P07 |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
508 (MLP2, SDA2). The deep learner not only outperformed the shallow ones and |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
509 previously published performance (in a statistically and qualitatively |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
510 significant way) but reaches human performance on both the 62-class task |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
511 and the 10-class (digits) task. In addition, as shown in the left of |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
512 Figure~\ref{fig:fig:improvements-charts}, the relative improvement in error |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
513 rate brought by self-taught learning is greater for the SDA, and these |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
514 differences with the MLP are statistically and qualitatively |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
515 significant. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
516 The left side of the figure shows the improvement to the clean |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
517 NIST test set error brought by the use of out-of-distribution examples |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
518 (i.e. the perturbed examples examples from NISTP or P07). |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
519 Relative change is measured by taking |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
520 (original model's error / perturbed-data model's error - 1). |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
521 The right side of |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
522 Figure~\ref{fig:fig:improvements-charts} shows the relative improvement |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
523 brought by the use of a multi-task setting, in which the same model is |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
524 trained for more classes than the target classes of interest (i.e. training |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
525 with all 62 classes when the target classes are respectively the digits, |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
526 lower-case, or upper-case characters). Again, whereas the gain from the |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
527 multi-task setting is marginal or negative for the MLP, it is substantial |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
528 for the SDA. Note that for these multi-task experiment, only the original |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
529 NIST dataset is used. For example, the MLP-digits bar shows the relative |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
530 improvement in MLP error rate on the NIST digits test set (1 - single-task |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
531 model's error / multi-task model's error). The single-task model is |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
532 trained with only 10 outputs (one per digit), seeing only digit examples, |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
533 whereas the multi-task model is trained with 62 outputs, with all 62 |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
534 character classes as examples. Hence the hidden units are shared across |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
535 all tasks. For the multi-task model, the digit error rate is measured by |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
536 comparing the correct digit class with the output class associated with the |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
537 maximum conditional probability among only the digit classes outputs. The |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
538 setting is similar for the other two target classes (lower case characters |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
539 and upper case characters). |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
540 |
| 475 | 541 \begin{figure}[h] |
| 542 \resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}\\ | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
543 \caption{Error bars indicate a 95\% confidence interval. 0 indicates training |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
544 on NIST, 1 on NISTP, and 2 on P07. Left: overall results |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
545 of all models, on 3 different test sets corresponding to the three |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
546 datasets. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
547 Right: error rates on NIST test digits only, along with the previous results from |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
548 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002,Milgram+al-2005} |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
549 respectively based on ART, nearest neighbors, MLPs, and SVMs.} |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
550 |
| 475 | 551 \label{fig:error-rates-charts} |
| 552 \end{figure} | |
| 553 | |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
554 %\vspace*{-1mm} |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
555 %\subsection{Perturbed Training Data More Helpful for SDA} |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
556 %\vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
557 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
558 %\vspace*{-1mm} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
559 %\subsection{Multi-Task Learning Effects} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
560 %\vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
561 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
562 \iffalse |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
563 As previously seen, the SDA is better able to benefit from the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
564 transformations applied to the data than the MLP. In this experiment we |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
565 define three tasks: recognizing digits (knowing that the input is a digit), |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
566 recognizing upper case characters (knowing that the input is one), and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
567 recognizing lower case characters (knowing that the input is one). We |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
568 consider the digit classification task as the target task and we want to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
569 evaluate whether training with the other tasks can help or hurt, and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
570 whether the effect is different for MLPs versus SDAs. The goal is to find |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
571 out if deep learning can benefit more (or less) from multiple related tasks |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
572 (i.e. the multi-task setting) compared to a corresponding purely supervised |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
573 shallow learner. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
574 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
575 We use a single hidden layer MLP with 1000 hidden units, and a SDA |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
576 with 3 hidden layers (1000 hidden units per layer), pre-trained and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
577 fine-tuned on NIST. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
578 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
579 Our results show that the MLP benefits marginally from the multi-task setting |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
580 in the case of digits (5\% relative improvement) but is actually hurt in the case |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
581 of characters (respectively 3\% and 4\% worse for lower and upper class characters). |
| 495 | 582 On the other hand the SDA benefited from the multi-task setting, with relative |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
583 error rate improvements of 27\%, 15\% and 13\% respectively for digits, |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
584 lower and upper case characters, as shown in Table~\ref{tab:multi-task}. |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
585 \fi |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
586 |
| 475 | 587 |
| 588 \begin{figure}[h] | |
| 589 \resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}\\ | |
|
504
e837ef6eef8c
commit early, commit often: a couple of changes to kick-start things
dumitru@dumitru.mtv.corp.google.com
parents:
501
diff
changeset
|
590 \caption{Charts corresponding to tables 2 (left) and 3 (right), from Appendix I.} |
| 475 | 591 \label{fig:improvements-charts} |
| 592 \end{figure} | |
| 593 | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
594 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
595 \section{Conclusions} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
596 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
597 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
598 We have found that the self-taught learning framework is more beneficial |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
599 to a deep learner than to a traditional shallow and purely |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
600 supervised learner. More precisely, |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
601 the conclusions are positive for all the questions asked in the introduction. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
602 %\begin{itemize} |
| 487 | 603 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
604 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
605 Do the good results previously obtained with deep architectures on the |
| 472 | 606 MNIST digits generalize to the setting of a much larger and richer (but similar) |
| 607 dataset, the NIST special database 19, with 62 classes and around 800k examples? | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
608 Yes, the SDA {\bf systematically outperformed the MLP and all the previously |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
609 published results on this dataset (as far as we know), in fact reaching human-level |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
610 performance} at round 17\% error on the 62-class task and 1.4\% on the digits. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
611 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
612 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
613 To what extent does the perturbation of input images (e.g. adding |
| 472 | 614 noise, affine transformations, background images) make the resulting |
| 615 classifier better not only on similarly perturbed images but also on | |
| 616 the {\em original clean examples}? Do deep architectures benefit more from such {\em out-of-distribution} | |
| 617 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework? | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
618 MLPs were helped by perturbed training examples when tested on perturbed input |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
619 images (65\% relative improvement on NISTP) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
620 but only marginally helped (5\% relative improvement on all classes) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
621 or even hurt (10\% relative loss on digits) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
622 with respect to clean examples . On the other hand, the deep SDAs |
| 472 | 623 were very significantly boosted by these out-of-distribution examples. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
624 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
625 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
626 Similarly, does the feature learning step in deep learning algorithms benefit more |
| 472 | 627 training with similar but different classes (i.e. a multi-task learning scenario) than |
| 628 a corresponding shallow and purely supervised architecture? | |
| 629 Whereas the improvement due to the multi-task setting was marginal or | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
630 negative for the MLP (from +5.6\% to -3.6\% relative change), |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
631 it was very significant for the SDA (from +13\% to +27\% relative change). |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
632 %\end{itemize} |
| 472 | 633 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
634 Why would deep learners benefit more from the self-taught learning framework? |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
635 The key idea is that the lower layers of the predictor compute a hierarchy |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
636 of features that can be shared across tasks or across variants of the |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
637 input distribution. Intermediate features that can be used in different |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
638 contexts can be estimated in a way that allows to share statistical |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
639 strength. Features extracted through many levels are more likely to |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
640 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest), |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
641 increasing the likelihood that they would be useful for a larger array |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
642 of tasks and input conditions. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
643 Therefore, we hypothesize that both depth and unsupervised |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
644 pre-training play a part in explaining the advantages observed here, and future |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
645 experiments could attempt at teasing apart these factors. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
646 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
647 A Flash demo of the recognizer (where both the MLP and the SDA can be compared) |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
648 can be executed on-line at {\tt http://deep.host22.com}. |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
649 |
|
498
7ff00c27c976
add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents:
496
diff
changeset
|
650 \newpage |
|
496
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
651 { |
|
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
652 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials} |
| 469 | 653 %\bibliographystyle{plainnat} |
| 654 \bibliographystyle{unsrtnat} | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
655 %\bibliographystyle{apalike} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
656 } |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
657 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
658 |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
659 \end{document} |