Mercurial > ift6266
annotate writeup/nips2010_submission.tex @ 549:ef172f4a322a
ca fitte
| author | Yoshua Bengio <bengioy@iro.umontreal.ca> |
|---|---|
| date | Wed, 02 Jun 2010 13:56:01 -0400 |
| parents | 34cb28249de0 |
| children | 662299f265ab |
| rev | line source |
|---|---|
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
1 \documentclass{article} % For LaTeX2e |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
2 \usepackage{nips10submit_e,times} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
3 |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
4 \usepackage{amsthm,amsmath,bbm} |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
5 \usepackage[psamsfonts]{amssymb} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
6 \usepackage{algorithm,algorithmic} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
7 \usepackage[utf8]{inputenc} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
8 \usepackage{graphicx,subfigure} |
| 469 | 9 \usepackage[numbers]{natbib} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
10 |
| 523 | 11 %\setlength\parindent{0mm} |
| 12 | |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
13 \title{Deep Self-Taught Learning for Handwritten Character Recognition} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
14 \author{The IFT6266 Gang} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
15 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
16 \begin{document} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
17 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
18 %\makeanontitle |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
19 \maketitle |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
20 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
21 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
22 \begin{abstract} |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
23 Recent theoretical and empirical work in statistical machine learning has |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
24 demonstrated the importance of learning algorithms for deep |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
25 architectures, i.e., function classes obtained by composing multiple |
|
511
d057941417ed
a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
510
diff
changeset
|
26 non-linear transformations. Self-taught learning (exploiting unlabeled |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
27 examples or examples from other distributions) has already been applied |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
28 to deep learners, but mostly to show the advantage of unlabeled |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
29 examples. Here we explore the advantage brought by {\em out-of-distribution |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
30 examples} and show that {\em deep learners benefit more from them than a |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
31 corresponding shallow learner}, in the area |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
32 of handwritten character recognition. In fact, we show that they reach |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
33 human-level performance on both handwritten digit classification and |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
34 62-class handwritten character recognition. For this purpose we |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
35 developed a powerful generator of stochastic variations and noise |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
36 processes character images, including not only affine transformations but |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
37 also slant, local elastic deformations, changes in thickness, background |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
38 images, grey level changes, contrast, occlusion, and various types of pixel and |
|
482
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
39 spatially correlated noise. The out-of-distribution examples are |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
40 obtained by training with these highly distorted images or |
|
ce69aa9204d8
changement au titre et reecriture abstract
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
480
diff
changeset
|
41 by including object classes different from those in the target test set. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
42 \end{abstract} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
43 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
44 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
45 \section{Introduction} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
46 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
47 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
48 Deep Learning has emerged as a promising new area of research in |
| 469 | 49 statistical machine learning (see~\citet{Bengio-2009} for a review). |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
50 Learning algorithms for deep architectures are centered on the learning |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
51 of useful representations of data, which are better suited to the task at hand. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
52 This is in great part inspired by observations of the mammalian visual cortex, |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
53 which consists of a chain of processing elements, each of which is associated with a |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
54 different representation of the raw visual input. In fact, |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
55 it was found recently that the features learnt in deep architectures resemble |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
56 those observed in the first two of these stages (in areas V1 and V2 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
57 of visual cortex)~\citep{HonglakL2008}, and that they become more and |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
58 more invariant to factors of variation (such as camera movement) in |
| 501 | 59 higher layers~\citep{Goodfellow2009}. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
60 Learning a hierarchy of features increases the |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
61 ease and practicality of developing representations that are at once |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
62 tailored to specific tasks, yet are able to borrow statistical strength |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
63 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
64 feature representation can lead to higher-level (more abstract, more |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
65 general) features that are more robust to unanticipated sources of |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
66 variance extant in real data. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
67 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
68 Whereas a deep architecture can in principle be more powerful than a |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
69 shallow one in terms of representation, depth appears to render the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
70 training problem more difficult in terms of optimization and local minima. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
71 It is also only recently that successful algorithms were proposed to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
72 overcome some of these difficulties. All are based on unsupervised |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
73 learning, often in an greedy layer-wise ``unsupervised pre-training'' |
| 469 | 74 stage~\citep{Bengio-2009}. One of these layer initialization techniques, |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
75 applied here, is the Denoising |
| 547 | 76 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see Figure~\ref{fig:da}), |
| 77 which | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
78 performed similarly or better than previously proposed Restricted Boltzmann |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
79 Machines in terms of unsupervised extraction of a hierarchy of features |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
80 useful for classification. The principle is that each layer starting from |
|
511
d057941417ed
a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
510
diff
changeset
|
81 the bottom is trained to encode its input (the output of the previous |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
82 layer) and to reconstruct it from a corrupted version. After this |
| 547 | 83 unsupervised initialization, the stack of DAs can be |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
84 converted into a deep supervised feedforward neural network and fine-tuned by |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
85 stochastic gradient descent. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
86 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
87 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
88 of semi-supervised and multi-task learning: the learner can exploit examples |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
89 that are unlabeled and/or come from a distribution different from the target |
|
532
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
90 distribution, e.g., from other classes that those of interest. |
|
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
91 It has already been shown that deep learners can clearly take advantage of |
|
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
92 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small}, |
|
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
93 but more needs to be done to explore the impact |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
94 of {\em out-of-distribution} examples and of the multi-task setting |
|
532
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
95 (one exception is~\citep{CollobertR2008}, but using very different kinds |
|
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
96 of learning algorithms). In particular the {\em relative |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
97 advantage} of deep learning for this settings has not been evaluated. |
| 512 | 98 The hypothesis explored here is that a deep hierarchy of features |
| 99 may be better able to provide sharing of statistical strength | |
| 100 between different regions in input space or different tasks, | |
| 101 as discussed in the conclusion. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
102 |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
103 In this paper we ask the following questions: |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
104 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
105 %\begin{enumerate} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
106 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
107 Do the good results previously obtained with deep architectures on the |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
108 MNIST digit images generalize to the setting of a much larger and richer (but similar) |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
109 dataset, the NIST special database 19, with 62 classes and around 800k examples? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
110 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
111 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
112 To what extent does the perturbation of input images (e.g. adding |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
113 noise, affine transformations, background images) make the resulting |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
114 classifiers better not only on similarly perturbed images but also on |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
115 the {\em original clean examples}? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
116 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
117 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
118 Do deep architectures {\em benefit more from such out-of-distribution} |
| 469 | 119 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
120 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
121 $\bullet$ %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
122 Similarly, does the feature learning step in deep learning algorithms benefit more |
|
466
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
123 training with similar but different classes (i.e. a multi-task learning scenario) than |
|
6205481bf33f
asking the questions
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
464
diff
changeset
|
124 a corresponding shallow and purely supervised architecture? |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
125 %\end{enumerate} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
126 |
|
511
d057941417ed
a few changes in the first section
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
510
diff
changeset
|
127 Our experimental results provide positive evidence towards all of these questions. |
| 547 | 128 To achieve these results, we introduce in the next section a sophisticated system |
| 129 for stochastically transforming character images. The conclusion discusses | |
| 130 the more general question of why deep learners may benefit so much from | |
| 131 the self-taught learning framework. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
132 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
133 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
134 \section{Perturbation and Transformation of Character Images} |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
135 \label{s:perturbations} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
136 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
137 |
| 467 | 138 This section describes the different transformations we used to stochastically |
| 547 | 139 transform source images in order to obtain data from a larger distribution which |
| 140 covers a domain substantially larger than the clean characters distribution from | |
| 141 which we start. Although character transformations have been used before to | |
| 142 improve character recognizers, this effort is on a large scale both | |
| 143 in number of classes and in the complexity of the transformations, hence | |
| 144 in the complexity of the learning task. | |
| 145 More details can | |
| 469 | 146 be found in this technical report~\citep{ift6266-tr-anonymous}. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
147 The code for these transformations (mostly python) is available at |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
148 {\tt http://anonymous.url.net}. All the modules in the pipeline share |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
149 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the |
| 467 | 150 amount of deformation or noise introduced. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
151 |
| 467 | 152 There are two main parts in the pipeline. The first one, |
| 153 from slant to pinch below, performs transformations. The second | |
| 154 part, from blur to contrast, adds different kinds of noise. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
155 |
| 523 | 156 \begin{figure}[ht] |
| 541 | 157 \vspace*{-2mm} |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
158 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/transfo.png}}} |
| 506 | 159 % TODO: METTRE LE NOM DE LA TRANSFO A COTE DE CHAQUE IMAGE |
| 501 | 160 \caption{Illustration of each transformation applied alone to the same image |
| 161 of an upper-case h (top left). First row (from left to right) : original image, slant, | |
| 162 thickness, affine transformation (translation, rotation, shear), | |
| 163 local elastic deformation; second row (from left to right) : | |
| 164 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to right) : | |
| 165 background image, salt and pepper noise, spatially Gaussian noise, scratches, | |
| 166 grey level and contrast changes.} | |
| 167 \label{fig:transfo} | |
| 541 | 168 \vspace*{-2mm} |
| 501 | 169 \end{figure} |
| 170 | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
171 {\large\bf Transformations} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
172 |
| 541 | 173 \vspace*{0.5mm} |
|
483
b9cdb464de5f
pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
482
diff
changeset
|
174 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
175 {\bf Slant.} |
| 541 | 176 Each row of the image is shifted |
| 495 | 177 proportionally to its height: $shift = round(slant \times height)$. |
| 541 | 178 $slant \sim U[-complexity,complexity]$. |
| 179 \vspace*{-1mm} | |
| 523 | 180 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
181 {\bf Thickness.} |
| 495 | 182 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82} |
| 467 | 183 are applied. The neighborhood of each pixel is multiplied |
| 184 element-wise with a {\em structuring element} matrix. | |
| 185 The pixel value is replaced by the maximum or the minimum of the resulting | |
| 186 matrix, respectively for dilation or erosion. Ten different structural elements with | |
| 187 increasing dimensions (largest is $5\times5$) were used. For each image, | |
| 188 randomly sample the operator type (dilation or erosion) with equal probability and one structural | |
| 541 | 189 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements |
| 190 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters). | |
| 191 A neutral element (no transformation) | |
| 192 is always present in the set. is applied. | |
| 193 \vspace*{-1mm} | |
| 523 | 194 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
195 {\bf Affine Transformations.} |
| 467 | 196 A $2 \times 3$ affine transform matrix (with |
| 197 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level. | |
| 541 | 198 Output pixel $(x,y)$ takes the value of input pixel |
| 199 nearest to $(ax+by+c,dx+ey+f)$, | |
| 200 producing scaling, translation, rotation and shearing. | |
| 467 | 201 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
202 forbid important rotations (not to confuse classes) but to give good |
| 467 | 203 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times |
| 204 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3 | |
| 205 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times | |
| 523 | 206 complexity]$. |
| 541 | 207 \vspace*{-1mm} |
| 523 | 208 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
209 {\bf Local Elastic Deformations.} |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
210 This filter induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short}, |
| 467 | 211 which provides more details. |
| 541 | 212 The intensity of the displacement fields is given by |
| 213 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are | |
| 214 convolved with a Gaussian 2D kernel (resulting in a blur) of | |
| 215 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$. | |
| 216 \vspace*{-1mm} | |
| 523 | 217 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
218 {\bf Pinch.} |
| 541 | 219 This is the ``Whirl and pinch'' GIMP filter but with whirl was set to 0. |
| 220 A pinch is ``similar to projecting the image onto an elastic | |
|
521
13816dbef6ed
des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
520
diff
changeset
|
221 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual). |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
222 For a square input image, this is akin to drawing a circle of |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
223 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
224 that disk (region inside circle) will have its value recalculated by taking |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
225 the value of another ``source'' pixel in the original image. The position of |
| 495 | 226 that source pixel is found on the line that goes through $C$ and $P$, but |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
227 at some other distance $d_2$. Define $d_1$ to be the distance between $P$ |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
228 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
229 d_1$, where $pinch$ is a parameter to the filter. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
230 The actual value is given by bilinear interpolation considering the pixels |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
231 around the (non-integer) source position thus found. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
232 Here $pinch \sim U[-complexity, 0.7 \times complexity]$. |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
233 |
| 541 | 234 \vspace*{0.5mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
235 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
236 {\large\bf Injecting Noise} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
237 |
| 541 | 238 \vspace*{0.5mm} |
|
483
b9cdb464de5f
pointeur a la demo
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
482
diff
changeset
|
239 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
240 {\bf Motion Blur.} |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
241 This is GIMP's ``linear motion blur'' |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
242 with parameters $length$ and $angle$. The value of |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
243 a pixel in the final image is approximately the mean value of the $length$ first pixels |
| 467 | 244 found by moving in the $angle$ direction. |
| 523 | 245 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$. |
| 541 | 246 \vspace*{-1mm} |
| 523 | 247 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
248 {\bf Occlusion.} |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
249 Selects a random rectangle from an {\em occluder} character |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
250 image and places it over the original {\em occluded} |
| 467 | 251 image. Pixels are combined by taking the max(occluder,occluded), |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
252 closer to black. The rectangle corners |
| 467 | 253 are sampled so that larger complexity gives larger rectangles. |
| 254 The destination position in the occluded image are also sampled | |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
255 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}). |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
256 This filter is skipped with probability 60\%. |
| 541 | 257 \vspace*{-1mm} |
| 523 | 258 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
259 {\bf Pixel Permutation.} |
| 467 | 260 This filter permutes neighbouring pixels. It selects first |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
261 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
262 sequentially exchanged with one other in as $V4$ neighbourhood. |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
263 This filter is skipped with probability 80\%. |
| 541 | 264 \vspace*{-1mm} |
| 523 | 265 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
266 {\bf Gaussian Noise.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
267 This filter simply adds, to each pixel of the image independently, a |
| 467 | 268 noise $\sim Normal(0(\frac{complexity}{10})^2)$. |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
269 This filter is skipped with probability 70\%. |
| 541 | 270 \vspace*{-1mm} |
| 523 | 271 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
272 {\bf Background Images.} |
| 469 | 273 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
274 background behind the letter, from a randomly chosen natural image, |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
275 with contrast adjustments depending on $complexity$, to preserve |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
276 more or less of the original character image. |
| 541 | 277 \vspace*{-1mm} |
| 523 | 278 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
279 {\bf Salt and Pepper Noise.} |
| 467 | 280 This filter adds noise $\sim U[0,1]$ to random subsets of pixels. |
| 281 The number of selected pixels is $0.2 \times complexity$. | |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
282 This filter is skipped with probability 75\%. |
| 541 | 283 \vspace*{-1mm} |
| 523 | 284 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
285 {\bf Spatially Gaussian Noise.} |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
286 Different regions of the image are spatially smoothed by convolving |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
287 the image is convolved with a symmetric Gaussian kernel of |
| 495 | 288 size and variance chosen uniformly in the ranges $[12,12 + 20 \times |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
289 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
290 between $0$ and $1$. We also create a symmetric averaging window, of the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
291 kernel size, with maximum value at the center. For each image we sample |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
292 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
293 averaging centers between the original image and the filtered one. We |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
294 initialize to zero a mask matrix of the image size. For each selected pixel |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
295 we add to the mask the averaging window centered to it. The final image is |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
296 computed from the following element-wise operation: $\frac{image + filtered |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
297 image \times mask}{mask+1}$. |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
298 This filter is skipped with probability 75\%. |
| 541 | 299 \vspace*{-1mm} |
| 523 | 300 |
|
474
bcf024e6ab23
fits now, but still now graphics
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
472
diff
changeset
|
301 {\bf Scratches.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
302 The scratches module places line-like white patches on the image. The |
|
517
0a5945249f2b
section 2, quick first pass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
511
diff
changeset
|
303 lines are heavily transformed images of the digit ``1'' (one), chosen |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
304 at random among 500 such 1 images, |
| 467 | 305 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
306 complexity)^2$, using bi-cubic interpolation. |
| 495 | 307 Two passes of a grey-scale morphological erosion filter |
| 467 | 308 are applied, reducing the width of the line |
| 309 by an amount controlled by $complexity$. | |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
310 This filter is skipped with probability 85\%. The probabilities |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
311 of applying 1, 2, or 3 patches are (50\%,30\%,20\%). |
| 541 | 312 \vspace*{-1mm} |
| 523 | 313 |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
314 {\bf Grey Level and Contrast Changes.} |
| 495 | 315 This filter changes the contrast and may invert the image polarity (white |
|
544
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
316 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$ |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
317 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The |
|
1cdfc17e890f
ca fitte maintenant
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
541
diff
changeset
|
318 polarity is inverted with probability 50\%. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
319 |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
320 \iffalse |
| 523 | 321 \begin{figure}[ht] |
| 538 | 322 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\ |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
323 \caption{Illustration of the pipeline of stochastic |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
324 transformations applied to the image of a lower-case \emph{t} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
325 (the upper left image). Each image in the pipeline (going from |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
326 left to right, first top line, then bottom line) shows the result |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
327 of applying one of the modules in the pipeline. The last image |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
328 (bottom right) is used as training example.} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
329 \label{fig:pipeline} |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
330 \end{figure} |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
331 \fi |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
332 |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
333 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
334 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
335 \section{Experimental Setup} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
336 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
337 |
| 472 | 338 Whereas much previous work on deep learning algorithms had been performed on |
|
516
092dae9a5040
make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents:
514
diff
changeset
|
339 the MNIST digits classification task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009}, |
| 472 | 340 with 60~000 examples, and variants involving 10~000 |
| 501 | 341 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want |
| 472 | 342 to focus here on the case of much larger training sets, from 10 times to |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
343 to 1000 times larger. |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
344 |
| 535 | 345 The first step in constructing the larger datasets (called NISTP and P07) is to sample from |
|
499
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
346 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas}, |
|
2b58eda9fc08
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
495
diff
changeset
|
347 and {\bf OCR data} (scanned machine printed characters). Once a character |
| 547 | 348 is sampled from one of these sources (chosen randomly), the second step is to |
| 349 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
350 |
| 547 | 351 To provide a baseline of error rate comparison we also estimate human performance |
| 352 on both the 62-class task and the 10-class digits task. | |
| 535 | 353 We compare the best MLPs against |
| 354 the best SDAs (both models' hyper-parameters are selected to minimize the validation set error), | |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
355 along with a comparison against a precise estimate |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
356 of human performance obtained via Amazon's Mechanical Turk (AMT) |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
357 service (http://mturk.com). |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
358 AMT users are paid small amounts |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
359 of money to perform tasks for which human intelligence is required. |
|
522
d41926a68993
remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
521
diff
changeset
|
360 Mechanical Turk has been used extensively in natural language processing and vision. |
|
d41926a68993
remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
521
diff
changeset
|
361 %processing \citep{SnowEtAl2008} and vision |
|
d41926a68993
remis les choses qui avaient disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
521
diff
changeset
|
362 %\citep{SorokinAndForsyth2008,whitehill09}. |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
363 AMT users were presented |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
364 with 10 character images and asked to choose 10 corresponding ASCII |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
365 characters. They were forced to make a hard choice among the |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
366 62 or 10 character classes (all classes or digits only). |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
367 Three users classified each image, allowing |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
368 to estimate inter-human variability. A total 2500 images/dataset were classified. |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
369 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
370 \vspace*{-1mm} |
| 472 | 371 \subsection{Data Sources} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
372 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
373 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
374 %\begin{itemize} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
375 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
376 {\bf NIST.} |
| 501 | 377 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995}, |
| 472 | 378 widely used for training and testing character |
|
516
092dae9a5040
make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents:
514
diff
changeset
|
379 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}. |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
380 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications, |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
381 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
382 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity. |
| 534 | 383 The fourth partition (called $hsf_4$, 82587 examples), |
| 384 experimentally recognized to be the most difficult one, is the one recommended | |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
385 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005} |
| 534 | 386 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for |
| 387 model selection. | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
388 The performances reported by previous work on that dataset mostly use only the digits. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
389 Here we use all the classes both in the training and testing phase. This is especially |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
390 useful to estimate the effect of a multi-task setting. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
391 Note that the distribution of the classes in the NIST training and test sets differs |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
392 substantially, with relatively many more digits in the test set, and more uniform distribution |
|
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
393 of letters in the test set, compared to the training set (in the latter, the letters are distributed |
|
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
394 more like the natural distribution of letters in text). |
| 549 | 395 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
396 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
397 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
398 {\bf Fonts.} |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
399 In order to have a good variety of sources we downloaded an important number of free fonts from: |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
400 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}. |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
401 % TODO: pointless to anonymize, it's not pointing to our work |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
402 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from. |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
403 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image, |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
404 directly as input to our models. |
| 549 | 405 \vspace*{-1mm} |
|
479
6593e67381a3
Added transformation figure
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents:
476
diff
changeset
|
406 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
407 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
408 {\bf Captchas.} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
409 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for |
| 472 | 410 generating characters of the same format as the NIST dataset. This software is based on |
| 495 | 411 a random character class generator and various kinds of transformations similar to those described in the previous sections. |
| 472 | 412 In order to increase the variability of the data generated, many different fonts are used for generating the characters. |
| 495 | 413 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
414 depending on the value of the complexity parameter provided by the user of the data source. |
|
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
415 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class? |
| 549 | 416 \vspace*{-1mm} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
417 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
418 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
419 {\bf OCR data.} |
| 472 | 420 A large set (2 million) of scanned, OCRed and manually verified machine-printed |
| 421 characters (from various documents and books) where included as an | |
| 422 additional source. This set is part of a larger corpus being collected by the Image Understanding | |
| 423 Pattern Recognition Research group lead by Thomas Breuel at University of Kaiserslautern | |
| 495 | 424 ({\tt http://www.iupr.com}), and which will be publicly released. |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
425 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
426 %\end{itemize} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
427 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
428 \vspace*{-1mm} |
| 472 | 429 \subsection{Data Sets} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
430 \vspace*{-1mm} |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
431 |
| 472 | 432 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label |
| 433 from one of the 62 character classes. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
434 %\begin{itemize} |
| 549 | 435 \vspace*{-1mm} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
436 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
437 %\item |
| 534 | 438 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has |
| 535 | 439 \{651668 / 80000 / 82587\} \{training / validation / test\} examples. |
| 549 | 440 \vspace*{-1mm} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
441 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
442 %\item |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
443 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
444 and sending them through the transformation pipeline described in section \ref{s:perturbations}. |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
445 For each new example to generate, a data source is selected with probability $10\%$ from the fonts, |
| 472 | 446 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
447 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$. |
| 535 | 448 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples. |
| 549 | 449 \vspace*{-1mm} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
450 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
451 %\item |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
452 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources) |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
453 except that we only apply |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
454 transformations from slant to pinch. Therefore, the character is |
| 495 | 455 transformed but no additional noise is added to the image, giving images |
| 534 | 456 closer to the NIST dataset. |
| 535 | 457 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
458 %\end{itemize} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
459 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
460 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
461 \subsection{Models and their Hyperparameters} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
462 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
463 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
464 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
465 hidden layer and with Stacked Denoising Auto-Encoders (SDA). |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
466 \emph{Note that all hyper-parameters are selected based on performance on the {\bf NISTP} validation set.} |
| 472 | 467 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
468 {\bf Multi-Layer Perceptrons (MLP).} |
| 472 | 469 Whereas previous work had compared deep architectures to both shallow MLPs and |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
470 SVMs, we only compared to MLPs here because of the very large datasets used |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
471 (making the use of SVMs computationally challenging because of their quadratic |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
472 scaling behavior). |
| 472 | 473 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized |
|
520
18a6379999fd
more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
519
diff
changeset
|
474 exponentials) on the output layer for estimating $P(class | image)$. |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
475 The number of hidden units is taken in $\{300,500,800,1000,1500\}$. |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
476 Training examples are presented in minibatches of size 20. A constant learning |
|
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
477 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$ |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
478 through preliminary experiments (measuring performance on a validation set), |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
479 and $0.1$ was then selected for optimizing on the whole training sets. |
| 549 | 480 \vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
481 |
|
521
13816dbef6ed
des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
520
diff
changeset
|
482 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
483 {\bf Stacked Denoising Auto-Encoders (SDA).} |
| 472 | 484 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs) |
| 485 can be used to initialize the weights of each layer of a deep MLP (with many hidden | |
|
520
18a6379999fd
more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
519
diff
changeset
|
486 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006}, |
|
18a6379999fd
more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
519
diff
changeset
|
487 apparently setting parameters in the |
| 472 | 488 basin of attraction of supervised gradient descent yielding better |
| 489 generalization~\citep{Erhan+al-2010}. It is hypothesized that the | |
| 490 advantage brought by this procedure stems from a better prior, | |
| 491 on the one hand taking advantage of the link between the input | |
| 492 distribution $P(x)$ and the conditional distribution of interest | |
| 493 $P(y|x)$ (like in semi-supervised learning), and on the other hand | |
| 494 taking advantage of the expressive power and bias implicit in the | |
| 495 deep architecture (whereby complex concepts are expressed as | |
| 496 compositions of simpler ones through a deep hierarchy). | |
|
530
8fe77eac344f
Clarifying the experimental setup, typos here and there
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
524
diff
changeset
|
497 |
| 547 | 498 \begin{figure}[ht] |
| 499 \vspace*{-2mm} | |
| 500 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}} | |
| 549 | 501 \vspace*{-2mm} |
| 547 | 502 \caption{Illustration of the computations and training criterion for the denoising |
| 503 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of | |
| 504 the layer (i.e. raw input or output of previous layer) | |
| 505 is corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$. | |
| 506 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which | |
| 507 is compared to the uncorrupted input $x$ through the loss function | |
| 508 $L_H(x,z)$, whose expected value is approximately minimized during training | |
| 509 by tuning $\theta$ and $\theta'$.} | |
| 510 \label{fig:da} | |
| 511 \vspace*{-2mm} | |
| 512 \end{figure} | |
| 513 | |
| 472 | 514 Here we chose to use the Denoising |
| 515 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for | |
| 516 these deep hierarchies of features, as it is very simple to train and | |
|
532
2e33885730cf
changements aux charts.ods
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
529
diff
changeset
|
517 explain (see Figure~\ref{fig:da}, as well as |
|
521
13816dbef6ed
des choses ont disparu
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
520
diff
changeset
|
518 tutorial and code there: {\tt http://deeplearning.net/tutorial}), |
| 472 | 519 provides immediate and efficient inference, and yielded results |
| 520 comparable or better than RBMs in series of experiments | |
|
519
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
521 \citep{VincentPLarochelleH2008}. During training, a Denoising |
|
eaa595ea2402
section 3 quickpass
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
518
diff
changeset
|
522 Auto-Encoder is presented with a stochastically corrupted version |
| 472 | 523 of the input and trained to reconstruct the uncorrupted input, |
| 524 forcing the hidden units to represent the leading regularities in | |
| 535 | 525 the data. Once it is trained, in a purely unsupervised way, |
| 526 its hidden units activations can | |
| 472 | 527 be used as inputs for training a second one, etc. |
| 528 After this unsupervised pre-training stage, the parameters | |
| 529 are used to initialize a deep MLP, which is fine-tuned by | |
| 530 the same standard procedure used to train them (see previous section). | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
531 The SDA hyper-parameters are the same as for the MLP, with the addition of the |
| 472 | 532 amount of corruption noise (we used the masking noise process, whereby a |
| 533 fixed proportion of the input values, randomly selected, are zeroed), and a | |
| 534 separate learning rate for the unsupervised pre-training stage (selected | |
| 535 from the same above set). The fraction of inputs corrupted was selected | |
| 536 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number | |
| 537 of hidden layers but it was fixed to 3 based on previous work with | |
| 547 | 538 SDAs on MNIST~\citep{VincentPLarochelleH2008}. |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
539 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
540 \vspace*{-1mm} |
| 523 | 541 |
| 542 \begin{figure}[ht] | |
| 541 | 543 \vspace*{-2mm} |
| 523 | 544 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}} |
| 549 | 545 \vspace*{-3mm} |
| 547 | 546 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained |
| 523 | 547 on NIST, 1 on NISTP, and 2 on P07. Left: overall results |
|
548
34cb28249de0
suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
547
diff
changeset
|
548 of all models, on NIST and NISTP test sets. |
| 523 | 549 Right: error rates on NIST test digits only, along with the previous results from |
| 550 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005} | |
| 551 respectively based on ART, nearest neighbors, MLPs, and SVMs.} | |
| 552 \label{fig:error-rates-charts} | |
| 541 | 553 \vspace*{-2mm} |
| 523 | 554 \end{figure} |
| 555 | |
| 556 | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
557 \section{Experimental Results} |
| 549 | 558 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
559 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
560 %\vspace*{-1mm} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
561 %\subsection{SDA vs MLP vs Humans} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
562 %\vspace*{-1mm} |
| 535 | 563 The models are either trained on NIST (MLP0 and SDA0), |
| 564 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested | |
| 565 on either NIST, NISTP or P07, either on all 62 classes | |
| 566 or only on the digits (considering only the outputs | |
| 567 associated with digit classes). | |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
568 Figure~\ref{fig:error-rates-charts} summarizes the results obtained, |
| 535 | 569 comparing Humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1, |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
570 SDA2), along with the previous results on the digits NIST special database |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
571 19 test set from the literature respectively based on ARTMAP neural |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
572 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search |
|
516
092dae9a5040
make the reference more compact.
Frederic Bastien <nouiz@nouiz.org>
parents:
514
diff
changeset
|
573 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
574 ~\citep{Milgram+al-2005}. More detailed and complete numerical results |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
575 (figures and tables, including standard errors on the error rates) can be |
| 535 | 576 found in Appendix I of the supplementary material. |
| 577 The deep learner not only outperformed the shallow ones and | |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
578 previously published performance (in a statistically and qualitatively |
| 535 | 579 significant way) but when trained with perturbed data |
| 580 reaches human performance on both the 62-class task | |
| 523 | 581 and the 10-class (digits) task. |
| 582 | |
| 583 \begin{figure}[ht] | |
| 549 | 584 \vspace*{-3mm} |
| 523 | 585 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}} |
| 549 | 586 \vspace*{-3mm} |
| 523 | 587 \caption{Relative improvement in error rate due to self-taught learning. |
| 588 Left: Improvement (or loss, when negative) | |
| 589 induced by out-of-distribution examples (perturbed data). | |
| 590 Right: Improvement (or loss, when negative) induced by multi-task | |
| 591 learning (training on all classes and testing only on either digits, | |
| 592 upper case, or lower-case). The deep learner (SDA) benefits more from | |
| 593 both self-taught learning scenarios, compared to the shallow MLP.} | |
| 594 \label{fig:improvements-charts} | |
| 595 \vspace*{-2mm} | |
| 596 \end{figure} | |
| 597 | |
| 598 In addition, as shown in the left of | |
| 599 Figure~\ref{fig:improvements-charts}, the relative improvement in error | |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
600 rate brought by self-taught learning is greater for the SDA, and these |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
601 differences with the MLP are statistically and qualitatively |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
602 significant. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
603 The left side of the figure shows the improvement to the clean |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
604 NIST test set error brought by the use of out-of-distribution examples |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
605 (i.e. the perturbed examples examples from NISTP or P07). |
| 547 | 606 Relative percent change is measured by taking |
|
548
34cb28249de0
suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
547
diff
changeset
|
607 $100 \% \times$ (original model's error / perturbed-data model's error - 1). |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
608 The right side of |
| 523 | 609 Figure~\ref{fig:improvements-charts} shows the relative improvement |
|
486
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
610 brought by the use of a multi-task setting, in which the same model is |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
611 trained for more classes than the target classes of interest (i.e. training |
|
877af97ee193
section resultats et appendice
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
485
diff
changeset
|
612 with all 62 classes when the target classes are respectively the digits, |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
613 lower-case, or upper-case characters). Again, whereas the gain from the |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
614 multi-task setting is marginal or negative for the MLP, it is substantial |
| 547 | 615 for the SDA. Note that to simplify these multi-task experiments, only the original |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
616 NIST dataset is used. For example, the MLP-digits bar shows the relative |
| 547 | 617 percent improvement in MLP error rate on the NIST digits test set |
|
548
34cb28249de0
suggestions de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
547
diff
changeset
|
618 is $100\% \times$ (1 - single-task |
|
493
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
619 model's error / multi-task model's error). The single-task model is |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
620 trained with only 10 outputs (one per digit), seeing only digit examples, |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
621 whereas the multi-task model is trained with 62 outputs, with all 62 |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
622 character classes as examples. Hence the hidden units are shared across |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
623 all tasks. For the multi-task model, the digit error rate is measured by |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
624 comparing the correct digit class with the output class associated with the |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
625 maximum conditional probability among only the digit classes outputs. The |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
626 setting is similar for the other two target classes (lower case characters |
|
a194ce5a4249
difference stat. sign.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
491
diff
changeset
|
627 and upper case characters). |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
628 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
629 %\vspace*{-1mm} |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
630 %\subsection{Perturbed Training Data More Helpful for SDA} |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
631 %\vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
632 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
633 %\vspace*{-1mm} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
634 %\subsection{Multi-Task Learning Effects} |
|
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
635 %\vspace*{-1mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
636 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
637 \iffalse |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
638 As previously seen, the SDA is better able to benefit from the |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
639 transformations applied to the data than the MLP. In this experiment we |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
640 define three tasks: recognizing digits (knowing that the input is a digit), |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
641 recognizing upper case characters (knowing that the input is one), and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
642 recognizing lower case characters (knowing that the input is one). We |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
643 consider the digit classification task as the target task and we want to |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
644 evaluate whether training with the other tasks can help or hurt, and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
645 whether the effect is different for MLPs versus SDAs. The goal is to find |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
646 out if deep learning can benefit more (or less) from multiple related tasks |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
647 (i.e. the multi-task setting) compared to a corresponding purely supervised |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
648 shallow learner. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
649 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
650 We use a single hidden layer MLP with 1000 hidden units, and a SDA |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
651 with 3 hidden layers (1000 hidden units per layer), pre-trained and |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
652 fine-tuned on NIST. |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
653 |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
654 Our results show that the MLP benefits marginally from the multi-task setting |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
655 in the case of digits (5\% relative improvement) but is actually hurt in the case |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
656 of characters (respectively 3\% and 4\% worse for lower and upper class characters). |
| 495 | 657 On the other hand the SDA benefited from the multi-task setting, with relative |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
658 error rate improvements of 27\%, 15\% and 13\% respectively for digits, |
|
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
659 lower and upper case characters, as shown in Table~\ref{tab:multi-task}. |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
660 \fi |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
661 |
| 475 | 662 |
| 549 | 663 \vspace*{-2mm} |
| 529 | 664 \section{Conclusions and Discussion} |
| 549 | 665 \vspace*{-2mm} |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
666 |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
667 We have found that the self-taught learning framework is more beneficial |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
668 to a deep learner than to a traditional shallow and purely |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
669 supervised learner. More precisely, |
|
520
18a6379999fd
more after lunch :)
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
519
diff
changeset
|
670 the answers are positive for all the questions asked in the introduction. |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
671 %\begin{itemize} |
| 487 | 672 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
673 $\bullet$ %\item |
| 547 | 674 {\bf Do the good results previously obtained with deep architectures on the |
| 549 | 675 MNIST digits generalize to a much larger and richer (but similar) |
| 547 | 676 dataset, the NIST special database 19, with 62 classes and around 800k examples}? |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
677 Yes, the SDA {\bf systematically outperformed the MLP and all the previously |
| 529 | 678 published results on this dataset} (the ones that we are aware of), {\bf in fact reaching human-level |
| 679 performance} at around 17\% error on the 62-class task and 1.4\% on the digits. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
680 |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
681 $\bullet$ %\item |
| 547 | 682 {\bf To what extent do self-taught learning scenarios help deep learners, |
| 683 and do they help them more than shallow supervised ones}? | |
| 529 | 684 We found that distorted training examples not only made the resulting |
| 685 classifier better on similarly perturbed images but also on | |
| 686 the {\em original clean examples}, and more importantly and more novel, | |
| 687 that deep architectures benefit more from such {\em out-of-distribution} | |
| 688 examples. MLPs were helped by perturbed training examples when tested on perturbed input | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
689 images (65\% relative improvement on NISTP) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
690 but only marginally helped (5\% relative improvement on all classes) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
691 or even hurt (10\% relative loss on digits) |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
692 with respect to clean examples . On the other hand, the deep SDAs |
| 472 | 693 were very significantly boosted by these out-of-distribution examples. |
| 529 | 694 Similarly, whereas the improvement due to the multi-task setting was marginal or |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
695 negative for the MLP (from +5.6\% to -3.6\% relative change), |
| 547 | 696 it was very significant for the SDA (from +13\% to +27\% relative change), |
| 697 which may be explained by the arguments below. | |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
698 %\end{itemize} |
| 472 | 699 |
|
524
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
700 In the original self-taught learning framework~\citep{RainaR2007}, the |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
701 out-of-sample examples were used as a source of unsupervised data, and |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
702 experiments showed its positive effects in a \emph{limited labeled data} |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
703 scenario. However, many of the results by \citet{RainaR2007} (who used a |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
704 shallow, sparse coding approach) suggest that the relative gain of self-taught |
| 536 | 705 learning diminishes as the number of labeled examples increases (essentially, |
| 529 | 706 a ``diminishing returns'' scenario occurs). We note instead that, for deep |
|
524
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
707 architectures, our experiments show that such a positive effect is accomplished |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
708 even in a scenario with a \emph{very large number of labeled examples}. |
|
07bc0ca8d246
added paragraph comparing "our" self-taught learning with "theirs"
Dumitru Erhan <dumitru.erhan@gmail.com>
parents:
523
diff
changeset
|
709 |
| 547 | 710 {\bf Why would deep learners benefit more from the self-taught learning framework}? |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
711 The key idea is that the lower layers of the predictor compute a hierarchy |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
712 of features that can be shared across tasks or across variants of the |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
713 input distribution. Intermediate features that can be used in different |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
714 contexts can be estimated in a way that allows to share statistical |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
715 strength. Features extracted through many levels are more likely to |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
716 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest), |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
717 increasing the likelihood that they would be useful for a larger array |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
718 of tasks and input conditions. |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
719 Therefore, we hypothesize that both depth and unsupervised |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
720 pre-training play a part in explaining the advantages observed here, and future |
|
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
721 experiments could attempt at teasing apart these factors. |
| 529 | 722 And why would deep learners benefit from the self-taught learning |
| 723 scenarios even when the number of labeled examples is very large? | |
| 724 We hypothesize that this is related to the hypotheses studied | |
| 725 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010} | |
| 726 it was found that online learning on a huge dataset did not make the | |
| 727 advantage of the deep learning bias vanish, a similar phenomenon | |
| 728 may be happening here. We hypothesize that unsupervised pre-training | |
| 729 of a deep hierarchy with self-taught learning initializes the | |
| 730 model in the basin of attraction of supervised gradient descent | |
| 731 that corresponds to better generalization. Furthermore, such good | |
| 732 basins of attraction are not discovered by pure supervised learning | |
| 733 (with or without self-taught settings), and more labeled examples | |
| 734 does not allow to go from the poorer basins of attraction discovered | |
| 735 by the purely supervised shallow models to the kind of better basins associated | |
| 736 with deep learning and self-taught learning. | |
|
502
2b35a6e5ece4
changements de Myriam
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
501
diff
changeset
|
737 |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
738 A Flash demo of the recognizer (where both the MLP and the SDA can be compared) |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
739 can be executed on-line at {\tt http://deep.host22.com}. |
|
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
740 |
|
498
7ff00c27c976
add missing file for bibtex and make it smaller.
Frederic Bastien <nouiz@nouiz.org>
parents:
496
diff
changeset
|
741 \newpage |
|
496
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
742 { |
|
e41007dd40e9
make the reference shorter.
Frederic Bastien <nouiz@nouiz.org>
parents:
495
diff
changeset
|
743 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials} |
| 469 | 744 %\bibliographystyle{plainnat} |
| 745 \bibliographystyle{unsrtnat} | |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
746 %\bibliographystyle{apalike} |
|
484
9a757d565e46
reduction de taille
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
483
diff
changeset
|
747 } |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
748 |
|
485
6beaf3328521
les tables enlevées
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
484
diff
changeset
|
749 |
|
464
24f4a8b53fcc
nips2010_submission.tex
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff
changeset
|
750 \end{document} |